Displaying All Providers and All Authors separately in Susper Angular App

In Susper results are retrieved from a lots of providers and authors. Displaying all authors’ and providers’ list on result page make appearance of result section odd and more complex. In this blog, I will describe how we have separated the list of all providers and authors from main result page and displayed it on a lightbox.

Steps:

  • Making a black overlay for lightbox:

First we of all to display the results on a lightbox,we to have a black overlay for this we need a div element and inside it we need a closing button.

Clicking on the black overlay should toggle status of light box therefore we have binded click event with showMore() method.

<div id="fade_analytics" class="black_overlay" (click)='showMore()'><a class="closeX" id="closeX">&#10006;</a></div>
  • Creating a white content to display all Providers or Authors:

Now we need a white block to display all Providers or all Authors so after creating a black overlay in background we need a white foreground.

Therefore I have implemented a div element and put all the chart elements inside the div element so that it will be displayed on a white foreground on a translucent black background.

<div id="light_analytics" class="white_content_analytics">
<div class="show_more"><h2 class="heading"><b>Analytics</b></h2><div id="filtersearch" *ngFor="let nav of navigation$|async; let i = index"><div class="card-container" id="relate" *ngIf="nav.displayname =='Provider' || nav.displayname =='Authors'"><h3  class="related-searches" *ngIf="i===currentelement"><b>Top {{nav.displayname}}<span *ngIf="nav.displayname.slice(-1)!=='s'">s</span></b></h3>
<div style="display: block;cursor:pointer;cursor: hand;" *ngFor="let element of  ((i != currentelement)? (nav.elements | slice:0:0) :nav.elements)">
</div></div></div></div>

The code to display All providers and All authors are contained inside the   white block and the logic to display the each section is also implemented.

  • Showing only 50 results on result page and implementing a button to open all providers or authors in light box:

Currently we are displaying the top five providers and authors and we provide a button to show more providers and authors which then displays 50 more results and then we display an interactive button that toggles display of all authors and providers on a lightbox. To display 5 and 50 results we use slice pipe in angular. The button Show All Providers/Authors is binded with showMore() function which toggles lightbox.

<div style="display: block;cursor:pointer;cursor: hand;" *ngFor="let element of ((i != selectedelement)? (nav.elements | slice:0:5) :(nav.elements | slice:0:50) )">
<a [routerLink]="['/search']" [queryParams]="changequerylook(element.modifier,element)" *ngIf="element.name" href="#" [style.color]="themeService.linkColor">
{{element.name}} <span class="badge badge-info">{{element.count}}</span>
</a><a  *ngIf="!element.name" style="color: black;cursor: default" href="#" [style.color]="themeService.linkColor">
Undefined Author <span class="badge badge-info">{{element.count}}</span></a></div>
<button class="showMoreBtn" (click)="showMore()" *ngIf="selectedelement===i" [style.color]="themeService.linkColor">Show All {{nav.displayname}}<span *ngIf="nav.displayname.slice(-1)!=='s'">s</span></button>

 

  • Implementing the logic to display all results on white content:

Now we need to implement the logic to toggle lightbox to display All Providers and All Authors. This logic is implemented mainly in showMore() function which toggles the lightbox containing the All Providers and All Authors

showMore() { this.selectedelement = -1;
if (!this.showMoreAnalytics) { this.showMoreAnalytics = true; document.getElementById('light_analytics').style.display = 'block'; document.getElementById('fade_analytics').style.display = 'block'; } else { this.showMoreAnalytics = false;
document.getElementById('light_analytics').style.display = 'none';
document.getElementById('fade_analytics').style.display = 'none';}}

Here we use showMoreAnalytics variable to check whether the user has asked to show all Providers or Authors. We also make selectedelement variable to -1 to hide the long list of 50 Providers or Authors when Lightbox displays all the Providers or Author. Finally we use document object in javascript and select the DOM elements by their id and set their display property this shows and hides the lightbox on result page.

Resources

1.YaCy Advanced Search Parameters

http://www.yacy-websuche.de/wiki/index.php/En:SearchParameters

2.W3School lightbox

https://www.w3schools.com/howto/howto_js_lightbox.asp

 

Continue ReadingDisplaying All Providers and All Authors separately in Susper Angular App

Adding New tests for Knowledge Service

Testing is done to test our application by executing some functions by creating instances of corresponding classes,executing functions and checking the actual behaviour of our app with expected result. The tools and frameworks used in Angular are Jasmine and Karma. In this blog, I will describe about how I have implemented tests for Newly Added Knowledge API service that helped us to increase overall code coverage by 1.05% in Susper .

Adding tests for Knowledge API:

We need to check the API that whether it is functioning or not and this can be done by using a mocked response (hardcoded response) for any query and then comparing this with the received response from the API. This will help us to check proper functioning of our API.

This is a common practice in Angular and to achieve this we will be using some dependencies like MockBackend, MockConnection, BaseRequestOptions provided by Angular.

import { MockBackend, MockConnection } from '@angular/http/testing';
import { Http, Jsonp, BaseRequestOptions, RequestMethod, Response, ResponseOptions, HttpModule, JsonpModule } from '@angular/http';

 

We will also need to define a Mock response, here I have used hard coded response for query India. Here is the mocked response.

export const MockKnowledgeApi = {
    results: [
        {"batchcomplete": "", "query":
        {"normalized":
        [{"from": "india", "to": "India"}], "pages":
        {"14533": {"pageid": 14533, "ns": 0, "title": "India", "extract": `India (IAST: Bh\u0101rat), also called the Republic of India (IAST: Bh\u0101rat Ga\u1e47ar\u0101jya), is a country in South Asia.
   }}}} ], MaxHits : 5 };

 

Now we will use a Mock constant mock_Http_provider for http and will inject instances of MockBackend and BaseRequestOptions.

const mockHttp_provider = {
  provide: Http,
  deps: [MockBackend, BaseRequestOptions],
  useFactory: (backend: MockBackend, options: BaseRequestOptions) => { return new Http(backend, options); } };

 

Now we need to add all the services and dependencies which we will be using in providers and we will inject the instances of Knowledgeapi Service and MockBackend in beforeEach function.

beforeEach(inject([KnowledgeapiService, MockBackend], (knowledgeService: KnowledgeapiService, mockBackend: MockBackend) => { service = knowledgeService;
    backend = mockBackend;}));

 

Now we will use the same query for which we have created the mocked response and we will be checking the response of from our API.

const searchquery = 'india';
connection.mockRespond(new Response(options));
      expect(connection.request.method).toEqual(RequestMethod.Get);
      expect(connection.request.url).toBe(   `https://en.wikipedia.org/w/api.php?&origin=*&format=json&action=query&prop=extracts&exintro=&explaintext=&` +
                    `titles=${searchquery}`);});
service.getSearchResults(searchquery).subscribe((res) => {
      expect(res).toEqual(MockKnowledgeApi);});

 

This will check the working of our API and if it is working then our test case will pass.

Like this we have implemented the tests for Knowledgeapi Service which helped us to test our API and increase overall code coverage significantly.

Resources

  1. Testing in Angular:https://angular.io/guide/testing
  2. Testing by Mockbackend:https://angular.io/api/http/testing/MockBackend
  3. Using MockBackend to simulate response: https://codecraft.tv/courses/angular/unit-testing/http-and-jsonp/#_using_the_code_mockbackend_code_to_simulate_a_response
Continue ReadingAdding New tests for Knowledge Service

Using routerLink to make result tabs as links

In Susper, it was required that different result sections i.e All, Images,Videos and News should behave as links. Links provide us an advantage of opening the different result tabs in a New Tab by right clicking on it. Also at the same time we wanted that Active tabs must not behave as links. We wanted that tabs should behave in a similar way as Google, Bing and other search engine.In this blog, I will describe how we have achieved this by using routerLink and [hidden] attribute in Angular.

Steps:

1.Adding routerLink attribute in anchor tag.

Now, To make any element as link we must enclose it in anchor tag, therefore we will use <a> tag enclosing the <li> tag to make it a link.

<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,resultDisplay:'all',mode:'text',nopagechange:'false',append:'false'}"><li [class.active_view]="Display('all')" (click)="docClick()">All</li></a>
<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,mode:'text',nopagechange:'false',append:'false',fq:'url_file_ext_s%3A(png%2BOR%2Bjpeg%2BOR%2Bjpg%2BOR%2Bgif)',resultDisplay:'images'}"><li [class.active_view]="Display('images')" (click)="imageClick()">Images</li></a>
<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,resultDisplay:'videos',mode:'text',nopagechange:'false',append:'false',fq:'url_file_ext_s%3A(avi%2BOR%2Bmov%2BOR%2Bflv%2BOR%2Bmp4)'}"><li [class.active_view]="Display('videos')" (click)="videoClick()">Videos</li></a>
<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,resultDisplay:'news',mode:'text',nopagechange:'false',append:'false'}"><li [class.active_view]="Display('news')" (click)="newsClick()">News</li></a>

Now we must navigate to a link on the click of this tabs, for this we can simply use href attribute which we earlier tried to use but there is a disadvantage of using href tag. Using href tab reloads the whole page when we try to navigate to another tab.

Therefore we will use routerLink to achieve the same functionality as Susper is a single page application, where the page should not be reloaded. routerLink navigates to the new url and the component is rendered in place of <router-outlet> without reloading the whole page.

After using routerLink, we will use queryParams attribute to pass routing parameters to a route. Since these parameters will be variable we must use string interpolation to set query parameters. We will use searchdata object and set query parameters according to its value.

Now we are done with making tabs as links. But as of now all tabs will behave a links even if it is an active tab. Therefore we need to hide the link property of any tab when it is active.

2.Hiding the link property of the active tab:

Now we will use to different elements for each tab, one as link and other as normal li element.For hiding the link elements when the tab is active we will use [hidden] attribute in angular. We will just hide the link element when the value of resultDisplay variable is similar to current tab.

<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,resultDisplay:'all',mode:'text',nopagechange:'false',append:'false'}" [hidden]="this.resultDisplay==='all'"><li [class.active_view]="Display('all')" (click)="docClick()">All</li></a>
        <li [class.active_view]="Display('all')" (click)="docClick()" *ngIf="Display('all')">All</li>

For hiding the simple li element we will use *ngIf attribute which will remove the simple li element from the DOM for un active tabs. We might have used *ngIf attribute to hide link elements but rendering a lot of elements on DOM slows down the page and affects performance.

We will follow this for rest of the tabs also and we are done with our changes.

Now we can the inactive tabs will behave as link and the active tabs will not behave as link similar to Google.

Resources

  1. Angular Routing: https://angular.io/guide/router
  2. ngIf and [hidden]: https://stackoverflow.com/questions/39777990/angular2-conditional-display-bind-to-hidden-property-vs-ngif-directive
  3. Angular queryParams: https://stackoverflow.com/a/46008187
Continue ReadingUsing routerLink to make result tabs as links

Adding News Tab in Susper

Most of the current search engines have a News tab where results are displayed which are fetched from different News Organisations. The latest results are displayed on top followed by the older ones. Current market leader, Google even uses AI and Machine Learning to analyze the contents before delivering it as results to the user. In this blog, I will describe how I have implemented a basic News Tab in Susper which show news results from dailymail.co.uk . This News tab can be improved in future to match the market leader.

Implementation of News tab in Susper:

Implementation of News tab in UI:

To implement News tab the current design pattern has been followed in Susper. Here is the code to show News tab on the results page.

<ul type="none" id="search-options">
       <li [class.active_view]="Display('news')" (click)="newsClick()">News</li>

Angular attribute binding is used to bind the active_view property with the “Display(‘news’)” function and the click event is associated with “newsClick()” function.

Here is the code for newsClick() function which filters the result using ‘site:’ parameter of yacy and then dispatches the ‘urldata’ object to ‘QueryServerAction(urldata)’ function in Query Action.

newsClick() {
   let urldata = Object.assign({}, this.searchdata);
   this.getPresentPage(1);
   this.resultDisplay = 'news';
   delete urldata.fq;
   urldata.rows = 10;
   if (urldata.query.substr(urldata.query.length - 25, 25) !== " site:www.dailymail.co.uk") {
     urldata.query += " site:www.dailymail.co.uk";
   } else {
     urldata.query += "";
   }
   urldata.resultDisplay = this.resultDisplay;
   this.store.dispatch(new queryactions.QueryServerAction(urldata));
   }

Here is the code for Query Action in query.ts file. It has two actions QueryAction and QueryServerAction

export const ActionTypes = {
 QUERYCHANGE: type('[Query] Change'),
 QUERYSERVER: type('[Query] Server'),
};
export class QueryAction implements Action {
 type = ActionTypes.QUERYCHANGE;

 constructor(public payload: any) {}
}
export class QueryServerAction implements Action {
 type = ActionTypes.QUERYSERVER;
 constructor(public payload: any) {}
}
export type Actions
 = QueryAction|QueryServerAction ;

Now the reducer takes either of the two query actions and the current state and returns the new state to the store.

Here is the code for the reducer query.ts

export const CHANGE = 'CHANGE';
export interface State {
 query: string;
 wholequery: any;
}
const initialState: State = {
 query: '',
 wholequery: {
   query: '',
   rows: 10,
   start: 0,
   mode: 'text'
 },
};
export function reducer(state: State = initialState, action: query.Actions): State {
 switch (action.type) {
   case query.ActionTypes.QUERYCHANGE: {
     const changeQuery = action.payload;
     return Object.assign({}, state, {
       query: changeQuery,
       wholequery: state.wholequery
     });
   }
   case query.ActionTypes.QUERYSERVER: {
     let serverQuery = Object.assign({}, action.payload);
     let resultCount = 10;
     if (localStorage.getItem('resultscount')) {
       resultCount = JSON.parse(localStorage.getItem('resultscount')).value || 10;
     }
     let instantsearch = JSON.parse(localStorage.getItem('instantsearch'));

     if (instantsearch && instantsearch.value) {
       resultCount = 10;
     }

     serverQuery.rows = resultCount;
     return Object.assign({}, state, {
       wholequery: serverQuery,
       query: state.query
     });
   }
   default: {
     return state;
   }
 }
}
export const getpresentquery = (state: State) => state.query;
export const getpresentwholequery = (state: State) => state.wholequery;

 

From the store the modified query is available to all other components of the appand is used by search service to get the filtered results.

The search results are then stored in observable items$ in results.components.ts and is then used in results.components.html to display results under News tab.

Here is the code to display the results.

 <div class="feed container">
     <div *ngFor="let item of items$|async" class="result">
       <div class="title">
         <a class="title-pointer" href="{{item.link}}">{{item.title}}</a>
       </div>
       <div class="link">
         <p>{{item.link}}</p>
       </div>
     </div>
   </div>

 

Resources

1.YaCy Search Parameters: http://www.yacywebsearch.net/wiki/index.php/En:SearchParameters

2.Redux in Angular: http://blog.ng-book.com/introduction-to-redux-with-typescript-and-angular-2/

3.Corresponding PR: https://github.com/fossasia/susper.com/pull/1023

Continue ReadingAdding News Tab in Susper

Deploying Angular App Susper on GitHub Pages

What are the issues we had with automatic deployment of Susper?

SUSPER project which is built on Angular is kept on master branch of the repository .

Whenever any PR is merged,Travis CI does all the builds, and checks for any error. After successful checking it triggers deployment script in SUSPER (deploy.sh) .

Here is the deployment script:

#!/bin/bash

# SOURCE_BRANCH & TARGET_BRANCH stores the name of different susper.com github branches.
SOURCE_BRANCH="master"
TARGET_BRANCH="gh-pages"

# Pull requests and commits to other branches shouldn't try to deploy.
if [ "$TRAVIS_PULL_REQUEST" != "false" -o "$TRAVIS_BRANCH" != "$SOURCE_BRANCH" ]; then
    echo "Skipping deploy; The request or commit is not on master"
    exit 0
fi

# Store some useful information into variables
REPO=`git config remote.origin.url`
SSH_REPO=${REPO/https:\/\/github.com\//git@github.com:}
SHA=`git rev-parse --verify HEAD`

# Decryption of the `deploy.enc`
openssl aes-256-cbc -k "$super_secret_password" -in deploy.enc -out deploy_key -d

# give `deploy_key`. the permission to read and write
chmod 600 deploy_key
eval `ssh-agent -s`
ssh-add deploy_key

# Cloning the repository to repo/ directory,
# Creating gh-pages branch if it doesn't exists else moving to that branch
git clone $REPO repo
cd repo
git checkout $TARGET_BRANCH || git checkout --orphan $TARGET_BRANCH
cd ..

# Setting up the username and email.
git config user.name "Travis CI"
git config user.email "$COMMIT_AUTHOR_EMAIL"

# Cleaning up the old repo's gh-pages branch except CNAME file and 404.html
find repo/* ! -name "CNAME" ! -name "404.html" -maxdepth 1  -exec rm -rf {} \; 2> /dev/null
cd repo

# commit the changes, move to SOURCE_BRANCH
git add --all
git commit -m "Travis CI Clean Deploy : ${SHA}"

git checkout $SOURCE_BRANCH

# Actual building and setup of current push or PR. Move to `TARGET_BRANCH` and move the `dist` folder
npm install
ng build --prod --aot
mv susper.xml dist/
mv 404.html dist/

git checkout $TARGET_BRANCH
mv dist/* .

# Staging the new build for commit; and then committing the latest build
git add .
git commit --amend --no-edit --allow-empty

# Actual push to gh-pages branch via Travis
git push --force $SSH_REPO $TARGET_BRANCH

This script starts after successfully checking the project (comments shows working of each command). The repository is cleaned,except some files like CNAME and 404.html and a commit is made. Then SUSPER’s build artifacts are generated in dist folder and all the files from dist folder are moved to gh-pages and the changes are amended. GitHub Pages engine then uses the build artifacts to generate static site at susper.com. But we faced a weird problem when we were unable to see changes from latest commits on susper.com. Slowly number of commits increased but still changes were not reflecting on susper.com .

What was the error in deployment of SUSPER?

All the changes were getting committed to the gh-pages branch properly. But the GitHub Pages engine was unable to process the branch to build static site. Due to this the site was not getting updated. The failing builds from gh-engine are notified to the owner via email.

The failing builds from gh-pages can be seen here https://github.com/fossasia/susper.com/commits/gh-pages

Between 21 May to 18 March.

There was a weird error ( Failure: The tag `chartjs` in line 4 on `node_modules/chart.js/docs/charts/bar.md` is not recognized liquid tag) while building the site from build artifacts. There was an issue with `gh-pages` engine as `node_modules/chart.js/docs/charts/bar.md`  was previously also present but then there was no such errors than.

Due to this error all commits which were made to SUSPER repository was not shown on susper.com as the site was not getting updated.

How we solved it?

We tried to search a lot about the above mentioned error but we were unable to find a proper solution. While searching we came across this question https://stackoverflow.com/questions/24713112/my-github-page-wont-update-its-content

on StackOverflow. There was no accepted answer for this question and every answer was different from other. So, we just reproduced the bug in our own repository by just making a similar repository to SUSPER with same branches. Once we successfully reproduced the bug. We tried to correlate all the answers to arrive at a solution to fix this bug. Luckily we found that any commit with no significant change from previous commit removed the error and allowed the gh-pages engine to successfully create the site. So we made a dummy commit in which we created a hidden file with no content and pushed it to gh-pages. This did the trick for us and gh-pages was now able build the site for us.

The whole problem was due to a bug in GitHub pages engine which got fixed by a dummy commit.

Resources

  1. Stackoverflow:https://stackoverflow.com/questions/24713112/my-github-page-wont-update-its-content
  2. Publishing to GitHub Pages:https://help.github.com/articles/configuring-a-publishing-source-for-github-pages/
  3. Commits(gh-pages):https://github.com/fossasia/susper.com/commits/gh-pages
Continue ReadingDeploying Angular App Susper on GitHub Pages

Adding Yarn as new Dependency Manager  along with NPM in Susper

Dependency managers are software modules that coordinate the integration of external libraries or packages into larger application stack. Dependency managers use configuration files like composer.json, package.json, build.gradle or pom.xml to determine: What dependency to get, What version of the dependency in particular and, Which repository to get them from. Currently SUSPER has only NPM as a dependency manager which is used to install all dependencies. In this blog, I will describe how we have added facebook’s Yarn as a new dependency manager in Susper

Lets checkout Yarn in detail:

Yarn is a fast and good alternative to NPM. One of the great advantages of Yarn is that while remaining compatible with the npm registry, it replaces the workflow for npm client or other package managers Yarn was created by Facebook, to solve some particular problems that were faced while using NPM. Yarn was developed to deal with inconsistency in dependency installation while scaling and to increase speed.

What is advantages of using Yarn?

  • Improving Network performance:Queuing up the requests and avoiding requests waterfalls helps to maximize network utilization.
  • Checks Package Integrity:Package integrity is checked after each install to avoid corrupt packages installation.
  • Checks Package Integrity:Package integrity is checked after each install to avoid corrupt packages installation.
  • Caching: Yarn helps to install the dependencies without an internet connection if the dependency has been previously installed on the system. This is done by caching.
  • Lock File: Lock files are used to make sure that the node_modules directory has the exact same structure on all development environments.

Source: https://yarnpkg.com/en/

How Yarn is installed along with NPM in SUSPER?

Installing Yarn is super easy. Here are the steps to setup Yarn along with NPM and begin using it as dependency manager.

On Debian or Ubuntu Linux, we can install Yarn via our Debian package repository. We will first need to configure the repository:

curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee 

/etc/apt/sources.list.d/yarn.list

 

Then simply use:

sudo apt-get update && sudo apt-get install yarn

 

Note: Ubuntu 17.04 comes with cmdtest installed by default. If anyone gets any errors from installing yarn, then remove it by sudo apt remove cmdtest first. Refer to this for more information.

If using nvm you can avoid the node installation by doing:

sudo apt-get install --no-install-recommends yarn

 

Test that Yarn is installed by running:

yarn --version

 

Now delete the node_modules folder so that all dependencies installed by npm is removed.

Now use yarn command in project’s repository.

yarn 

 

Wait while dependencies are installed and then we will be done.

What is happening ?

Yarn has created a lock file  yarn.lock. After each operation the file is updated (installing, updating or removing packages) to keep the track of exact package version. If kept in our Git repository we can see that the exact same result in node_modules is made available to all systems.

Resources

  1. Yarn: https://yarnpkg.com/en/
  2. Announcement of Yarn: https://code.facebook.com/posts/1840075619545360
  3. Yarn Vs NPM: https://stackoverflow.com/questions/40027819/when-to-use-yarn-over-npm-what-are-the-differences
Continue ReadingAdding Yarn as new Dependency Manager  along with NPM in Susper