Getting results for Multi-word Query and showing limited results in Knowledge Graph

In Susper knowledge graph,we were unable to get results for multi word query from Wikipedia API and also we were unable to decide how much information should be shown in knowledge graph which was retrieved from Wikipedia API.For example for a query donald trump we do not get information (https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=donald%20trump) . Also for searching for any query like india (https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=india) we get a lot of information. Earlier we used Angular slice pipe to display only 600 characters in knowledge graph (Infobox) but almost for all queries the sentences were terminated before it was completed. In this blog, I will describe how we solved both problems in knowledge graph.

Getting results for multi word query from Wikipedia API:

On searching a lot we found that the results were present on Wikipedia for multi word queries but these queries must have its starting letters as capital letters. For example we do not get results for queries such as donald trump

i.e https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=donald%20trump

but when we make a query for Donald Trump  

i.e

https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=Donald%20Trump  

But since in most of the search queries users use small letters, we need to convert the query such that each word starts with a capital letter. Since the Knowledge Graph has been implemented using ngrx pattern, this logic was easily implemented in knowledge effects. For this we will be selecting each word in query by using regular expression and then we will use toUppercase() and toLowerCase() methods of javascript and will capitalize every word of the query.

toTitleCase(str) {
    return str.replace(
        /\w\S*/g,
        function(txt) {
            return txt.charAt(0).toUpperCase() + txt.substr(1).toLowerCase();
        }
    );
}

this.knowledgeservice.getSearchResults(this.toTitleCase(querypay.query))

 

Like this we now get results for almost all queries.

Limiting information in Knowledge Graph without terminating the sentences:

To solve this issue, we decided to show only four lines of data retrieved from Wikipedia API

For this we have implemented a getPosition function which will take three parameters a string, a substring and a index. Here string is the whole string from which the function will return the index of index position substring.

getPosition(string, subString, index) {
    return string.split(subString, index).join(subString).length;
 }
this.description = this.description.slice(0, this.getPosition(this.description, '.', 4) + 1);

 

Here, We get the index of 4th ‘.’ full stop by getPosition() function and then pass it to javascript slice method to slice the string upto that position.

Using this we limited results to 4 lines without terminating each line in middle.

References

1.W3Schools Regular Expression: https://www.w3schools.com/jsref/jsref_obj_regexp.asp

2.Javascript Slice method: https://www.w3schools.com/jsref/jsref_slice_array.asp

3.Wikipedia API: https://www.mediawiki.org/wiki/API:Main_page

Displaying All Providers and All Authors separately in Susper Angular App

In Susper results are retrieved from a lots of providers and authors. Displaying all authors’ and providers’ list on result page make appearance of result section odd and more complex. In this blog, I will describe how we have separated the list of all providers and authors from main result page and displayed it on a lightbox.

Steps:

  • Making a black overlay for lightbox:

First we of all to display the results on a lightbox,we to have a black overlay for this we need a div element and inside it we need a closing button.

Clicking on the black overlay should toggle status of light box therefore we have binded click event with showMore() method.

<div id="fade_analytics" class="black_overlay" (click)='showMore()'><a class="closeX" id="closeX">&#10006;</a></div>
  • Creating a white content to display all Providers or Authors:

Now we need a white block to display all Providers or all Authors so after creating a black overlay in background we need a white foreground.

Therefore I have implemented a div element and put all the chart elements inside the div element so that it will be displayed on a white foreground on a translucent black background.

<div id="light_analytics" class="white_content_analytics">
<div class="show_more"><h2 class="heading"><b>Analytics</b></h2><div id="filtersearch" *ngFor="let nav of navigation$|async; let i = index"><div class="card-container" id="relate" *ngIf="nav.displayname =='Provider' || nav.displayname =='Authors'"><h3  class="related-searches" *ngIf="i===currentelement"><b>Top {{nav.displayname}}<span *ngIf="nav.displayname.slice(-1)!=='s'">s</span></b></h3>
<div style="display: block;cursor:pointer;cursor: hand;" *ngFor="let element of  ((i != currentelement)? (nav.elements | slice:0:0) :nav.elements)">
</div></div></div></div>

The code to display All providers and All authors are contained inside the   white block and the logic to display the each section is also implemented.

  • Showing only 50 results on result page and implementing a button to open all providers or authors in light box:

Currently we are displaying the top five providers and authors and we provide a button to show more providers and authors which then displays 50 more results and then we display an interactive button that toggles display of all authors and providers on a lightbox. To display 5 and 50 results we use slice pipe in angular. The button Show All Providers/Authors is binded with showMore() function which toggles lightbox.

<div style="display: block;cursor:pointer;cursor: hand;" *ngFor="let element of ((i != selectedelement)? (nav.elements | slice:0:5) :(nav.elements | slice:0:50) )">
<a [routerLink]="['/search']" [queryParams]="changequerylook(element.modifier,element)" *ngIf="element.name" href="#" [style.color]="themeService.linkColor">
{{element.name}} <span class="badge badge-info">{{element.count}}</span>
</a><a  *ngIf="!element.name" style="color: black;cursor: default" href="#" [style.color]="themeService.linkColor">
Undefined Author <span class="badge badge-info">{{element.count}}</span></a></div>
<button class="showMoreBtn" (click)="showMore()" *ngIf="selectedelement===i" [style.color]="themeService.linkColor">Show All {{nav.displayname}}<span *ngIf="nav.displayname.slice(-1)!=='s'">s</span></button>

 

  • Implementing the logic to display all results on white content:

Now we need to implement the logic to toggle lightbox to display All Providers and All Authors. This logic is implemented mainly in showMore() function which toggles the lightbox containing the All Providers and All Authors

showMore() { this.selectedelement = -1;
if (!this.showMoreAnalytics) { this.showMoreAnalytics = true; document.getElementById('light_analytics').style.display = 'block'; document.getElementById('fade_analytics').style.display = 'block'; } else { this.showMoreAnalytics = false;
document.getElementById('light_analytics').style.display = 'none';
document.getElementById('fade_analytics').style.display = 'none';}}

Here we use showMoreAnalytics variable to check whether the user has asked to show all Providers or Authors. We also make selectedelement variable to -1 to hide the long list of 50 Providers or Authors when Lightbox displays all the Providers or Author. Finally we use document object in javascript and select the DOM elements by their id and set their display property this shows and hides the lightbox on result page.

Resources

1.YaCy Advanced Search Parameters

http://www.yacy-websuche.de/wiki/index.php/En:SearchParameters

2.W3School lightbox

https://www.w3schools.com/howto/howto_js_lightbox.asp

 

Adding New tests for Knowledge Service

Testing is done to test our application by executing some functions by creating instances of corresponding classes,executing functions and checking the actual behaviour of our app with expected result. The tools and frameworks used in Angular are Jasmine and Karma. In this blog, I will describe about how I have implemented tests for Newly Added Knowledge API service that helped us to increase overall code coverage by 1.05% in Susper .

Adding tests for Knowledge API:

We need to check the API that whether it is functioning or not and this can be done by using a mocked response (hardcoded response) for any query and then comparing this with the received response from the API. This will help us to check proper functioning of our API.

This is a common practice in Angular and to achieve this we will be using some dependencies like MockBackend, MockConnection, BaseRequestOptions provided by Angular.

import { MockBackend, MockConnection } from '@angular/http/testing';
import { Http, Jsonp, BaseRequestOptions, RequestMethod, Response, ResponseOptions, HttpModule, JsonpModule } from '@angular/http';

 

We will also need to define a Mock response, here I have used hard coded response for query India. Here is the mocked response.

export const MockKnowledgeApi = {
    results: [
        {"batchcomplete": "", "query":
        {"normalized":
        [{"from": "india", "to": "India"}], "pages":
        {"14533": {"pageid": 14533, "ns": 0, "title": "India", "extract": `India (IAST: Bh\u0101rat), also called the Republic of India (IAST: Bh\u0101rat Ga\u1e47ar\u0101jya), is a country in South Asia.
   }}}} ], MaxHits : 5 };

 

Now we will use a Mock constant mock_Http_provider for http and will inject instances of MockBackend and BaseRequestOptions.

const mockHttp_provider = {
  provide: Http,
  deps: [MockBackend, BaseRequestOptions],
  useFactory: (backend: MockBackend, options: BaseRequestOptions) => { return new Http(backend, options); } };

 

Now we need to add all the services and dependencies which we will be using in providers and we will inject the instances of Knowledgeapi Service and MockBackend in beforeEach function.

beforeEach(inject([KnowledgeapiService, MockBackend], (knowledgeService: KnowledgeapiService, mockBackend: MockBackend) => { service = knowledgeService;
    backend = mockBackend;}));

 

Now we will use the same query for which we have created the mocked response and we will be checking the response of from our API.

const searchquery = 'india';
connection.mockRespond(new Response(options));
      expect(connection.request.method).toEqual(RequestMethod.Get);
      expect(connection.request.url).toBe(   `https://en.wikipedia.org/w/api.php?&origin=*&format=json&action=query&prop=extracts&exintro=&explaintext=&` +
                    `titles=${searchquery}`);});
service.getSearchResults(searchquery).subscribe((res) => {
      expect(res).toEqual(MockKnowledgeApi);});

 

This will check the working of our API and if it is working then our test case will pass.

Like this we have implemented the tests for Knowledgeapi Service which helped us to test our API and increase overall code coverage significantly.

Resources

  1. Testing in Angular:https://angular.io/guide/testing
  2. Testing by Mockbackend:https://angular.io/api/http/testing/MockBackend
  3. Using MockBackend to simulate response: https://codecraft.tv/courses/angular/unit-testing/http-and-jsonp/#_using_the_code_mockbackend_code_to_simulate_a_response

Using routerLink to make result tabs as links

In Susper, it was required that different result sections i.e All, Images,Videos and News should behave as links. Links provide us an advantage of opening the different result tabs in a New Tab by right clicking on it. Also at the same time we wanted that Active tabs must not behave as links. We wanted that tabs should behave in a similar way as Google, Bing and other search engine.In this blog, I will describe how we have achieved this by using routerLink and [hidden] attribute in Angular.

Steps:

1.Adding routerLink attribute in anchor tag.

Now, To make any element as link we must enclose it in anchor tag, therefore we will use <a> tag enclosing the <li> tag to make it a link.

<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,resultDisplay:'all',mode:'text',nopagechange:'false',append:'false'}"><li [class.active_view]="Display('all')" (click)="docClick()">All</li></a>
<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,mode:'text',nopagechange:'false',append:'false',fq:'url_file_ext_s%3A(png%2BOR%2Bjpeg%2BOR%2Bjpg%2BOR%2Bgif)',resultDisplay:'images'}"><li [class.active_view]="Display('images')" (click)="imageClick()">Images</li></a>
<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,resultDisplay:'videos',mode:'text',nopagechange:'false',append:'false',fq:'url_file_ext_s%3A(avi%2BOR%2Bmov%2BOR%2Bflv%2BOR%2Bmp4)'}"><li [class.active_view]="Display('videos')" (click)="videoClick()">Videos</li></a>
<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,resultDisplay:'news',mode:'text',nopagechange:'false',append:'false'}"><li [class.active_view]="Display('news')" (click)="newsClick()">News</li></a>

Now we must navigate to a link on the click of this tabs, for this we can simply use href attribute which we earlier tried to use but there is a disadvantage of using href tag. Using href tab reloads the whole page when we try to navigate to another tab.

Therefore we will use routerLink to achieve the same functionality as Susper is a single page application, where the page should not be reloaded. routerLink navigates to the new url and the component is rendered in place of <router-outlet> without reloading the whole page.

After using routerLink, we will use queryParams attribute to pass routing parameters to a route. Since these parameters will be variable we must use string interpolation to set query parameters. We will use searchdata object and set query parameters according to its value.

Now we are done with making tabs as links. But as of now all tabs will behave a links even if it is an active tab. Therefore we need to hide the link property of any tab when it is active.

2.Hiding the link property of the active tab:

Now we will use to different elements for each tab, one as link and other as normal li element.For hiding the link elements when the tab is active we will use [hidden] attribute in angular. We will just hide the link element when the value of resultDisplay variable is similar to current tab.

<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,resultDisplay:'all',mode:'text',nopagechange:'false',append:'false'}" [hidden]="this.resultDisplay==='all'"><li [class.active_view]="Display('all')" (click)="docClick()">All</li></a>
        <li [class.active_view]="Display('all')" (click)="docClick()" *ngIf="Display('all')">All</li>

For hiding the simple li element we will use *ngIf attribute which will remove the simple li element from the DOM for un active tabs. We might have used *ngIf attribute to hide link elements but rendering a lot of elements on DOM slows down the page and affects performance.

We will follow this for rest of the tabs also and we are done with our changes.

Now we can the inactive tabs will behave as link and the active tabs will not behave as link similar to Google.

Resources

  1. Angular Routing: https://angular.io/guide/router
  2. ngIf and [hidden]: https://stackoverflow.com/questions/39777990/angular2-conditional-display-bind-to-hidden-property-vs-ngif-directive
  3. Angular queryParams: https://stackoverflow.com/a/46008187

Deploying Susper On Heroku

In Susper, currently we have two branches, master and development branch. The master branch is deployed on susper.com and we needed a deployment of development branch. For this we are using heroku and in this blog I will describe how we have deployed Susper’s development branch on heroku at http://susper-dev.herokuapp.com

Here are the steps:

1.Get a free heroku account:

To deploy our apps on heroku we must have a heroku account which can be created for free on https://www.heroku.com.

2.Create a new app on heroku:

After creating an account we need to create an app with a name and a region, for Susper we have used name susper-dev and region as United States.

3.Link the app to Susper’s development branch:

After successfully creating the app we need to link our app to Susper’s development branch and in Deploy section and then we have to enable automatic deployment from development branch after CI passes.

  1. Setup the Susper Project:

Now we have deployed our and we need to configure it so that it can successfully start on heroku. Following are the steps to configure Susper.

a) Add node and npm engine in package.json:

Now we need to tell heroku which npm and node version to use while running our app,this can be done by defining an engine field in package.json and adding npm and node versions as values as shown here:

"engines": {
   "node": "8.2.1",
   "npm": "6.0.1"
 }

 

b) Adding a postinstall field under scripts field in package.json:

Now we need to tell heroku the command that we will be using to build our app. Since on susper.com we are using ng build –prod –build-optimizer that builds our app for production by optimizing the build artifacts. Therefore on heroku also we will be using the same command:

"postinstall": "ng build --prod --build-optimizer"

 

c) Defining dependencies and typescript version for our project:This can be done by defining the @angular/cli ,@angular/compiler-cli and typescript and their version under dependencies field in package.json. In Susper we are using the following dependency versions.

"@angular/cli": "^1.1.0",
"@angular/compiler": "4.1.3",
"typescript": "~2.3.4"

 

d) Creating a javascript file to install express server and run our app: Locally when we run ng serve angular cli automatically creates a express server where our app is deployed locally but on heroku we will be required to start our express server which will be used by our app using a javascript file. Here is the code for starting the server.

//Install the server
const express = require('express');
const path = require('path');
const app = express();
// Serving the static file from dist folder
app.use(express.static(__dirname + '/dist'));
app.get('/*', function(request,response) {
response.sendFile(path.join(__dirname+'/dist/index.html'));
});
// Starting the app and listening on default heroku port.
app.listen(process.env.PORT || 8080);

 

Now we need to run this file, for this we will change start command in package.json file under script to start this file.

"start": "node server.js"

 

Now everytime a new commit is made to development branch our app will be automatically deployed on heroku at https://susper-dev.herokuapp.com

Resources

 

 

Displaying Charts on Lightbox

Susper displayed charts which shows graphical representation of Result Frequency and Protocol Distribution(http or https). But since results from these charts added little value it was decided to display charts on a click of a button on a lightbox. (Issue: https://github.com/fossasia/susper.com/issues/1066 ). In this blog,I will describe how I have implemented charts on lightbox.

Uniting the graphs(Line and Bar Graph) in a same class:

Before beginning the implementation of lightbox, we should unite all the components that will be displayed on lightbox in a single class.

Here I have put the different types of charts in a class named graph.

<div class="graph large"> <div class="linegraph"><canvas baseChart
           [datasets]="lineChartData"
           [labels]="lineChartLabels"
           [options]="lineChartOptions"
           [chartType]="lineChartType"
           [colors]="lineChartColors"
></canvas></div><div class="bargraph"><canvas baseChart
           [datasets]="barChartData"
           [labels]="barChartLabels"
           [options]="barChartOptions"
           [chartType]="barChartType"
           [colors]="lineChartColors"
   ></canvas></div></div>

Implementing button to toggle display of light box:

To show and hide lightbox along with charts we must have a button.

Clicking on which should trigger a function which will internally execute the logic in typescript file to show and hide the lightbox.

<button class="btn" id="toggle-chart-button" (click)="BoxToggle()">
           {{analyticsStatus}} Analytics
 </button>

Creating a translucent black background:

Now to have a black overlay for lightbox we need a div element and inside it we need a closing button.

Clicking on the black overlay should toggle status of light box therefore we have binded click event with BoxToggle() method.

<div id="fade" class="black_overlay" (click)='BoxToggle()'><a class="closeX" id="closeX">&#10006;</a></div>

Creating a white block to display charts:

Now after creating a black overlay in background we need a white foreground to display charts.

Therefore I have implemented a div element and put all the chart elements inside the div element so that it will be displayed on a white foreground on a translucent black background.

<div id="light" class="white_content">
<div class="graph large"><div class="linegraph"><canvas baseChart
           [datasets]="lineChartData"
           [labels]="lineChartLabels"
           [options]="lineChartOptions"
           [chartType]="lineChartType"
           [colors]="lineChartColors"
   ></canvas></div>
<div class="bargraph"><canvas baseChart
           [datasets]="barChartData"
           [labels]="barChartLabels"
           [options]="barChartOptions"
           [chartType]="barChartType"
           [colors]="lineChartColors"
   ></canvas></div></div></div>

CSS for all above elements:

All the CSS which is written for charts, toggle button, white block and black overlay

To make it look attractive and similar to market leader can be found here.

Implementing the logic in typescript file:

Now we have created all the elements in frontend to display the lightbox but we need a little logic to show and hide the lightbox on clicking the button. For this I have used a variable analyticsStatus which is a string and is initialised with value ‘Show Chart’. Its value is displayed on the toggle button. I have used this variable with BoxToggle() function for implementing the logic for showing and hiding the lightbox.

This function checks the status of message and toggles it along with toggling lightbox with some javascript code. The code for the function is here.

BoxToggle() { if (this.analyticsStatus === 'Show Chart') {
     this.analyticsStatus = 'Hide Chart';
     document.getElementById('light').style.display = 'block';
     document.getElementById('fade').style.display = 'block';
   } else {
     this.analyticsStatus = 'Show Chart';
     document.getElementById('light').style.display = 'none';
     document.getElementById('fade').style.display = 'none'; }}

Resources

1.Creating a lightbox: https://www.emanueleferonato.com/2007/08/22/create-a-lightbox-effect-only-with-css-no-javascript-needed/

2.W3School lightbox: https://www.w3schools.com/howto/howto_js_lightbox.asp

3.Angular 2 Lightbox: https://lokeshdhakar.com/projects/lightbox2/

Extending the News Feature to Show results from multiple organisations

News Tab in Susper was earlier implemented to show results only from a single organisation using  site: modifier for query facet. In this blog I will discuss about how I have modified the current News Tab to show results from various News organisation like BBC, Al Jazeera, The Guardian etc.

Implementation:

Step 1:

Creating a JSON file to store organisations detail:

We need to decide from which organisations  we will fetch results using YaCy Server and then display it in Susper. We have provided a JSON file where user can add or delete News sources easily. The results will only limited to the organisations which are present in the JSON file.

Step 2:

Creating a service to fetch details from JSON file:

Now after creating the the JSON file we need a service which will fetch results from the JSON file according to our need. This service will be a simple  Angular Service having a class GetJsonService and it will access the newsFile.json and map the results in JSON format. Here is the service which does this task for us.

Step 3:

Creating a service to fetch news accordingly:

Now after fetching the JSON result we need a service to fetch the News Results from YaCy Server. I have created a separate service to do this task where I have fetched results using site: modifier from each organisation and returned the results.The code for the news.service.ts is below.

export class NewsService {
 constructor(private jsonp: Jsonp) { }
 getSearchResults(searchquery, org) {
   let searchURL = 'https://yacy.searchlab.eu/solr/select?query=';
   searchURL += searchquery.query + ' site:' + org;
   let params = new URLSearchParams();
   for (let key in searchquery) {
     if (searchquery.hasOwnProperty(key)) {
       params.set(key, searchquery[key]); } }
  //Set other parameters
   return this.jsonp
     .get(searchURL, {search: params}).map(res =>
       res.json()[0]
     ).catch(this.handleError); }

 

Step 4:

Updating the results section:

Now we have a service that gives results from a single organisation and a JSON list of organisations. Now in results.component.ts we can simply subscribe to getJsonService and in a loop we will call getNewsService by changing the organisation in every iteration. We will then check that whether we are getting the valid results or not (undefined). The results which are not valid can cause errors when we will try to read any field of an undefined variable. Then, We will simply append the 2 result items from each organisation in an empty array and later use this array to show results.

this.getJsonService.getJSON().subscribe(res => { this.newsResponse = [];
for (let i = 0; i < res.newsOrgs.length; i++) { this.getNewsService.getSearchResults(querydata,res.newsOrgs[i].provider).subscribe( response => {
    if (response.channels[0].items[0] !== undefined) {
    this.newsResponse.push(response.channels[0].items[0]); }
    if (response.channels[0].items[1] !== undefined) {
    this.newsResponse.push(response.channels[0].items[1]); } } ); }
    });

 

The newsClick() function is activated on clicking News Tab and it updates the query and its details in store.

Step 5

Displaying the results:

Now we will modify the results.component.html to show results from new newsResponse array which have 2 results each from 5 organisations.

For this we will use each item of newsResponse using *ngFor and display its title,link and description in html template. We will also use [style.color] property of our element and set the color according to theme.

<div *ngFor="let item of newsResponse" class="result"> <div class="title">
<a class="title-pointer" href="{{item.link}}" [style.color]="themeService.titleColor">{{item.title}}</a>
</div> <div class="link">
<p  [style.color]="themeService.linkColor">{{item.link}}</p>
 </div> </div>

 

Here is the view of Susper’s News Tab where we are getting results from 5 different organisations.

Resources

  1. YaCy Modifiers: http://www.yacy-websuche.de/wiki/index.php/En:SearchParameters
  1. Angular Services: https://angular.io/tutorial/toh-pt4
  2. Reading JSON data in Angular: https://stackoverflow.com/questions/43275995/angular4-how-do-access-local-json

 

Adding News Tab in Susper

Most of the current search engines have a News tab where results are displayed which are fetched from different News Organisations. The latest results are displayed on top followed by the older ones. Current market leader, Google even uses AI and Machine Learning to analyze the contents before delivering it as results to the user. In this blog, I will describe how I have implemented a basic News Tab in Susper which show news results from dailymail.co.uk . This News tab can be improved in future to match the market leader.

Implementation of News tab in Susper:

Implementation of News tab in UI:

To implement News tab the current design pattern has been followed in Susper. Here is the code to show News tab on the results page.

<ul type="none" id="search-options">
       <li [class.active_view]="Display('news')" (click)="newsClick()">News</li>

Angular attribute binding is used to bind the active_view property with the “Display(‘news’)” function and the click event is associated with “newsClick()” function.

Here is the code for newsClick() function which filters the result using ‘site:’ parameter of yacy and then dispatches the ‘urldata’ object to ‘QueryServerAction(urldata)’ function in Query Action.

newsClick() {
   let urldata = Object.assign({}, this.searchdata);
   this.getPresentPage(1);
   this.resultDisplay = 'news';
   delete urldata.fq;
   urldata.rows = 10;
   if (urldata.query.substr(urldata.query.length - 25, 25) !== " site:www.dailymail.co.uk") {
     urldata.query += " site:www.dailymail.co.uk";
   } else {
     urldata.query += "";
   }
   urldata.resultDisplay = this.resultDisplay;
   this.store.dispatch(new queryactions.QueryServerAction(urldata));
   }

Here is the code for Query Action in query.ts file. It has two actions QueryAction and QueryServerAction

export const ActionTypes = {
 QUERYCHANGE: type('[Query] Change'),
 QUERYSERVER: type('[Query] Server'),
};
export class QueryAction implements Action {
 type = ActionTypes.QUERYCHANGE;

 constructor(public payload: any) {}
}
export class QueryServerAction implements Action {
 type = ActionTypes.QUERYSERVER;
 constructor(public payload: any) {}
}
export type Actions
 = QueryAction|QueryServerAction ;

Now the reducer takes either of the two query actions and the current state and returns the new state to the store.

Here is the code for the reducer query.ts

export const CHANGE = 'CHANGE';
export interface State {
 query: string;
 wholequery: any;
}
const initialState: State = {
 query: '',
 wholequery: {
   query: '',
   rows: 10,
   start: 0,
   mode: 'text'
 },
};
export function reducer(state: State = initialState, action: query.Actions): State {
 switch (action.type) {
   case query.ActionTypes.QUERYCHANGE: {
     const changeQuery = action.payload;
     return Object.assign({}, state, {
       query: changeQuery,
       wholequery: state.wholequery
     });
   }
   case query.ActionTypes.QUERYSERVER: {
     let serverQuery = Object.assign({}, action.payload);
     let resultCount = 10;
     if (localStorage.getItem('resultscount')) {
       resultCount = JSON.parse(localStorage.getItem('resultscount')).value || 10;
     }
     let instantsearch = JSON.parse(localStorage.getItem('instantsearch'));

     if (instantsearch && instantsearch.value) {
       resultCount = 10;
     }

     serverQuery.rows = resultCount;
     return Object.assign({}, state, {
       wholequery: serverQuery,
       query: state.query
     });
   }
   default: {
     return state;
   }
 }
}
export const getpresentquery = (state: State) => state.query;
export const getpresentwholequery = (state: State) => state.wholequery;

 

From the store the modified query is available to all other components of the appand is used by search service to get the filtered results.

The search results are then stored in observable items$ in results.components.ts and is then used in results.components.html to display results under News tab.

Here is the code to display the results.

 <div class="feed container">
     <div *ngFor="let item of items$|async" class="result">
       <div class="title">
         <a class="title-pointer" href="{{item.link}}">{{item.title}}</a>
       </div>
       <div class="link">
         <p>{{item.link}}</p>
       </div>
     </div>
   </div>

 

Resources

1.YaCy Search Parameters: http://www.yacywebsearch.net/wiki/index.php/En:SearchParameters

2.Redux in Angular: http://blog.ng-book.com/introduction-to-redux-with-typescript-and-angular-2/

3.Corresponding PR: https://github.com/fossasia/susper.com/pull/1023

Deploying Angular App Susper on GitHub Pages

What are the issues we had with automatic deployment of Susper?

SUSPER project which is built on Angular is kept on master branch of the repository .

Whenever any PR is merged,Travis CI does all the builds, and checks for any error. After successful checking it triggers deployment script in SUSPER (deploy.sh) .

Here is the deployment script:

#!/bin/bash

# SOURCE_BRANCH & TARGET_BRANCH stores the name of different susper.com github branches.
SOURCE_BRANCH="master"
TARGET_BRANCH="gh-pages"

# Pull requests and commits to other branches shouldn't try to deploy.
if [ "$TRAVIS_PULL_REQUEST" != "false" -o "$TRAVIS_BRANCH" != "$SOURCE_BRANCH" ]; then
    echo "Skipping deploy; The request or commit is not on master"
    exit 0
fi

# Store some useful information into variables
REPO=`git config remote.origin.url`
SSH_REPO=${REPO/https:\/\/github.com\//[email protected]:}
SHA=`git rev-parse --verify HEAD`

# Decryption of the `deploy.enc`
openssl aes-256-cbc -k "$super_secret_password" -in deploy.enc -out deploy_key -d

# give `deploy_key`. the permission to read and write
chmod 600 deploy_key
eval `ssh-agent -s`
ssh-add deploy_key

# Cloning the repository to repo/ directory,
# Creating gh-pages branch if it doesn't exists else moving to that branch
git clone $REPO repo
cd repo
git checkout $TARGET_BRANCH || git checkout --orphan $TARGET_BRANCH
cd ..

# Setting up the username and email.
git config user.name "Travis CI"
git config user.email "$COMMIT_AUTHOR_EMAIL"

# Cleaning up the old repo's gh-pages branch except CNAME file and 404.html
find repo/* ! -name "CNAME" ! -name "404.html" -maxdepth 1  -exec rm -rf {} \; 2> /dev/null
cd repo

# commit the changes, move to SOURCE_BRANCH
git add --all
git commit -m "Travis CI Clean Deploy : ${SHA}"

git checkout $SOURCE_BRANCH

# Actual building and setup of current push or PR. Move to `TARGET_BRANCH` and move the `dist` folder
npm install
ng build --prod --aot
mv susper.xml dist/
mv 404.html dist/

git checkout $TARGET_BRANCH
mv dist/* .

# Staging the new build for commit; and then committing the latest build
git add .
git commit --amend --no-edit --allow-empty

# Actual push to gh-pages branch via Travis
git push --force $SSH_REPO $TARGET_BRANCH

This script starts after successfully checking the project (comments shows working of each command). The repository is cleaned,except some files like CNAME and 404.html and a commit is made. Then SUSPER’s build artifacts are generated in dist folder and all the files from dist folder are moved to gh-pages and the changes are amended. GitHub Pages engine then uses the build artifacts to generate static site at susper.com. But we faced a weird problem when we were unable to see changes from latest commits on susper.com. Slowly number of commits increased but still changes were not reflecting on susper.com .

What was the error in deployment of SUSPER?

All the changes were getting committed to the gh-pages branch properly. But the GitHub Pages engine was unable to process the branch to build static site. Due to this the site was not getting updated. The failing builds from gh-engine are notified to the owner via email.

The failing builds from gh-pages can be seen here https://github.com/fossasia/susper.com/commits/gh-pages

Between 21 May to 18 March.

There was a weird error ( Failure: The tag `chartjs` in line 4 on `node_modules/chart.js/docs/charts/bar.md` is not recognized liquid tag) while building the site from build artifacts. There was an issue with `gh-pages` engine as `node_modules/chart.js/docs/charts/bar.md`  was previously also present but then there was no such errors than.

Due to this error all commits which were made to SUSPER repository was not shown on susper.com as the site was not getting updated.

How we solved it?

We tried to search a lot about the above mentioned error but we were unable to find a proper solution. While searching we came across this question https://stackoverflow.com/questions/24713112/my-github-page-wont-update-its-content

on StackOverflow. There was no accepted answer for this question and every answer was different from other. So, we just reproduced the bug in our own repository by just making a similar repository to SUSPER with same branches. Once we successfully reproduced the bug. We tried to correlate all the answers to arrive at a solution to fix this bug. Luckily we found that any commit with no significant change from previous commit removed the error and allowed the gh-pages engine to successfully create the site. So we made a dummy commit in which we created a hidden file with no content and pushed it to gh-pages. This did the trick for us and gh-pages was now able build the site for us.

The whole problem was due to a bug in GitHub pages engine which got fixed by a dummy commit.

Resources

  1. Stackoverflow:https://stackoverflow.com/questions/24713112/my-github-page-wont-update-its-content
  2. Publishing to GitHub Pages:https://help.github.com/articles/configuring-a-publishing-source-for-github-pages/
  3. Commits(gh-pages):https://github.com/fossasia/susper.com/commits/gh-pages

Adding Yarn as new Dependency Manager  along with NPM in Susper

Dependency managers are software modules that coordinate the integration of external libraries or packages into larger application stack. Dependency managers use configuration files like composer.json, package.json, build.gradle or pom.xml to determine: What dependency to get, What version of the dependency in particular and, Which repository to get them from. Currently SUSPER has only NPM as a dependency manager which is used to install all dependencies. In this blog, I will describe how we have added facebook’s Yarn as a new dependency manager in Susper

Lets checkout Yarn in detail:

Yarn is a fast and good alternative to NPM. One of the great advantages of Yarn is that while remaining compatible with the npm registry, it replaces the workflow for npm client or other package managers Yarn was created by Facebook, to solve some particular problems that were faced while using NPM. Yarn was developed to deal with inconsistency in dependency installation while scaling and to increase speed.

What is advantages of using Yarn?

  • Improving Network performance:Queuing up the requests and avoiding requests waterfalls helps to maximize network utilization.
  • Checks Package Integrity:Package integrity is checked after each install to avoid corrupt packages installation.
  • Checks Package Integrity:Package integrity is checked after each install to avoid corrupt packages installation.
  • Caching: Yarn helps to install the dependencies without an internet connection if the dependency has been previously installed on the system. This is done by caching.
  • Lock File: Lock files are used to make sure that the node_modules directory has the exact same structure on all development environments.

Source: https://yarnpkg.com/en/

How Yarn is installed along with NPM in SUSPER?

Installing Yarn is super easy. Here are the steps to setup Yarn along with NPM and begin using it as dependency manager.

On Debian or Ubuntu Linux, we can install Yarn via our Debian package repository. We will first need to configure the repository:

curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee 

/etc/apt/sources.list.d/yarn.list

 

Then simply use:

sudo apt-get update && sudo apt-get install yarn

 

Note: Ubuntu 17.04 comes with cmdtest installed by default. If anyone gets any errors from installing yarn, then remove it by sudo apt remove cmdtest first. Refer to this for more information.

If using nvm you can avoid the node installation by doing:

sudo apt-get install --no-install-recommends yarn

 

Test that Yarn is installed by running:

yarn --version

 

Now delete the node_modules folder so that all dependencies installed by npm is removed.

Now use yarn command in project’s repository.

yarn 

 

Wait while dependencies are installed and then we will be done.

What is happening ?

Yarn has created a lock file  yarn.lock. After each operation the file is updated (installing, updating or removing packages) to keep the track of exact package version. If kept in our Git repository we can see that the exact same result in node_modules is made available to all systems.

Resources

  1. Yarn: https://yarnpkg.com/en/
  2. Announcement of Yarn: https://code.facebook.com/posts/1840075619545360
  3. Yarn Vs NPM: https://stackoverflow.com/questions/40027819/when-to-use-yarn-over-npm-what-are-the-differences