Getting results for Multi-word Query and showing limited results in Knowledge Graph

In Susper knowledge graph,we were unable to get results for multi word query from Wikipedia API and also we were unable to decide how much information should be shown in knowledge graph which was retrieved from Wikipedia API.For example for a query donald trump we do not get information (https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=donald%20trump) . Also for searching for any query like india (https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=india) we get a lot of information. Earlier we used Angular slice pipe to display only 600 characters in knowledge graph (Infobox) but almost for all queries the sentences were terminated before it was completed. In this blog, I will describe how we solved both problems in knowledge graph.

Getting results for multi word query from Wikipedia API:

On searching a lot we found that the results were present on Wikipedia for multi word queries but these queries must have its starting letters as capital letters. For example we do not get results for queries such as donald trump

i.e https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=donald%20trump

but when we make a query for Donald Trump  

i.e

https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=Donald%20Trump  

But since in most of the search queries users use small letters, we need to convert the query such that each word starts with a capital letter. Since the Knowledge Graph has been implemented using ngrx pattern, this logic was easily implemented in knowledge effects. For this we will be selecting each word in query by using regular expression and then we will use toUppercase() and toLowerCase() methods of javascript and will capitalize every word of the query.

toTitleCase(str) {
    return str.replace(
        /\w\S*/g,
        function(txt) {
            return txt.charAt(0).toUpperCase() + txt.substr(1).toLowerCase();
        }
    );
}

this.knowledgeservice.getSearchResults(this.toTitleCase(querypay.query))

 

Like this we now get results for almost all queries.

Limiting information in Knowledge Graph without terminating the sentences:

To solve this issue, we decided to show only four lines of data retrieved from Wikipedia API

For this we have implemented a getPosition function which will take three parameters a string, a substring and a index. Here string is the whole string from which the function will return the index of index position substring.

getPosition(string, subString, index) {
    return string.split(subString, index).join(subString).length;
 }
this.description = this.description.slice(0, this.getPosition(this.description, '.', 4) + 1);

 

Here, We get the index of 4th ‘.’ full stop by getPosition() function and then pass it to javascript slice method to slice the string upto that position.

Using this we limited results to 4 lines without terminating each line in middle.

References

1.W3Schools Regular Expression: https://www.w3schools.com/jsref/jsref_obj_regexp.asp

2.Javascript Slice method: https://www.w3schools.com/jsref/jsref_slice_array.asp

3.Wikipedia API: https://www.mediawiki.org/wiki/API:Main_page

Displaying All Providers and All Authors separately in Susper Angular App

In Susper results are retrieved from a lots of providers and authors. Displaying all authors’ and providers’ list on result page make appearance of result section odd and more complex. In this blog, I will describe how we have separated the list of all providers and authors from main result page and displayed it on a lightbox.

Steps:

  • Making a black overlay for lightbox:

First we of all to display the results on a lightbox,we to have a black overlay for this we need a div element and inside it we need a closing button.

Clicking on the black overlay should toggle status of light box therefore we have binded click event with showMore() method.

<div id="fade_analytics" class="black_overlay" (click)='showMore()'><a class="closeX" id="closeX">&#10006;</a></div>
  • Creating a white content to display all Providers or Authors:

Now we need a white block to display all Providers or all Authors so after creating a black overlay in background we need a white foreground.

Therefore I have implemented a div element and put all the chart elements inside the div element so that it will be displayed on a white foreground on a translucent black background.

<div id="light_analytics" class="white_content_analytics">
<div class="show_more"><h2 class="heading"><b>Analytics</b></h2><div id="filtersearch" *ngFor="let nav of navigation$|async; let i = index"><div class="card-container" id="relate" *ngIf="nav.displayname =='Provider' || nav.displayname =='Authors'"><h3  class="related-searches" *ngIf="i===currentelement"><b>Top {{nav.displayname}}<span *ngIf="nav.displayname.slice(-1)!=='s'">s</span></b></h3>
<div style="display: block;cursor:pointer;cursor: hand;" *ngFor="let element of  ((i != currentelement)? (nav.elements | slice:0:0) :nav.elements)">
</div></div></div></div>

The code to display All providers and All authors are contained inside the   white block and the logic to display the each section is also implemented.

  • Showing only 50 results on result page and implementing a button to open all providers or authors in light box:

Currently we are displaying the top five providers and authors and we provide a button to show more providers and authors which then displays 50 more results and then we display an interactive button that toggles display of all authors and providers on a lightbox. To display 5 and 50 results we use slice pipe in angular. The button Show All Providers/Authors is binded with showMore() function which toggles lightbox.

<div style="display: block;cursor:pointer;cursor: hand;" *ngFor="let element of ((i != selectedelement)? (nav.elements | slice:0:5) :(nav.elements | slice:0:50) )">
<a [routerLink]="['/search']" [queryParams]="changequerylook(element.modifier,element)" *ngIf="element.name" href="#" [style.color]="themeService.linkColor">
{{element.name}} <span class="badge badge-info">{{element.count}}</span>
</a><a  *ngIf="!element.name" style="color: black;cursor: default" href="#" [style.color]="themeService.linkColor">
Undefined Author <span class="badge badge-info">{{element.count}}</span></a></div>
<button class="showMoreBtn" (click)="showMore()" *ngIf="selectedelement===i" [style.color]="themeService.linkColor">Show All {{nav.displayname}}<span *ngIf="nav.displayname.slice(-1)!=='s'">s</span></button>

 

  • Implementing the logic to display all results on white content:

Now we need to implement the logic to toggle lightbox to display All Providers and All Authors. This logic is implemented mainly in showMore() function which toggles the lightbox containing the All Providers and All Authors

showMore() { this.selectedelement = -1;
if (!this.showMoreAnalytics) { this.showMoreAnalytics = true; document.getElementById('light_analytics').style.display = 'block'; document.getElementById('fade_analytics').style.display = 'block'; } else { this.showMoreAnalytics = false;
document.getElementById('light_analytics').style.display = 'none';
document.getElementById('fade_analytics').style.display = 'none';}}

Here we use showMoreAnalytics variable to check whether the user has asked to show all Providers or Authors. We also make selectedelement variable to -1 to hide the long list of 50 Providers or Authors when Lightbox displays all the Providers or Author. Finally we use document object in javascript and select the DOM elements by their id and set their display property this shows and hides the lightbox on result page.

Resources

1.YaCy Advanced Search Parameters

http://www.yacy-websuche.de/wiki/index.php/En:SearchParameters

2.W3School lightbox

https://www.w3schools.com/howto/howto_js_lightbox.asp

 

Adding New tests for Knowledge Service

Testing is done to test our application by executing some functions by creating instances of corresponding classes,executing functions and checking the actual behaviour of our app with expected result. The tools and frameworks used in Angular are Jasmine and Karma. In this blog, I will describe about how I have implemented tests for Newly Added Knowledge API service that helped us to increase overall code coverage by 1.05% in Susper .

Adding tests for Knowledge API:

We need to check the API that whether it is functioning or not and this can be done by using a mocked response (hardcoded response) for any query and then comparing this with the received response from the API. This will help us to check proper functioning of our API.

This is a common practice in Angular and to achieve this we will be using some dependencies like MockBackend, MockConnection, BaseRequestOptions provided by Angular.

import { MockBackend, MockConnection } from '@angular/http/testing';
import { Http, Jsonp, BaseRequestOptions, RequestMethod, Response, ResponseOptions, HttpModule, JsonpModule } from '@angular/http';

 

We will also need to define a Mock response, here I have used hard coded response for query India. Here is the mocked response.

export const MockKnowledgeApi = {
    results: [
        {"batchcomplete": "", "query":
        {"normalized":
        [{"from": "india", "to": "India"}], "pages":
        {"14533": {"pageid": 14533, "ns": 0, "title": "India", "extract": `India (IAST: Bh\u0101rat), also called the Republic of India (IAST: Bh\u0101rat Ga\u1e47ar\u0101jya), is a country in South Asia.
   }}}} ], MaxHits : 5 };

 

Now we will use a Mock constant mock_Http_provider for http and will inject instances of MockBackend and BaseRequestOptions.

const mockHttp_provider = {
  provide: Http,
  deps: [MockBackend, BaseRequestOptions],
  useFactory: (backend: MockBackend, options: BaseRequestOptions) => { return new Http(backend, options); } };

 

Now we need to add all the services and dependencies which we will be using in providers and we will inject the instances of Knowledgeapi Service and MockBackend in beforeEach function.

beforeEach(inject([KnowledgeapiService, MockBackend], (knowledgeService: KnowledgeapiService, mockBackend: MockBackend) => { service = knowledgeService;
    backend = mockBackend;}));

 

Now we will use the same query for which we have created the mocked response and we will be checking the response of from our API.

const searchquery = 'india';
connection.mockRespond(new Response(options));
      expect(connection.request.method).toEqual(RequestMethod.Get);
      expect(connection.request.url).toBe(   `https://en.wikipedia.org/w/api.php?&origin=*&format=json&action=query&prop=extracts&exintro=&explaintext=&` +
                    `titles=${searchquery}`);});
service.getSearchResults(searchquery).subscribe((res) => {
      expect(res).toEqual(MockKnowledgeApi);});

 

This will check the working of our API and if it is working then our test case will pass.

Like this we have implemented the tests for Knowledgeapi Service which helped us to test our API and increase overall code coverage significantly.

Resources

  1. Testing in Angular:https://angular.io/guide/testing
  2. Testing by Mockbackend:https://angular.io/api/http/testing/MockBackend
  3. Using MockBackend to simulate response: https://codecraft.tv/courses/angular/unit-testing/http-and-jsonp/#_using_the_code_mockbackend_code_to_simulate_a_response

Using routerLink to make result tabs as links

In Susper, it was required that different result sections i.e All, Images,Videos and News should behave as links. Links provide us an advantage of opening the different result tabs in a New Tab by right clicking on it. Also at the same time we wanted that Active tabs must not behave as links. We wanted that tabs should behave in a similar way as Google, Bing and other search engine.In this blog, I will describe how we have achieved this by using routerLink and [hidden] attribute in Angular.

Steps:

1.Adding routerLink attribute in anchor tag.

Now, To make any element as link we must enclose it in anchor tag, therefore we will use <a> tag enclosing the <li> tag to make it a link.

<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,resultDisplay:'all',mode:'text',nopagechange:'false',append:'false'}"><li [class.active_view]="Display('all')" (click)="docClick()">All</li></a>
<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,mode:'text',nopagechange:'false',append:'false',fq:'url_file_ext_s%3A(png%2BOR%2Bjpeg%2BOR%2Bjpg%2BOR%2Bgif)',resultDisplay:'images'}"><li [class.active_view]="Display('images')" (click)="imageClick()">Images</li></a>
<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,resultDisplay:'videos',mode:'text',nopagechange:'false',append:'false',fq:'url_file_ext_s%3A(avi%2BOR%2Bmov%2BOR%2Bflv%2BOR%2Bmp4)'}"><li [class.active_view]="Display('videos')" (click)="videoClick()">Videos</li></a>
<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,resultDisplay:'news',mode:'text',nopagechange:'false',append:'false'}"><li [class.active_view]="Display('news')" (click)="newsClick()">News</li></a>

Now we must navigate to a link on the click of this tabs, for this we can simply use href attribute which we earlier tried to use but there is a disadvantage of using href tag. Using href tab reloads the whole page when we try to navigate to another tab.

Therefore we will use routerLink to achieve the same functionality as Susper is a single page application, where the page should not be reloaded. routerLink navigates to the new url and the component is rendered in place of <router-outlet> without reloading the whole page.

After using routerLink, we will use queryParams attribute to pass routing parameters to a route. Since these parameters will be variable we must use string interpolation to set query parameters. We will use searchdata object and set query parameters according to its value.

Now we are done with making tabs as links. But as of now all tabs will behave a links even if it is an active tab. Therefore we need to hide the link property of any tab when it is active.

2.Hiding the link property of the active tab:

Now we will use to different elements for each tab, one as link and other as normal li element.For hiding the link elements when the tab is active we will use [hidden] attribute in angular. We will just hide the link element when the value of resultDisplay variable is similar to current tab.

<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,resultDisplay:'all',mode:'text',nopagechange:'false',append:'false'}" [hidden]="this.resultDisplay==='all'"><li [class.active_view]="Display('all')" (click)="docClick()">All</li></a>
        <li [class.active_view]="Display('all')" (click)="docClick()" *ngIf="Display('all')">All</li>

For hiding the simple li element we will use *ngIf attribute which will remove the simple li element from the DOM for un active tabs. We might have used *ngIf attribute to hide link elements but rendering a lot of elements on DOM slows down the page and affects performance.

We will follow this for rest of the tabs also and we are done with our changes.

Now we can the inactive tabs will behave as link and the active tabs will not behave as link similar to Google.

Resources

  1. Angular Routing: https://angular.io/guide/router
  2. ngIf and [hidden]: https://stackoverflow.com/questions/39777990/angular2-conditional-display-bind-to-hidden-property-vs-ngif-directive
  3. Angular queryParams: https://stackoverflow.com/a/46008187

Deploying Susper On Heroku

In Susper, currently we have two branches, master and development branch. The master branch is deployed on susper.com and we needed a deployment of development branch. For this we are using heroku and in this blog I will describe how we have deployed Susper’s development branch on heroku at http://susper-dev.herokuapp.com

Here are the steps:

1.Get a free heroku account:

To deploy our apps on heroku we must have a heroku account which can be created for free on https://www.heroku.com.

2.Create a new app on heroku:

After creating an account we need to create an app with a name and a region, for Susper we have used name susper-dev and region as United States.

3.Link the app to Susper’s development branch:

After successfully creating the app we need to link our app to Susper’s development branch and in Deploy section and then we have to enable automatic deployment from development branch after CI passes.

  1. Setup the Susper Project:

Now we have deployed our and we need to configure it so that it can successfully start on heroku. Following are the steps to configure Susper.

a) Add node and npm engine in package.json:

Now we need to tell heroku which npm and node version to use while running our app,this can be done by defining an engine field in package.json and adding npm and node versions as values as shown here:

"engines": {
   "node": "8.2.1",
   "npm": "6.0.1"
 }

 

b) Adding a postinstall field under scripts field in package.json:

Now we need to tell heroku the command that we will be using to build our app. Since on susper.com we are using ng build –prod –build-optimizer that builds our app for production by optimizing the build artifacts. Therefore on heroku also we will be using the same command:

"postinstall": "ng build --prod --build-optimizer"

 

c) Defining dependencies and typescript version for our project:This can be done by defining the @angular/cli ,@angular/compiler-cli and typescript and their version under dependencies field in package.json. In Susper we are using the following dependency versions.

"@angular/cli": "^1.1.0",
"@angular/compiler": "4.1.3",
"typescript": "~2.3.4"

 

d) Creating a javascript file to install express server and run our app: Locally when we run ng serve angular cli automatically creates a express server where our app is deployed locally but on heroku we will be required to start our express server which will be used by our app using a javascript file. Here is the code for starting the server.

//Install the server
const express = require('express');
const path = require('path');
const app = express();
// Serving the static file from dist folder
app.use(express.static(__dirname + '/dist'));
app.get('/*', function(request,response) {
response.sendFile(path.join(__dirname+'/dist/index.html'));
});
// Starting the app and listening on default heroku port.
app.listen(process.env.PORT || 8080);

 

Now we need to run this file, for this we will change start command in package.json file under script to start this file.

"start": "node server.js"

 

Now everytime a new commit is made to development branch our app will be automatically deployed on heroku at https://susper-dev.herokuapp.com

Resources

 

 

Extending the News Feature to Show results from multiple organisations

News Tab in Susper was earlier implemented to show results only from a single organisation using  site: modifier for query facet. In this blog I will discuss about how I have modified the current News Tab to show results from various News organisation like BBC, Al Jazeera, The Guardian etc.

Implementation:

Step 1:

Creating a JSON file to store organisations detail:

We need to decide from which organisations  we will fetch results using YaCy Server and then display it in Susper. We have provided a JSON file where user can add or delete News sources easily. The results will only limited to the organisations which are present in the JSON file.

Step 2:

Creating a service to fetch details from JSON file:

Now after creating the the JSON file we need a service which will fetch results from the JSON file according to our need. This service will be a simple  Angular Service having a class GetJsonService and it will access the newsFile.json and map the results in JSON format. Here is the service which does this task for us.

Step 3:

Creating a service to fetch news accordingly:

Now after fetching the JSON result we need a service to fetch the News Results from YaCy Server. I have created a separate service to do this task where I have fetched results using site: modifier from each organisation and returned the results.The code for the news.service.ts is below.

export class NewsService {
 constructor(private jsonp: Jsonp) { }
 getSearchResults(searchquery, org) {
   let searchURL = 'https://yacy.searchlab.eu/solr/select?query=';
   searchURL += searchquery.query + ' site:' + org;
   let params = new URLSearchParams();
   for (let key in searchquery) {
     if (searchquery.hasOwnProperty(key)) {
       params.set(key, searchquery[key]); } }
  //Set other parameters
   return this.jsonp
     .get(searchURL, {search: params}).map(res =>
       res.json()[0]
     ).catch(this.handleError); }

 

Step 4:

Updating the results section:

Now we have a service that gives results from a single organisation and a JSON list of organisations. Now in results.component.ts we can simply subscribe to getJsonService and in a loop we will call getNewsService by changing the organisation in every iteration. We will then check that whether we are getting the valid results or not (undefined). The results which are not valid can cause errors when we will try to read any field of an undefined variable. Then, We will simply append the 2 result items from each organisation in an empty array and later use this array to show results.

this.getJsonService.getJSON().subscribe(res => { this.newsResponse = [];
for (let i = 0; i < res.newsOrgs.length; i++) { this.getNewsService.getSearchResults(querydata,res.newsOrgs[i].provider).subscribe( response => {
    if (response.channels[0].items[0] !== undefined) {
    this.newsResponse.push(response.channels[0].items[0]); }
    if (response.channels[0].items[1] !== undefined) {
    this.newsResponse.push(response.channels[0].items[1]); } } ); }
    });

 

The newsClick() function is activated on clicking News Tab and it updates the query and its details in store.

Step 5

Displaying the results:

Now we will modify the results.component.html to show results from new newsResponse array which have 2 results each from 5 organisations.

For this we will use each item of newsResponse using *ngFor and display its title,link and description in html template. We will also use [style.color] property of our element and set the color according to theme.

<div *ngFor="let item of newsResponse" class="result"> <div class="title">
<a class="title-pointer" href="{{item.link}}" [style.color]="themeService.titleColor">{{item.title}}</a>
</div> <div class="link">
<p  [style.color]="themeService.linkColor">{{item.link}}</p>
 </div> </div>

 

Here is the view of Susper’s News Tab where we are getting results from 5 different organisations.

Resources

  1. YaCy Modifiers: http://www.yacy-websuche.de/wiki/index.php/En:SearchParameters
  1. Angular Services: https://angular.io/tutorial/toh-pt4
  2. Reading JSON data in Angular: https://stackoverflow.com/questions/43275995/angular4-how-do-access-local-json

 

Adding News Tab in Susper

Most of the current search engines have a News tab where results are displayed which are fetched from different News Organisations. The latest results are displayed on top followed by the older ones. Current market leader, Google even uses AI and Machine Learning to analyze the contents before delivering it as results to the user. In this blog, I will describe how I have implemented a basic News Tab in Susper which show news results from dailymail.co.uk . This News tab can be improved in future to match the market leader.

Implementation of News tab in Susper:

Implementation of News tab in UI:

To implement News tab the current design pattern has been followed in Susper. Here is the code to show News tab on the results page.

<ul type="none" id="search-options">
       <li [class.active_view]="Display('news')" (click)="newsClick()">News</li>

Angular attribute binding is used to bind the active_view property with the “Display(‘news’)” function and the click event is associated with “newsClick()” function.

Here is the code for newsClick() function which filters the result using ‘site:’ parameter of yacy and then dispatches the ‘urldata’ object to ‘QueryServerAction(urldata)’ function in Query Action.

newsClick() {
   let urldata = Object.assign({}, this.searchdata);
   this.getPresentPage(1);
   this.resultDisplay = 'news';
   delete urldata.fq;
   urldata.rows = 10;
   if (urldata.query.substr(urldata.query.length - 25, 25) !== " site:www.dailymail.co.uk") {
     urldata.query += " site:www.dailymail.co.uk";
   } else {
     urldata.query += "";
   }
   urldata.resultDisplay = this.resultDisplay;
   this.store.dispatch(new queryactions.QueryServerAction(urldata));
   }

Here is the code for Query Action in query.ts file. It has two actions QueryAction and QueryServerAction

export const ActionTypes = {
 QUERYCHANGE: type('[Query] Change'),
 QUERYSERVER: type('[Query] Server'),
};
export class QueryAction implements Action {
 type = ActionTypes.QUERYCHANGE;

 constructor(public payload: any) {}
}
export class QueryServerAction implements Action {
 type = ActionTypes.QUERYSERVER;
 constructor(public payload: any) {}
}
export type Actions
 = QueryAction|QueryServerAction ;

Now the reducer takes either of the two query actions and the current state and returns the new state to the store.

Here is the code for the reducer query.ts

export const CHANGE = 'CHANGE';
export interface State {
 query: string;
 wholequery: any;
}
const initialState: State = {
 query: '',
 wholequery: {
   query: '',
   rows: 10,
   start: 0,
   mode: 'text'
 },
};
export function reducer(state: State = initialState, action: query.Actions): State {
 switch (action.type) {
   case query.ActionTypes.QUERYCHANGE: {
     const changeQuery = action.payload;
     return Object.assign({}, state, {
       query: changeQuery,
       wholequery: state.wholequery
     });
   }
   case query.ActionTypes.QUERYSERVER: {
     let serverQuery = Object.assign({}, action.payload);
     let resultCount = 10;
     if (localStorage.getItem('resultscount')) {
       resultCount = JSON.parse(localStorage.getItem('resultscount')).value || 10;
     }
     let instantsearch = JSON.parse(localStorage.getItem('instantsearch'));

     if (instantsearch && instantsearch.value) {
       resultCount = 10;
     }

     serverQuery.rows = resultCount;
     return Object.assign({}, state, {
       wholequery: serverQuery,
       query: state.query
     });
   }
   default: {
     return state;
   }
 }
}
export const getpresentquery = (state: State) => state.query;
export const getpresentwholequery = (state: State) => state.wholequery;

 

From the store the modified query is available to all other components of the appand is used by search service to get the filtered results.

The search results are then stored in observable items$ in results.components.ts and is then used in results.components.html to display results under News tab.

Here is the code to display the results.

 <div class="feed container">
     <div *ngFor="let item of items$|async" class="result">
       <div class="title">
         <a class="title-pointer" href="{{item.link}}">{{item.title}}</a>
       </div>
       <div class="link">
         <p>{{item.link}}</p>
       </div>
     </div>
   </div>

 

Resources

1.YaCy Search Parameters: http://www.yacywebsearch.net/wiki/index.php/En:SearchParameters

2.Redux in Angular: http://blog.ng-book.com/introduction-to-redux-with-typescript-and-angular-2/

3.Corresponding PR: https://github.com/fossasia/susper.com/pull/1023

Adding Yarn as new Dependency Manager  along with NPM in Susper

Dependency managers are software modules that coordinate the integration of external libraries or packages into larger application stack. Dependency managers use configuration files like composer.json, package.json, build.gradle or pom.xml to determine: What dependency to get, What version of the dependency in particular and, Which repository to get them from. Currently SUSPER has only NPM as a dependency manager which is used to install all dependencies. In this blog, I will describe how we have added facebook’s Yarn as a new dependency manager in Susper

Lets checkout Yarn in detail:

Yarn is a fast and good alternative to NPM. One of the great advantages of Yarn is that while remaining compatible with the npm registry, it replaces the workflow for npm client or other package managers Yarn was created by Facebook, to solve some particular problems that were faced while using NPM. Yarn was developed to deal with inconsistency in dependency installation while scaling and to increase speed.

What is advantages of using Yarn?

  • Improving Network performance:Queuing up the requests and avoiding requests waterfalls helps to maximize network utilization.
  • Checks Package Integrity:Package integrity is checked after each install to avoid corrupt packages installation.
  • Checks Package Integrity:Package integrity is checked after each install to avoid corrupt packages installation.
  • Caching: Yarn helps to install the dependencies without an internet connection if the dependency has been previously installed on the system. This is done by caching.
  • Lock File: Lock files are used to make sure that the node_modules directory has the exact same structure on all development environments.

Source: https://yarnpkg.com/en/

How Yarn is installed along with NPM in SUSPER?

Installing Yarn is super easy. Here are the steps to setup Yarn along with NPM and begin using it as dependency manager.

On Debian or Ubuntu Linux, we can install Yarn via our Debian package repository. We will first need to configure the repository:

curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee 

/etc/apt/sources.list.d/yarn.list

 

Then simply use:

sudo apt-get update && sudo apt-get install yarn

 

Note: Ubuntu 17.04 comes with cmdtest installed by default. If anyone gets any errors from installing yarn, then remove it by sudo apt remove cmdtest first. Refer to this for more information.

If using nvm you can avoid the node installation by doing:

sudo apt-get install --no-install-recommends yarn

 

Test that Yarn is installed by running:

yarn --version

 

Now delete the node_modules folder so that all dependencies installed by npm is removed.

Now use yarn command in project’s repository.

yarn 

 

Wait while dependencies are installed and then we will be done.

What is happening ?

Yarn has created a lock file  yarn.lock. After each operation the file is updated (installing, updating or removing packages) to keep the track of exact package version. If kept in our Git repository we can see that the exact same result in node_modules is made available to all systems.

Resources

  1. Yarn: https://yarnpkg.com/en/
  2. Announcement of Yarn: https://code.facebook.com/posts/1840075619545360
  3. Yarn Vs NPM: https://stackoverflow.com/questions/40027819/when-to-use-yarn-over-npm-what-are-the-differences

Using Wikipedia API for knowledge graph in SUSPER

Knowledge Graph is way to give a brief description about search query by connecting it to a real world entity. This helps users to get information about exactly what they want. Previously Susper had a Knowledge Graph which was implemented using DBpedia API. But since DBpedia do not provide content over HTTPS connections therefore the content was blocked on susper.com and there was a need to implement the Knowledge Graph using a new API that provide contents over HTTPS. In this blog, I will describe how getting a knowledge graph was made possible using Wikipedia API.

What is Wikipedia API ?

The MediaWiki action API is a web service that provides convenient access to wiki features, data, and metadata over HTTP, via a URL usually at api.php. Clients request particular “actions” by specifying an action parameter, mainly action=query to get information.

The endpoint :

https://en.wikipedia.org/w/api.php

The format :

format=json This tells the API that we want data to be returned in JSON format.

The action :

action=query

The MediaWiki web service API implements dozens of actions and extensions implement many more; the dynamically generated API help documents all available actions on a wiki. In this case, we’re using the “query” action to get some information.

The complete API which is used in SUSPER to extract information of a query is :

https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=japan

Where titles=Search_Query, here Japan

How it is implemented in SUSPER?

For implementing it a service has been created which fetches information by setting various URL parameters. This result can be fetched by creating an instance of service and passing search query to getsearchresults(searchquery) function.

export class KnowledgeapiService {
 server = 'https://en.wikipedia.org';
 searchURL = this.server + '/w/api.php?';
 homepage = 'http://susper.com';
 logo = '../images/susper.svg';
 constructor(private http: Http,
             private jsonp: Jsonp,
             private store: Store<fromRoot.State>) {
 }
 getsearchresults(searchquery) {
   let params = new URLSearchParams();
   params.set('origin', '*');
   params.set('format', 'json');
   params.set('action', 'query');
   params.set('prop', 'extracts');
   params.set('exintro', '');
   params.set('explaintext', '');
   params.set('titles', searchquery);
   let headers = new Headers({ 'Accept': 'application/json' });
   let options = new RequestOptions({ headers: headers, search: params });
   return this.http
     .get(this.searchURL, options).map(res =>
         res.json().query.pages
     ).catch(this.handleError);
}

 

Since the result obtained is an observable therefore we have to subscribe for it and then extract information to local variables in infobox.component.ts file.

export class InfoboxComponent implements OnInit {
 public title: string;
 public description: string;
 query$: any;
 resultsearch = '/search';
 constructor(private knowledgeservice: KnowledgeapiService,
             private route: Router,
             private activatedroute: ActivatedRoute,
             private store: Store<fromRoot.State>,
             private ref: ChangeDetectorRef) {
   this.query$ = store.select(fromRoot.getquery);
   this.query$.subscribe( query => {
     if (query) {
       this.knowledgeservice.getsearchresults(query).subscribe(res => {
         const pageId = Object.keys(res)[0];
         if (res[pageId].extract) {
           this.title = res[pageId].title;
           this.description = res[pageId].extract;
         } else {
           this.title = '';
           this.description = '';
         }
       });
     }
   });
 }

The variable title and description are used to display results on results page.

<div *ngIf=“this.description” class=“card”>
  <div>
    <h2><b>{{this.title}}</b></h2>
    <p>{{this.description | slice:0:600}}<a href=‘https://en.wikipedia.org/wiki/{{this.title}}’>..more at Wikipedia</a></p>
  </div>
</div>

Resources

1.MediaWiki API : https://www.mediawiki.org/wiki/API:Main_page

2.Stackoverflow : https://stackoverflow.com/questions/8555320/is-there-a-clean-wikipedia-api-just-for-retrieve-content-summary

3.Angular Docs : https://angular.io/tutorial/toh-pt4

Integrating YaCy Grid Locally with Susper

The YaCy Grid is the second-generation implementation of YaCy, a peer-to-peer search engine.The search results can be improved to a great extent by using YaCy-Grid as the new backend for SUSPER. YaCy Grid is the best choice for distributed search topology. The legacy YaCy is made for decentralised and also distributed network. While both the networks are distributed,the YaCy-Grid is centralized and legacy YaCy is decentralized. YaCy Grid facilitates a lot with scaling that will be in our hand and can be done in all aspects​(loading, parsing, indexing) with computing power we choose. In YaCy,Solr is embedded. But in YaCy Grid,we will get elasticsearch cluster.​They are both built around the core underlying search library Lucene.But ​elasticsearch will help us to scale almost indefinitely. In this blog, I will show you how to integrate YaCy Grid with Susper locally and how to use it to fetch results.

Implementing YaCy Grid with Susper:

Before using YaCy Grid we need to first setup YaCy Grid and crawl to url using crawl start API, more information about that can be found here Implementing YaCy Grid with Susper and Setting up YaCy Grid locally.

So, once we are done with setup and crawling, we need to begin using its APIs in Susper. Following are some easy steps in which we can show results from YaCy Grid in a separate tab is Susper.

Step 1:

Creating a service to fetch results:

In order to fetch results from local YaCy Grid server we need to create a service to fetch results from local YaCy Grid server. Here is the class in grid-service.ts which fetches results for us.

export class GridSearchService {
 server = 'http://127.0.0.1:8100';
 searchURL = this.server + '/yacy/grid/mcp/index/yacysearch.json?query=';
 constructor(private http: Http,
             private jsonp: Jsonp,
             private store: Store<fromRoot.State>) {
 }
 getSearchResults(searchquery) { 
   return this.http
     .get(this.searchURL+searchquery).map(res =>
         res.json()
     ).catch(this.handleError);
 }

 

Step 2:

Modifying results.component.ts file

In order to get results from grid-service.ts in results.component.ts we must need to create an instance of the service and use this instance to get the results and store it in variables results.component.ts file and then use these variables to show results in results template. Following is the code that does this for us

ngOnInit() {
   this.grid.getSearchResults(this.searchdata.query).subscribe(res=>{
     this.gridResult=res.channels;
   });
 }

 

gridClick(){
   this.getPresentPage(1);
   this.resultDisplay = 'grid';
   this.totalgridresults=this.gridResult[0].totalResults;
   this.gridmessage='About ' + this.totalgridresults + ' results';
   this.gridItems=this.gridResult[0].items;
  
   console.log(this.gridItems);
 }

 

Step 3:

Creating a New tab to show results from YaCy Grid:

Now we need to create a tab in the template where we can use local variables in results.component.ts to show the results following the current design pattern here is the code for that

<li [class.active_view]="Display('grid')" (click)="gridClick()">YaCy_Grid</li>

<!--YaCy Grid-->
 <div class="container-fluid">
     <div class="result message-bar" *ngIf="totalgridresults > 0 && Display('grid')">
       {{gridmessage}}
     </div>
     <div class="autocorrect">
       <app-auto-correct [hidden]="hideAutoCorrect"></app-auto-correct>
     </div>
   </div>
 <div class="grid-result" *ngIf="Display('grid')">
   <div class="feed container">
       <div *ngFor="let item of gridItems" class="result">
         <div class="title">
           <a class="title-pointer" href="{{item.link}}" [style.color]="themeService.titleColor">{{item.title}}</a>
         </div>
         <div class="link">
           <p [style.color]="themeService.linkColor">{{item.link}}</p>
         </div>
         <div class="description">
           <p [style.color]="themeService.descriptionColor">{{item.pubDate|date:'MMMM d, yyyy'}} - {{item.description}}</p>
         </div>
       </div>
   </div>
 </div>
 <!-- END -->

 

Step 4:

Starting YaCy Grid Locally:

Now all we need is to start YaCy Grid server locally. To start it go in yacy_grid_mcp folder and use

python bin/start_elasticsearch.py

 

This will start elasticsearch from its respective script.Next use

python bin/start_rabbitmq.py

 

This will start RabbitMQ server with the required configuration.Next useThis will start elasticsearch from its respective script.Next use

gradle run

 

To start YaCy Grid locally.

Now we are all done we just need to start Susper using

ng serve

 

command and type a search query and move to YaCy_Grid tab to see results from YaCy Grid Server.

Here is the image which shows results from YaCy Grid in Susper

Resources