Getting results for Multi-word Query and showing limited results in Knowledge Graph

In Susper knowledge graph,we were unable to get results for multi word query from Wikipedia API and also we were unable to decide how much information should be shown in knowledge graph which was retrieved from Wikipedia API.For example for a query donald trump we do not get information (https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=donald%20trump) . Also for searching for any query like india (https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=india) we get a lot of information. Earlier we used Angular slice pipe to display only 600 characters in knowledge graph (Infobox) but almost for all queries the sentences were terminated before it was completed. In this blog, I will describe how we solved both problems in knowledge graph.

Getting results for multi word query from Wikipedia API:

On searching a lot we found that the results were present on Wikipedia for multi word queries but these queries must have its starting letters as capital letters. For example we do not get results for queries such as donald trump

i.e https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=donald%20trump

but when we make a query for Donald Trump  

i.e

https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=Donald%20Trump  

But since in most of the search queries users use small letters, we need to convert the query such that each word starts with a capital letter. Since the Knowledge Graph has been implemented using ngrx pattern, this logic was easily implemented in knowledge effects. For this we will be selecting each word in query by using regular expression and then we will use toUppercase() and toLowerCase() methods of javascript and will capitalize every word of the query.

toTitleCase(str) {
    return str.replace(
        /\w\S*/g,
        function(txt) {
            return txt.charAt(0).toUpperCase() + txt.substr(1).toLowerCase();
        }
    );
}

this.knowledgeservice.getSearchResults(this.toTitleCase(querypay.query))

 

Like this we now get results for almost all queries.

Limiting information in Knowledge Graph without terminating the sentences:

To solve this issue, we decided to show only four lines of data retrieved from Wikipedia API

For this we have implemented a getPosition function which will take three parameters a string, a substring and a index. Here string is the whole string from which the function will return the index of index position substring.

getPosition(string, subString, index) {
    return string.split(subString, index).join(subString).length;
 }
this.description = this.description.slice(0, this.getPosition(this.description, '.', 4) + 1);

 

Here, We get the index of 4th ‘.’ full stop by getPosition() function and then pass it to javascript slice method to slice the string upto that position.

Using this we limited results to 4 lines without terminating each line in middle.

References

1.W3Schools Regular Expression: https://www.w3schools.com/jsref/jsref_obj_regexp.asp

2.Javascript Slice method: https://www.w3schools.com/jsref/jsref_slice_array.asp

3.Wikipedia API: https://www.mediawiki.org/wiki/API:Main_page

Continue ReadingGetting results for Multi-word Query and showing limited results in Knowledge Graph

Displaying All Providers and All Authors separately in Susper Angular App

In Susper results are retrieved from a lots of providers and authors. Displaying all authors’ and providers’ list on result page make appearance of result section odd and more complex. In this blog, I will describe how we have separated the list of all providers and authors from main result page and displayed it on a lightbox.

Steps:

  • Making a black overlay for lightbox:

First we of all to display the results on a lightbox,we to have a black overlay for this we need a div element and inside it we need a closing button.

Clicking on the black overlay should toggle status of light box therefore we have binded click event with showMore() method.

<div id="fade_analytics" class="black_overlay" (click)='showMore()'><a class="closeX" id="closeX">&#10006;</a></div>
  • Creating a white content to display all Providers or Authors:

Now we need a white block to display all Providers or all Authors so after creating a black overlay in background we need a white foreground.

Therefore I have implemented a div element and put all the chart elements inside the div element so that it will be displayed on a white foreground on a translucent black background.

<div id="light_analytics" class="white_content_analytics">
<div class="show_more"><h2 class="heading"><b>Analytics</b></h2><div id="filtersearch" *ngFor="let nav of navigation$|async; let i = index"><div class="card-container" id="relate" *ngIf="nav.displayname =='Provider' || nav.displayname =='Authors'"><h3  class="related-searches" *ngIf="i===currentelement"><b>Top {{nav.displayname}}<span *ngIf="nav.displayname.slice(-1)!=='s'">s</span></b></h3>
<div style="display: block;cursor:pointer;cursor: hand;" *ngFor="let element of  ((i != currentelement)? (nav.elements | slice:0:0) :nav.elements)">
</div></div></div></div>

The code to display All providers and All authors are contained inside the   white block and the logic to display the each section is also implemented.

  • Showing only 50 results on result page and implementing a button to open all providers or authors in light box:

Currently we are displaying the top five providers and authors and we provide a button to show more providers and authors which then displays 50 more results and then we display an interactive button that toggles display of all authors and providers on a lightbox. To display 5 and 50 results we use slice pipe in angular. The button Show All Providers/Authors is binded with showMore() function which toggles lightbox.

<div style="display: block;cursor:pointer;cursor: hand;" *ngFor="let element of ((i != selectedelement)? (nav.elements | slice:0:5) :(nav.elements | slice:0:50) )">
<a [routerLink]="['/search']" [queryParams]="changequerylook(element.modifier,element)" *ngIf="element.name" href="#" [style.color]="themeService.linkColor">
{{element.name}} <span class="badge badge-info">{{element.count}}</span>
</a><a  *ngIf="!element.name" style="color: black;cursor: default" href="#" [style.color]="themeService.linkColor">
Undefined Author <span class="badge badge-info">{{element.count}}</span></a></div>
<button class="showMoreBtn" (click)="showMore()" *ngIf="selectedelement===i" [style.color]="themeService.linkColor">Show All {{nav.displayname}}<span *ngIf="nav.displayname.slice(-1)!=='s'">s</span></button>

 

  • Implementing the logic to display all results on white content:

Now we need to implement the logic to toggle lightbox to display All Providers and All Authors. This logic is implemented mainly in showMore() function which toggles the lightbox containing the All Providers and All Authors

showMore() { this.selectedelement = -1;
if (!this.showMoreAnalytics) { this.showMoreAnalytics = true; document.getElementById('light_analytics').style.display = 'block'; document.getElementById('fade_analytics').style.display = 'block'; } else { this.showMoreAnalytics = false;
document.getElementById('light_analytics').style.display = 'none';
document.getElementById('fade_analytics').style.display = 'none';}}

Here we use showMoreAnalytics variable to check whether the user has asked to show all Providers or Authors. We also make selectedelement variable to -1 to hide the long list of 50 Providers or Authors when Lightbox displays all the Providers or Author. Finally we use document object in javascript and select the DOM elements by their id and set their display property this shows and hides the lightbox on result page.

Resources

1.YaCy Advanced Search Parameters

http://www.yacy-websuche.de/wiki/index.php/En:SearchParameters

2.W3School lightbox

https://www.w3schools.com/howto/howto_js_lightbox.asp

 

Continue ReadingDisplaying All Providers and All Authors separately in Susper Angular App

Adding New tests for Knowledge Service

Testing is done to test our application by executing some functions by creating instances of corresponding classes,executing functions and checking the actual behaviour of our app with expected result. The tools and frameworks used in Angular are Jasmine and Karma. In this blog, I will describe about how I have implemented tests for Newly Added Knowledge API service that helped us to increase overall code coverage by 1.05% in Susper .

Adding tests for Knowledge API:

We need to check the API that whether it is functioning or not and this can be done by using a mocked response (hardcoded response) for any query and then comparing this with the received response from the API. This will help us to check proper functioning of our API.

This is a common practice in Angular and to achieve this we will be using some dependencies like MockBackend, MockConnection, BaseRequestOptions provided by Angular.

import { MockBackend, MockConnection } from '@angular/http/testing';
import { Http, Jsonp, BaseRequestOptions, RequestMethod, Response, ResponseOptions, HttpModule, JsonpModule } from '@angular/http';

 

We will also need to define a Mock response, here I have used hard coded response for query India. Here is the mocked response.

export const MockKnowledgeApi = {
    results: [
        {"batchcomplete": "", "query":
        {"normalized":
        [{"from": "india", "to": "India"}], "pages":
        {"14533": {"pageid": 14533, "ns": 0, "title": "India", "extract": `India (IAST: Bh\u0101rat), also called the Republic of India (IAST: Bh\u0101rat Ga\u1e47ar\u0101jya), is a country in South Asia.
   }}}} ], MaxHits : 5 };

 

Now we will use a Mock constant mock_Http_provider for http and will inject instances of MockBackend and BaseRequestOptions.

const mockHttp_provider = {
  provide: Http,
  deps: [MockBackend, BaseRequestOptions],
  useFactory: (backend: MockBackend, options: BaseRequestOptions) => { return new Http(backend, options); } };

 

Now we need to add all the services and dependencies which we will be using in providers and we will inject the instances of Knowledgeapi Service and MockBackend in beforeEach function.

beforeEach(inject([KnowledgeapiService, MockBackend], (knowledgeService: KnowledgeapiService, mockBackend: MockBackend) => { service = knowledgeService;
    backend = mockBackend;}));

 

Now we will use the same query for which we have created the mocked response and we will be checking the response of from our API.

const searchquery = 'india';
connection.mockRespond(new Response(options));
      expect(connection.request.method).toEqual(RequestMethod.Get);
      expect(connection.request.url).toBe(   `https://en.wikipedia.org/w/api.php?&origin=*&format=json&action=query&prop=extracts&exintro=&explaintext=&` +
                    `titles=${searchquery}`);});
service.getSearchResults(searchquery).subscribe((res) => {
      expect(res).toEqual(MockKnowledgeApi);});

 

This will check the working of our API and if it is working then our test case will pass.

Like this we have implemented the tests for Knowledgeapi Service which helped us to test our API and increase overall code coverage significantly.

Resources

  1. Testing in Angular:https://angular.io/guide/testing
  2. Testing by Mockbackend:https://angular.io/api/http/testing/MockBackend
  3. Using MockBackend to simulate response: https://codecraft.tv/courses/angular/unit-testing/http-and-jsonp/#_using_the_code_mockbackend_code_to_simulate_a_response
Continue ReadingAdding New tests for Knowledge Service

Using routerLink to make result tabs as links

In Susper, it was required that different result sections i.e All, Images,Videos and News should behave as links. Links provide us an advantage of opening the different result tabs in a New Tab by right clicking on it. Also at the same time we wanted that Active tabs must not behave as links. We wanted that tabs should behave in a similar way as Google, Bing and other search engine.In this blog, I will describe how we have achieved this by using routerLink and [hidden] attribute in Angular.

Steps:

1.Adding routerLink attribute in anchor tag.

Now, To make any element as link we must enclose it in anchor tag, therefore we will use <a> tag enclosing the <li> tag to make it a link.

<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,resultDisplay:'all',mode:'text',nopagechange:'false',append:'false'}"><li [class.active_view]="Display('all')" (click)="docClick()">All</li></a>
<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,mode:'text',nopagechange:'false',append:'false',fq:'url_file_ext_s%3A(png%2BOR%2Bjpeg%2BOR%2Bjpg%2BOR%2Bgif)',resultDisplay:'images'}"><li [class.active_view]="Display('images')" (click)="imageClick()">Images</li></a>
<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,resultDisplay:'videos',mode:'text',nopagechange:'false',append:'false',fq:'url_file_ext_s%3A(avi%2BOR%2Bmov%2BOR%2Bflv%2BOR%2Bmp4)'}"><li [class.active_view]="Display('videos')" (click)="videoClick()">Videos</li></a>
<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,resultDisplay:'news',mode:'text',nopagechange:'false',append:'false'}"><li [class.active_view]="Display('news')" (click)="newsClick()">News</li></a>

Now we must navigate to a link on the click of this tabs, for this we can simply use href attribute which we earlier tried to use but there is a disadvantage of using href tag. Using href tab reloads the whole page when we try to navigate to another tab.

Therefore we will use routerLink to achieve the same functionality as Susper is a single page application, where the page should not be reloaded. routerLink navigates to the new url and the component is rendered in place of <router-outlet> without reloading the whole page.

After using routerLink, we will use queryParams attribute to pass routing parameters to a route. Since these parameters will be variable we must use string interpolation to set query parameters. We will use searchdata object and set query parameters according to its value.

Now we are done with making tabs as links. But as of now all tabs will behave a links even if it is an active tab. Therefore we need to hide the link property of any tab when it is active.

2.Hiding the link property of the active tab:

Now we will use to different elements for each tab, one as link and other as normal li element.For hiding the link elements when the tab is active we will use [hidden] attribute in angular. We will just hide the link element when the value of resultDisplay variable is similar to current tab.

<a [routerLink]="['/search']" [queryParams]="{query:this.searchdata.query,start:this.searchdata.start,rows:this.searchdata.rows,resultDisplay:'all',mode:'text',nopagechange:'false',append:'false'}" [hidden]="this.resultDisplay==='all'"><li [class.active_view]="Display('all')" (click)="docClick()">All</li></a>
        <li [class.active_view]="Display('all')" (click)="docClick()" *ngIf="Display('all')">All</li>

For hiding the simple li element we will use *ngIf attribute which will remove the simple li element from the DOM for un active tabs. We might have used *ngIf attribute to hide link elements but rendering a lot of elements on DOM slows down the page and affects performance.

We will follow this for rest of the tabs also and we are done with our changes.

Now we can the inactive tabs will behave as link and the active tabs will not behave as link similar to Google.

Resources

  1. Angular Routing: https://angular.io/guide/router
  2. ngIf and [hidden]: https://stackoverflow.com/questions/39777990/angular2-conditional-display-bind-to-hidden-property-vs-ngif-directive
  3. Angular queryParams: https://stackoverflow.com/a/46008187
Continue ReadingUsing routerLink to make result tabs as links

Deploying Susper On Heroku

In Susper, currently we have two branches, master and development branch. The master branch is deployed on susper.com and we needed a deployment of development branch. For this we are using heroku and in this blog I will describe how we have deployed Susper’s development branch on heroku at http://susper-dev.herokuapp.com

Here are the steps:

1.Get a free heroku account:

To deploy our apps on heroku we must have a heroku account which can be created for free on https://www.heroku.com.

2.Create a new app on heroku:

After creating an account we need to create an app with a name and a region, for Susper we have used name susper-dev and region as United States.

3.Link the app to Susper’s development branch:

After successfully creating the app we need to link our app to Susper’s development branch and in Deploy section and then we have to enable automatic deployment from development branch after CI passes.

  1. Setup the Susper Project:

Now we have deployed our and we need to configure it so that it can successfully start on heroku. Following are the steps to configure Susper.

a) Add node and npm engine in package.json:

Now we need to tell heroku which npm and node version to use while running our app,this can be done by defining an engine field in package.json and adding npm and node versions as values as shown here:

"engines": {
   "node": "8.2.1",
   "npm": "6.0.1"
 }

 

b) Adding a postinstall field under scripts field in package.json:

Now we need to tell heroku the command that we will be using to build our app. Since on susper.com we are using ng build –prod –build-optimizer that builds our app for production by optimizing the build artifacts. Therefore on heroku also we will be using the same command:

"postinstall": "ng build --prod --build-optimizer"

 

c) Defining dependencies and typescript version for our project:This can be done by defining the @angular/cli ,@angular/compiler-cli and typescript and their version under dependencies field in package.json. In Susper we are using the following dependency versions.

"@angular/cli": "^1.1.0",
"@angular/compiler": "4.1.3",
"typescript": "~2.3.4"

 

d) Creating a javascript file to install express server and run our app: Locally when we run ng serve angular cli automatically creates a express server where our app is deployed locally but on heroku we will be required to start our express server which will be used by our app using a javascript file. Here is the code for starting the server.

//Install the server
const express = require('express');
const path = require('path');
const app = express();
// Serving the static file from dist folder
app.use(express.static(__dirname + '/dist'));
app.get('/*', function(request,response) {
response.sendFile(path.join(__dirname+'/dist/index.html'));
});
// Starting the app and listening on default heroku port.
app.listen(process.env.PORT || 8080);

 

Now we need to run this file, for this we will change start command in package.json file under script to start this file.

"start": "node server.js"

 

Now everytime a new commit is made to development branch our app will be automatically deployed on heroku at https://susper-dev.herokuapp.com

Resources

 

 

Continue ReadingDeploying Susper On Heroku

Displaying Charts on Lightbox

Susper displayed charts which shows graphical representation of Result Frequency and Protocol Distribution(http or https). But since results from these charts added little value it was decided to display charts on a click of a button on a lightbox. (Issue: https://github.com/fossasia/susper.com/issues/1066 ). In this blog,I will describe how I have implemented charts on lightbox.

Uniting the graphs(Line and Bar Graph) in a same class:

Before beginning the implementation of lightbox, we should unite all the components that will be displayed on lightbox in a single class.

Here I have put the different types of charts in a class named graph.

<div class="graph large"> <div class="linegraph"><canvas baseChart
           [datasets]="lineChartData"
           [labels]="lineChartLabels"
           [options]="lineChartOptions"
           [chartType]="lineChartType"
           [colors]="lineChartColors"
></canvas></div><div class="bargraph"><canvas baseChart
           [datasets]="barChartData"
           [labels]="barChartLabels"
           [options]="barChartOptions"
           [chartType]="barChartType"
           [colors]="lineChartColors"
   ></canvas></div></div>

Implementing button to toggle display of light box:

To show and hide lightbox along with charts we must have a button.

Clicking on which should trigger a function which will internally execute the logic in typescript file to show and hide the lightbox.

<button class="btn" id="toggle-chart-button" (click)="BoxToggle()">
           {{analyticsStatus}} Analytics
 </button>

Creating a translucent black background:

Now to have a black overlay for lightbox we need a div element and inside it we need a closing button.

Clicking on the black overlay should toggle status of light box therefore we have binded click event with BoxToggle() method.

<div id="fade" class="black_overlay" (click)='BoxToggle()'><a class="closeX" id="closeX">&#10006;</a></div>

Creating a white block to display charts:

Now after creating a black overlay in background we need a white foreground to display charts.

Therefore I have implemented a div element and put all the chart elements inside the div element so that it will be displayed on a white foreground on a translucent black background.

<div id="light" class="white_content">
<div class="graph large"><div class="linegraph"><canvas baseChart
           [datasets]="lineChartData"
           [labels]="lineChartLabels"
           [options]="lineChartOptions"
           [chartType]="lineChartType"
           [colors]="lineChartColors"
   ></canvas></div>
<div class="bargraph"><canvas baseChart
           [datasets]="barChartData"
           [labels]="barChartLabels"
           [options]="barChartOptions"
           [chartType]="barChartType"
           [colors]="lineChartColors"
   ></canvas></div></div></div>

CSS for all above elements:

All the CSS which is written for charts, toggle button, white block and black overlay

To make it look attractive and similar to market leader can be found here.

Implementing the logic in typescript file:

Now we have created all the elements in frontend to display the lightbox but we need a little logic to show and hide the lightbox on clicking the button. For this I have used a variable analyticsStatus which is a string and is initialised with value ‘Show Chart’. Its value is displayed on the toggle button. I have used this variable with BoxToggle() function for implementing the logic for showing and hiding the lightbox.

This function checks the status of message and toggles it along with toggling lightbox with some javascript code. The code for the function is here.

BoxToggle() { if (this.analyticsStatus === 'Show Chart') {
     this.analyticsStatus = 'Hide Chart';
     document.getElementById('light').style.display = 'block';
     document.getElementById('fade').style.display = 'block';
   } else {
     this.analyticsStatus = 'Show Chart';
     document.getElementById('light').style.display = 'none';
     document.getElementById('fade').style.display = 'none'; }}

Resources

1.Creating a lightbox: https://www.emanueleferonato.com/2007/08/22/create-a-lightbox-effect-only-with-css-no-javascript-needed/

2.W3School lightbox: https://www.w3schools.com/howto/howto_js_lightbox.asp

3.Angular 2 Lightbox: https://lokeshdhakar.com/projects/lightbox2/

Continue ReadingDisplaying Charts on Lightbox

Extending the News Feature to Show results from multiple organisations

News Tab in Susper was earlier implemented to show results only from a single organisation using  site: modifier for query facet. In this blog I will discuss about how I have modified the current News Tab to show results from various News organisation like BBC, Al Jazeera, The Guardian etc.

Implementation:

Step 1:

Creating a JSON file to store organisations detail:

We need to decide from which organisations  we will fetch results using YaCy Server and then display it in Susper. We have provided a JSON file where user can add or delete News sources easily. The results will only limited to the organisations which are present in the JSON file.

Step 2:

Creating a service to fetch details from JSON file:

Now after creating the the JSON file we need a service which will fetch results from the JSON file according to our need. This service will be a simple  Angular Service having a class GetJsonService and it will access the newsFile.json and map the results in JSON format. Here is the service which does this task for us.

Step 3:

Creating a service to fetch news accordingly:

Now after fetching the JSON result we need a service to fetch the News Results from YaCy Server. I have created a separate service to do this task where I have fetched results using site: modifier from each organisation and returned the results.The code for the news.service.ts is below.

export class NewsService {
 constructor(private jsonp: Jsonp) { }
 getSearchResults(searchquery, org) {
   let searchURL = 'https://yacy.searchlab.eu/solr/select?query=';
   searchURL += searchquery.query + ' site:' + org;
   let params = new URLSearchParams();
   for (let key in searchquery) {
     if (searchquery.hasOwnProperty(key)) {
       params.set(key, searchquery[key]); } }
  //Set other parameters
   return this.jsonp
     .get(searchURL, {search: params}).map(res =>
       res.json()[0]
     ).catch(this.handleError); }

 

Step 4:

Updating the results section:

Now we have a service that gives results from a single organisation and a JSON list of organisations. Now in results.component.ts we can simply subscribe to getJsonService and in a loop we will call getNewsService by changing the organisation in every iteration. We will then check that whether we are getting the valid results or not (undefined). The results which are not valid can cause errors when we will try to read any field of an undefined variable. Then, We will simply append the 2 result items from each organisation in an empty array and later use this array to show results.

this.getJsonService.getJSON().subscribe(res => { this.newsResponse = [];
for (let i = 0; i < res.newsOrgs.length; i++) { this.getNewsService.getSearchResults(querydata,res.newsOrgs[i].provider).subscribe( response => {
    if (response.channels[0].items[0] !== undefined) {
    this.newsResponse.push(response.channels[0].items[0]); }
    if (response.channels[0].items[1] !== undefined) {
    this.newsResponse.push(response.channels[0].items[1]); } } ); }
    });

 

The newsClick() function is activated on clicking News Tab and it updates the query and its details in store.

Step 5

Displaying the results:

Now we will modify the results.component.html to show results from new newsResponse array which have 2 results each from 5 organisations.

For this we will use each item of newsResponse using *ngFor and display its title,link and description in html template. We will also use [style.color] property of our element and set the color according to theme.

<div *ngFor="let item of newsResponse" class="result"> <div class="title">
<a class="title-pointer" href="{{item.link}}" [style.color]="themeService.titleColor">{{item.title}}</a>
</div> <div class="link">
<p  [style.color]="themeService.linkColor">{{item.link}}</p>
 </div> </div>

 

Here is the view of Susper’s News Tab where we are getting results from 5 different organisations.

Resources

  1. YaCy Modifiers: http://www.yacy-websuche.de/wiki/index.php/En:SearchParameters
  1. Angular Services: https://angular.io/tutorial/toh-pt4
  2. Reading JSON data in Angular: https://stackoverflow.com/questions/43275995/angular4-how-do-access-local-json

 

Continue ReadingExtending the News Feature to Show results from multiple organisations

Deploying Angular App Susper on GitHub Pages

What are the issues we had with automatic deployment of Susper?

SUSPER project which is built on Angular is kept on master branch of the repository .

Whenever any PR is merged,Travis CI does all the builds, and checks for any error. After successful checking it triggers deployment script in SUSPER (deploy.sh) .

Here is the deployment script:

#!/bin/bash

# SOURCE_BRANCH & TARGET_BRANCH stores the name of different susper.com github branches.
SOURCE_BRANCH="master"
TARGET_BRANCH="gh-pages"

# Pull requests and commits to other branches shouldn't try to deploy.
if [ "$TRAVIS_PULL_REQUEST" != "false" -o "$TRAVIS_BRANCH" != "$SOURCE_BRANCH" ]; then
    echo "Skipping deploy; The request or commit is not on master"
    exit 0
fi

# Store some useful information into variables
REPO=`git config remote.origin.url`
SSH_REPO=${REPO/https:\/\/github.com\//git@github.com:}
SHA=`git rev-parse --verify HEAD`

# Decryption of the `deploy.enc`
openssl aes-256-cbc -k "$super_secret_password" -in deploy.enc -out deploy_key -d

# give `deploy_key`. the permission to read and write
chmod 600 deploy_key
eval `ssh-agent -s`
ssh-add deploy_key

# Cloning the repository to repo/ directory,
# Creating gh-pages branch if it doesn't exists else moving to that branch
git clone $REPO repo
cd repo
git checkout $TARGET_BRANCH || git checkout --orphan $TARGET_BRANCH
cd ..

# Setting up the username and email.
git config user.name "Travis CI"
git config user.email "$COMMIT_AUTHOR_EMAIL"

# Cleaning up the old repo's gh-pages branch except CNAME file and 404.html
find repo/* ! -name "CNAME" ! -name "404.html" -maxdepth 1  -exec rm -rf {} \; 2> /dev/null
cd repo

# commit the changes, move to SOURCE_BRANCH
git add --all
git commit -m "Travis CI Clean Deploy : ${SHA}"

git checkout $SOURCE_BRANCH

# Actual building and setup of current push or PR. Move to `TARGET_BRANCH` and move the `dist` folder
npm install
ng build --prod --aot
mv susper.xml dist/
mv 404.html dist/

git checkout $TARGET_BRANCH
mv dist/* .

# Staging the new build for commit; and then committing the latest build
git add .
git commit --amend --no-edit --allow-empty

# Actual push to gh-pages branch via Travis
git push --force $SSH_REPO $TARGET_BRANCH

This script starts after successfully checking the project (comments shows working of each command). The repository is cleaned,except some files like CNAME and 404.html and a commit is made. Then SUSPER’s build artifacts are generated in dist folder and all the files from dist folder are moved to gh-pages and the changes are amended. GitHub Pages engine then uses the build artifacts to generate static site at susper.com. But we faced a weird problem when we were unable to see changes from latest commits on susper.com. Slowly number of commits increased but still changes were not reflecting on susper.com .

What was the error in deployment of SUSPER?

All the changes were getting committed to the gh-pages branch properly. But the GitHub Pages engine was unable to process the branch to build static site. Due to this the site was not getting updated. The failing builds from gh-engine are notified to the owner via email.

The failing builds from gh-pages can be seen here https://github.com/fossasia/susper.com/commits/gh-pages

Between 21 May to 18 March.

There was a weird error ( Failure: The tag `chartjs` in line 4 on `node_modules/chart.js/docs/charts/bar.md` is not recognized liquid tag) while building the site from build artifacts. There was an issue with `gh-pages` engine as `node_modules/chart.js/docs/charts/bar.md`  was previously also present but then there was no such errors than.

Due to this error all commits which were made to SUSPER repository was not shown on susper.com as the site was not getting updated.

How we solved it?

We tried to search a lot about the above mentioned error but we were unable to find a proper solution. While searching we came across this question https://stackoverflow.com/questions/24713112/my-github-page-wont-update-its-content

on StackOverflow. There was no accepted answer for this question and every answer was different from other. So, we just reproduced the bug in our own repository by just making a similar repository to SUSPER with same branches. Once we successfully reproduced the bug. We tried to correlate all the answers to arrive at a solution to fix this bug. Luckily we found that any commit with no significant change from previous commit removed the error and allowed the gh-pages engine to successfully create the site. So we made a dummy commit in which we created a hidden file with no content and pushed it to gh-pages. This did the trick for us and gh-pages was now able build the site for us.

The whole problem was due to a bug in GitHub pages engine which got fixed by a dummy commit.

Resources

  1. Stackoverflow:https://stackoverflow.com/questions/24713112/my-github-page-wont-update-its-content
  2. Publishing to GitHub Pages:https://help.github.com/articles/configuring-a-publishing-source-for-github-pages/
  3. Commits(gh-pages):https://github.com/fossasia/susper.com/commits/gh-pages
Continue ReadingDeploying Angular App Susper on GitHub Pages

Using Wikipedia API for knowledge graph in SUSPER

Knowledge Graph is way to give a brief description about search query by connecting it to a real world entity. This helps users to get information about exactly what they want. Previously Susper had a Knowledge Graph which was implemented using DBpedia API. But since DBpedia do not provide content over HTTPS connections therefore the content was blocked on susper.com and there was a need to implement the Knowledge Graph using a new API that provide contents over HTTPS. In this blog, I will describe how getting a knowledge graph was made possible using Wikipedia API.

What is Wikipedia API ?

The MediaWiki action API is a web service that provides convenient access to wiki features, data, and metadata over HTTP, via a URL usually at api.php. Clients request particular “actions” by specifying an action parameter, mainly action=query to get information.

The endpoint :

https://en.wikipedia.org/w/api.php

The format :

format=json This tells the API that we want data to be returned in JSON format.

The action :

action=query

The MediaWiki web service API implements dozens of actions and extensions implement many more; the dynamically generated API help documents all available actions on a wiki. In this case, we’re using the “query” action to get some information.

The complete API which is used in SUSPER to extract information of a query is :

https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=japan

Where titles=Search_Query, here Japan

How it is implemented in SUSPER?

For implementing it a service has been created which fetches information by setting various URL parameters. This result can be fetched by creating an instance of service and passing search query to getsearchresults(searchquery) function.

export class KnowledgeapiService {
 server = 'https://en.wikipedia.org';
 searchURL = this.server + '/w/api.php?';
 homepage = 'http://susper.com';
 logo = '../images/susper.svg';
 constructor(private http: Http,
             private jsonp: Jsonp,
             private store: Store<fromRoot.State>) {
 }
 getsearchresults(searchquery) {
   let params = new URLSearchParams();
   params.set('origin', '*');
   params.set('format', 'json');
   params.set('action', 'query');
   params.set('prop', 'extracts');
   params.set('exintro', '');
   params.set('explaintext', '');
   params.set('titles', searchquery);
   let headers = new Headers({ 'Accept': 'application/json' });
   let options = new RequestOptions({ headers: headers, search: params });
   return this.http
     .get(this.searchURL, options).map(res =>
         res.json().query.pages
     ).catch(this.handleError);
}

 

Since the result obtained is an observable therefore we have to subscribe for it and then extract information to local variables in infobox.component.ts file.

export class InfoboxComponent implements OnInit {
 public title: string;
 public description: string;
 query$: any;
 resultsearch = '/search';
 constructor(private knowledgeservice: KnowledgeapiService,
             private route: Router,
             private activatedroute: ActivatedRoute,
             private store: Store<fromRoot.State>,
             private ref: ChangeDetectorRef) {
   this.query$ = store.select(fromRoot.getquery);
   this.query$.subscribe( query => {
     if (query) {
       this.knowledgeservice.getsearchresults(query).subscribe(res => {
         const pageId = Object.keys(res)[0];
         if (res[pageId].extract) {
           this.title = res[pageId].title;
           this.description = res[pageId].extract;
         } else {
           this.title = '';
           this.description = '';
         }
       });
     }
   });
 }

The variable title and description are used to display results on results page.

<div *ngIf=“this.description” class=“card”>
  <div>
    <h2><b>{{this.title}}</b></h2>
    <p>{{this.description | slice:0:600}}<a href=‘https://en.wikipedia.org/wiki/{{this.title}}’>..more at Wikipedia</a></p>
  </div>
</div>

Resources

1.MediaWiki API : https://www.mediawiki.org/wiki/API:Main_page

2.Stackoverflow : https://stackoverflow.com/questions/8555320/is-there-a-clean-wikipedia-api-just-for-retrieve-content-summary

3.Angular Docs : https://angular.io/tutorial/toh-pt4

Continue ReadingUsing Wikipedia API for knowledge graph in SUSPER

Adding ‘Voice Search’ Feature in Loklak Search

It is beneficial to have a voice search feature embedded in a search engine like loklak.org based on PWA (Progressive Web Application) architecture. This will allow users to make query using voice on phone or computer. For integrating voice search, JavaScript Web Speech API (also known as webkitSpeechRecognition) is used which allows us to add speech recognition feature in any website or a web application.

Integration Process

For using webkitSpeechRecognition, a separate typescript service is being created along with its relevant unit test

  1. speech.service.ts
  2. speech.service.spec.ts

The main idea to implement the voice search was

  1. To create an injectable speech service.
  2. Write methods for starting and stopping voice record in it.
  3. Import and inject the created speech service into root app module.
  4. Use the created speech service into required HomeComponent and FeedHeaderComponent.

Structure of Speech Service

The speech service mainly consists of two methods

  • record() method which accepts lang as string input which specifies the language code for speech recording and returns a string Observable.

   record(lang: string): Observable<string> {
           return Observable.create(observe => {
                 const webkitSpeechRecognition }: IWindow = 
                        <IWindow>window;
                 this.recognition = new webkitSpeechRecognition();
                 this.recognition.continuous = true;
                 this.recognition.interimResults = true;
                 this.recognition.onresult = take => 
                 this.zone.run(()                            
                 observe.next(take
                 .results.item
                 (take.results.
                 length - 1).
                 item(0).
                 transcript);
                 this.recognition
                 .onerror = 
                 err => observe
                 .error(err);
                 this.recognition.onend 
                 = () => observe
                 .complete();
                 this.recognition.lang = 
                 lang;
                 this.recognition.start( 
             );
        });
     }

 

Using observe.complete() allows speech recognition action to stop when the user stops speaking.

  • stoprecord() method which is used to stop the current instance of recording.

 stoprecord() {
       if (this.recognition) {
           this.recognition.stop();
       }
   }

 

stop() method is used to stop the current instance of speech recognition.

Using Speech Service in required Component

In Loklak Search, the speech service have been included in two main components i.e. HomeComponent and FeedHeaderComponent. The basic idea of using the created speech service in these components is same.

In the TypeScript (not spec.ts) file of the two components, firstly import the SpeechService and create its object in constructor.

constructor( private speech: SpeechService ) { }

 

Secondly, define a speechRecognition() method and use the created instance or object of SpeechService in it to record speech and set the query as the recorded speech input. Here, default language has been set up as ‘en_US’ i.e. English.

 stoprecord() {
       if (this.recognition) {
           this.recognition.stop();
       }
   }

 

After user clicks on speech icon, this method will be called and the recorded speech will be set as the query and there will be router navigation to the /search page where the result will be displayed.

Resources

Continue ReadingAdding ‘Voice Search’ Feature in Loklak Search