Holding query search in Loklak

Continuously searching user’s input while the user kept typing is not good. It has certain drawbacks which result in continuous action dispatch and loading of the result page which depicts the site to be slow, but actually it is not. To prevent continuous loading and searching while user types a query in Loklak Search, it was important to keep hold on query until user completes typing and then start searching for the results. I am writing this blog post to emphasise a proper solution to constantly keep track of the query and search on an appropriate event.

Adding keypress event

First and foremost task would be to add keypress event to input tag on home and results page.

<input     
     

     (keypress)=“onEnter($event)” 
     [formControl]=“searchInputControl

/>

 

formControl will constantly keep track of the user input and will be stored in searchInputControl, while keypress event will activate onEnter() method firing an $event as an input to onEnter() method.

Adding onEnter() method

Note: Keeping track of special keypress event code 13 which is equivalent for an enter key press.

Add a new method onEnter() with event as an input which will execute its function only when the event’s keypress code is 13 (Enter key press). One point should be noted here that query to be searched should not be empty or containing leading and trailing spaces or should not contain only spaces (treated as an incomplete query to be searched).

onEnter(event: any) {  
   if (event.which === 13) {  

      if (this.searchInputControl.value.trim() !== ”) {

        // Actual work of query search will go here.

  }

}

Add query searching and migration code under onEnter()

Now comes the last part to shift query search and route migration code from setupSearchField() to the portion mentioned above in comments. We need to take one point in consideration that from now on we actually do not need to subscribe to formControl in ngOnInit() since now it depends on user query input.

Complete onEnter() in home component file

After completion, onEnter() method should be similar to:

onEnter(event: any) {
   if (event.which === 13) {
       if (this._queryControl.value.trim() !== ”) {
           this.store.dispatch(new 
               queryAction.RelocationAfterQuerySetAction());
           this.store.dispatch(new suggestAction
               .SuggestAction(this._queryControl.value.trim()));
           this.store.dispatch(new queryAction
               .InputValueChangeAction(this._queryControl.value.trim()
           ));
           this.router.navigate([`/search`],
               { queryParams: { query: this._queryControl.value.trim() }
               , skipLocationChange: true } );
           this.store.dispatch(new titleAction
           .SetTitleAction(this._queryControl.value.trim()
               +   Loklak Search’));
           this.getDataFromStore();
       }
   }
}

 

Since we are not depending on ngOnInit() therefore we also need to append getDataFromStore() to onEnter() method.

Complete onEnter() in feed-header component file

After completion, onEnter() method should be similar to:

onEnter(event: any) {
   if (event.which === 13) {
       if (this.searchInputControl.value.trim() !== '') {
           this.searchEvent.emit(this.searchInputControl
               .value.trim());
           this.setupSuggestBoxClosing();
       }
   }
}

Testing

Try typing different formats of query on home and results page of loklak.

Resources

Adding Push endpoint to send data from Loklak Search to Loklak Server

To provide enriched and sufficient amount of data to Loklak, Loklak Server should have multiple sources of data. The api/push.json endpoint of loklak server is used in Loklak to post the search result object to server. It will increase the amount and quality of data on server once the Twitter api is supported by Loklak (Work is in progress to add support for twitter api in loklak).

Creating Push Service

The idea is to create a separate service for making a Post request to server. First step would be to create a new ‘PushService’ under ‘services/’ using:

ng g service services/push

Creating model for Push Api Response

Before starting to write code for push service, create a new model for the type of response data obtained from Post request to ‘api/push.json’. For this, create a new file push.ts under ‘models/’ with the code given below and export the respective push interface method in index file.

export interface PushApiResponse {
   status: string;
   records: number;
   mps: number;
   message: string;
}

Writing Post request in Push Service

Next step would be to create a Post request to api/push.json using HttpClient module. Import necessary dependencies and create an object of HttpClient module in constructor and write a PostData() method which would take the data to be send, makes a Post request and returns the Observable of PushApiResponse created above.

import { Injectable } from ‘@angular/core’;
import {
   HttpClient,
   HttpHeaders,
   HttpParams
} from ‘@angular/common/http’;
import { Observable } from ‘rxjs’;
import {
	ApiResponse,
	PushApiResponse
} from ‘../models’;

@Injectable({
   providedIn: ‘root’
})
export class PushService {

   constructor( private http: HttpClient ) { }
   public postData(data: ApiResponse):
   		Observable<PushApiResponse> {

	const httpUrl = ‘https://api.loklak.org/
		api/push.json’;
	const headers = new HttpHeaders({
		‘Content-Type’: ‘application/
			x-www-form-urlencoded’,
		‘Accept’: ‘application/json’,
		‘cache-control’: ‘no-cache’
	});
	const {search_metadata, statuses} = data;
	
	// Converting the object to JSON string.
	const dataToSend = JSON.stringify({
		search_metadata: search_metadata,
		statuses});
	
	// Setting the data to send in
	// HttpParams() with key as ‘data’
	const body = new HttpParams()
		.set(‘data’, dataToSend);
	
	// Making a Post request to api/push.json
	// endpoint. Response Object is converted
	// to PushApiResponse type.
	return this.http.post<PushApiResponse>(
		httpUrl, body, {headers:
		headers
	});
   }
}

 

Note: Data (dataToSend) send to backend should be exactly in same format as obtained from server.

Pushing data into service dynamically

Now the main part is to provide the data to be send into the service. To make it dynamic, import the Push Service in ‘api-search.effects.ts’ file under effects and create the object of Push Service in its constructor.

import { PushService } from ‘../services’;
constructor(
   
   private pushService: PushService
) { }

 

Now, call the pushService object inside ‘relocateAfterSearchSuccess$’ effect method and pass the search response data (payload value of search success action) inside Push Service’s postData() method.

@Effect()
relocateAfterSearchSuccess$: Observable<Action>
   = this.actions$
       .pipe(
           ofType(
               apiAction.ActionTypes
			   	.SEARCH_COMPLETE_SUCCESS,
               apiAction.ActionTypes
			   	.SEARCH_COMPLETE_FAIL
           ),
           withLatestFrom(this.store$),
           map(([action, state]) => {
               this.pushService
			   .postData(action[‘payload’]);
           
       );

Testing Successful Push to Backend

To test the success of Post request, subscribe to the response data and print the response data on console. You should see something like:

Where each of these is a response of one successful Post request.

Resources

Creating new config to set Title of Loklak Webpages

For a Search Engine and Progressive Web Application (PWA) like Loklak, it is important to change the title of each web page with the appropriate query term or configuration of the current active page (A great title can result in higher click-throughs and more traffic to the site, and hence its followed by search engines like Google). For solving this problem, I am writing this blog post which creates a new configuration to change the title dynamically across the whole project by using just one Action thereby maximizing reusability of the code and minimizing duplication.

Create Action for title

We would need to create a new separate title action following the architecture in loklak.

Create a new title.ts file under Actions with the following code:

import { Action } from ‘@ngrx/store’;
export const ActionTypes = {
    SET_TITLE: ‘[Title] SET TITLE’
};
export class SetTitleAction implements Action {
    type = ActionTypes.SET_TITLE;
    constructor (public payload: string) { }
}
export type Actions
    = SetTitleAction;

Create reducer for title

For accessing the current title of the webpage, it is important to create a reducer function for the title following the pattern in loklak.

import * as titleAction from ‘../actions/title’;
export interface State {
	title: string;
}

export const initialState: State = {
	title: '',
};

export function reducer(state: State =
	initialState, action: titleAction.Actions):
	State {
	switch (action.type) {
		case titleAction.ActionTypes.SET_TITLE:
		{
			const titleObt: string =
				action.payload;
			return Object.assign({}, state,
			{
				title: titleObt
			});
		}
		default: {
			return state;
		}
	}
}
export const getTitle =
	(state: State) => state.title;

Create selector and add reducer function to root reducer

The title can be accessed from the current state from store when it would be added to the root state and reducer. Firstly import the title reducer created above and define the title property in root reducer and create a selector for the same.

import * as fromTitle from ‘./title’;
export interface State {
	
	title: fromTitle.State;
}

export const reducers: ActionReducerMap<State> = {
	
	title: fromTitle.reducer
};

export const getTitleState = (state: State) => state.title;

export const getTitle = createSelector(getTitleState,
   fromTitle.getTitle);

Create Effect to change actual title

Now comes the actual part which controls the changing of webpage title using ‘Title’ property of Platform-Browser. Create a new title.effects.ts file under effects using pattern in loklak.

import { Injectable } from ‘@angular/core’;
import { Title } from ‘@angular/platform-browser’;
import { Effect, Actions, ofType }
	from ‘@ngrx/effects’;
import { Observable } from ‘rxjs’;
import { map } from ‘rxjs/operators’;
import * as titleAction from ‘../actions/title’;

@Injectable()
export class SetTitleEffects {

	@Effect({ dispatch: false })
	resetTitle$: Observable<void> = this.actions$
		.pipe(
			ofType(
				titleAction.ActionTypes
					.SET_TITLE
			),
			map((action) => {
			const title =
		action[‘payload’];
			this.titleService
		.setTitle(title);
			})
		);

	constructor(
		private actions$: Actions,
		private titleService: Title
	) { }
}

 

Here the ‘payload’ actually contains the query searched by the user, which will be used update the title.

Note: Before adding a new Effect, the old Effect needs to be removed.

After creating the Effect, add it into app.module.ts and export it in the index.ts.

import {
	
	SetTitleEffects
} from ‘./effects’;

EffectsModule.forRoot([
	
	SetTitleEffects
])

How to set title using this configuration?

Now comes the interesting part. It becomes very easy to set title from any service or component by following the procedure given below.

  • Import the created title action.

import * as titleAction from ‘../actions/title’;

 

  • Set the required title by dispatching the SetTitleAction.
this.store.dispatch(new titleAction
	.SetTitleAction(‘Loklak Search’));

 

You’re all set now! The title has been changed to ‘Loklak Search’ which is what we expect.

Resources

Dispatching a search action to get result for a query in Loklak Search

For a new comer in Loklak, it’s not really easy to understand the complete codebase to just get results for a query without explicitly using navigation route i.e. ‘/search’. In order to help newcomers to easily understand a simple way to get results for a query in the codebase, I am writing this blog post.

Setting up a query

A sample searchQuery will be created of type Query. And following discussion will provide a way to get the results for a sample query – ‘from:Fossasia’ (Last 30 days tweets from FOSSASIA).

searchQuery: Query = {
    displayString: ‘from:FOSSASIA,
    queryString: ‘from:FOSSASIA’,
    routerString: ‘from:FOSSASIA’,
    filter: { video: false, image: false },
    location: null,
    timeBound: { since: null, until: null },
    from: true
}

 

Process discussed here can be used to get results in any class file within Loklak Search.

Using necessary imports

First step would be to add all necessary imports.

import { OnInit } from ‘@angular/core’;
import { Store } from ‘@ngrx/store’;
import { Observable } from ‘rxjs/Observable’;
import * as fromRoot from ‘../../reducers’;
import { Query } from ‘../../models/query’;
import * as searchAction from ‘../../actions/api’;
import { ApiResponseResult } from ‘../../models/api-response’;

 

Note: No need to import OnInit, if the class already implements it (OnInit is one of the Angular’s Lifecycle Hooks which controls the changes when the component containing it gets loaded initially).

The Query is used to define the type of searchQuery variable created above. And ApiResponseResult is the type of result which will be obtained on searching the searchQuery.

Declaring response variables and store object

Next step would be to declare the variables with the type of response and create store object of type Store with the current State of the application.

public isSearching$: Observable<boolean>;
public apiResponseResults$: Observable<ApiResponseResult[]>;
constructor(
    private store: Store<fromRoot.State>
) { }

 

isSearching$ holds an Observable which can be subscribed to get a boolean value which states the current status of searching of query. apiResponseResults$ holds the actual response of query.

Dispatching a SearchAction

The most crucial part comes here to dispatch a SearchAction with the created searchQuery as payload. A new method would be created with void as return type to dispatch the SearchAction.

private queryFromURL(): void {
    this.store.dispatch(
        new searchAction.SearchAction(this.searchQuery)
    );
}

Selecting and storing results from store

A new method would be created to store the current searching status along with the response of input query in isSearching$ and apiResponseResults$ respectively. Now the selector for the reducer function corresponding to the search and api response entities would be called.

private getDataFromStore(): void {
    this.isSearching$ = 
        this.store.select(fromRoot.getSearchLoading);
    this.apiResponseResults$ =   
        this.store.select(
            fromRoot.getApiResponseEntities
    );
}

Dispatching and Selecting on view Init

create ngOnInit() method, If you have not already done and call the respective dispatching and select method inside it respectively.

ngOnInit() {
    this.queryFromURL();
    this.getDataFromStore();
}

How should one test the results?

Call these lines inside any method and look for the results on browser console window.

console.log(this.isSearching$);
console.log(this.apiResponseResults$);

Resources

Adding ‘Scroll to Top’ in Loklak Search

It is really important for a Progressive Web Application like Loklak to be user friendly. To enhance the user experience and help user scroll up through the long list of results, a new ‘Scroll to Top’ feature has been added to Loklak.

Integration Process

The feature is built using HostListener and Inject property of Angular, which allows to use Window and Document functionalities similar to Javascript.

Adding HostListener

First step would be to import HostListener and Inject, and create property of document using with Inject.

import { HostListener, Inject } from ‘@angular/core’;
navIsFixed: boolean;
constructor(
    @Inject(DOCUMENT) private document: Document
) { }

 

Next step would be to create method to detect the current state of window, and scroll to the top of window.

@HostListener(‘window:scroll’, [])
onWindowScroll() {
    if (window.pageYOffset
        || document.documentElement.scrollTop
        || document.body.scrollTop > 100) {
                this.navIsFixed = true;
    } else if (this.navIsFixed && window.pageYOffset
        || document.documentElement.scrollTop
            || document.body.scrollTop < 10) {
                    this.navIsFixed = false;
            }
        } scrollToTop() {
            (function smoothscroll() {
                const currentScroll =
                    document.documentElement.scrollTop
                        || document.body.scrollTop;
            if (currentScroll > 0) {
            window.requestAnimationFrame(smoothscroll);
            window.scrollTo(0,
                currentScroll  (currentScroll / 5));
        }
    })();
}

 

The last step would be to add user interface part i.e. HTML and CSS component for the Scroll to Top feature.

Creating user interface

HTML

HTML portion contains a div tag with class set to CSS created and a button with click event attached to it to call the scrollToTop() method created above.

CSS

The CSS class corresponding to the div tag provides the layout of the button.

button {
   display: block;
   margin: 32px auto;
   font-size: 1.1em;
   padding: 5px;
   font-weight: 700;
   background: transparent;
   border: none;
   cursor: pointer;
   height: 40px;
   width: 40px;
   font-size: 30px;
   color: white;
   background-color: #d23e42;
   border-radius: 76px;
   &:focus {
       border: none;
       outline: none;
   }
}
.scroll-to-top {
  position: fixed;
  bottom: 15px;
  right: 15px;
  opacity: 0;
  transition: all .2s ease-in-out;
}
.show-scroll {
  opacity: 1;
  transition: all .2s ease-in-out;
}

 

The button style creates a round button with an interactive css properties. scroll-to-top class defines the behaviour of the div tag when user scrolls on the results page.

How to use?

Goto Loklak and search a query. On results page, scroll down the window to see upward arrow contained in a red circle like:

On clicking the button, user should be moved to the top of results page.

Resources

Adding Speech Component in Loklak Search

Speech recognition service for voice search is already embedded in Loklak. Now the idea is to use this service and create a new separate component for voice recognition with an interactive and user friendly interface. This blog will cover every single portion of an Angular’s redux based component from writing actions and reducers to the use of created component in other required components.

Creating Action and Reducer Function

The main idea to create an Action is to control the flow of use of Speech Component in Loklak Search. The Speech Component will be called on and off based on this Action.

Here, the first step is to create speech.ts file in actions folder with the following code:

import { Action } from '@ngrx/store';

export const ActionTypes = {
   MODE_CHANGE: '[Speech] Change',
};

export class SearchAction implements Action {
   type = ActionTypes.MODE_CHANGE;

   constructor(public payload: any) {}
}

export type Actions
   = SearchAction;

 

In the above segment, only one action (MODE_CHANGE) has been created which is like a boolean value which is being returned as true or false i.e. whether the speech component is currently in use or not. This is a basic format to be followed in creating an Action which is being followed in Loklak Search. The next step would be to create speech.ts file in reducers folder with the following code:

import { Action } from '@ngrx/store';
import * as speech from '../actions/speech';
export const MODE_CHANGE = 'MODE_CHANGE';

export interface State {
   speechStatus: boolean;
}
export const initialState: State = {
   speechStatus: false
};
export function reducer(state: State = initialState,
   action: speech.Actions): State {
        switch (action.type) {
            case speech.ActionTypes.MODE_CHANGE: {
                const response = action.payload;
                return Object.assign({}, state,
                {speechStatus: response});
       }
       default: {
           return state;
       }
   }
}
export const getspeechStatus = (state: State) =>
    state.speechStatus;

 

It follows the format of reducer functions created in Loklak Search. Here, the main key point is the state creation and type of value it is storing i.e. State is containing a speechStatus of type boolean. Defining an initial state with speechStatus value false (Considering initially the Speech Component will not be in use). The reducer function a new state by toggling the input state based on the type of Action created above and it returns the input state by default. At last wrapping the state as a function and returning the state’s speechStatus value.

Third and last step in this section would be to create a selector for the above reducer function in the root reducer index file.

Import and add speech from speech reducer file into the general state in root reducer file. And at last export the created selector function for speech reducer.

import * as fromSpeech from './speech';
export interface State {
   ...
   speech: fromSpeech.State;
}
export const getSpeechState = (state: State) =>
    state.speech;
export const getspeechStatus = createSelector(
    getSpeechState, fromSpeech.getspeechStatus);

Creating Speech Component

Now comes the main part to create and define the functioning of Speech Component. For creating the basic Speech Component, following command is used:

ng generate component app/speech --module=app

 

It will automatically create and provide Speech Component in app.module.ts. The working structure of Speech Component has been followed as of Google’s voice recognition feature for voice searching. Rather than providing description of each single line of code the following portion will cover the main code responsible for the functioning of Speech Component.

Importing and defining speech service in constructor:

import {
    SpeechService
} from '../services/speech.service';
constructor(
       private speech: SpeechService,
       private store: Store<fromRoot.State>,
       private router: Router
   ) {
       this.resultspage =this.router.url
       .toString().includes('/search');
       if (this.resultspage) {
           this.shadowleft = '-103px';
           this.shadowtop = '-102px';
       }
       this.speechRecognition();
}
speechRecognition() {
    this.speech.record('en_US').subscribe(voice =>
        this.onquery(voice));
}

 

When the Speech Component is called, speechRecognition() method will start recording speech (It will use the record() method from speech service to record the user voice).

For fluctuating border height and color of voice search icon, a resettimer() method is created.

randomize(min, max) {
    let x;
    x = (Math.random() * (max - min) + min);
    return x;
}
resettimer(recheck: boolean = false) {
    this.subscription.unsubscribe();
    this.timer = Observable.timer(0, 100);
    this.subscription = this.timer.subscribe(t => {
    this.ticks = t;
        if (t % 10 === 0 && t <= 20) {
            this.buttoncolor = '#f44';
            this.miccolor = '#fff';
            this.borderheight =
                this.randomize(0.7, 1);
            if (this.resultspage) {
                this.borderheight =
                    this.randomize(0.35, 0.5);
            }
            if (!recheck) {
                this.resettimer(true);
            }
        }
        if (t === 20) {
            this.borderheight = 0;
        }
        if (t === 30) {
            this.subscription.unsubscribe();
            this.store.dispatch(new speechactions
            .SearchAction(false));
        }
    });
}

 

The randomize() method provides a random number between min and max value.

To put on check and display status as message on things like whether microphone is working, or user has spoken something, or if the speech is being recorded, based on the time elapsed in calling of speech component and actual voice recording, the following portion of code is written in ngOnInit() method.

ngOnInit() {
    this.timer = Observable.timer(1500, 2000);
    this.subscription = this.timer.subscribe(t => {
        this.ticks = t;
        if (t === 1) {
            this.message = 'Listening...';
        }
        if (t === 4) {
            this.message = 'Please check your 
            microphone and volume levels.';
            this.miccolor = '#C2C2C2';
        }
        if (t === 6) {
            this.subscription.unsubscribe();
            this.store.dispatch(new speechactions
                .SearchAction(false));
        }
    });
}

 

The logic can be understood as if the elapsed time is 1 sec, it means it is listening to the speaker’s voice. And if the elapsed time is 4 sec, it means there is something wrong and user will be asked to check for the microphone and volume levels. At last if it tends to 6 seconds, then the Speech Component will be called off with the dispatched Action as false which is defined above (That means it is no longer in use).

Embed Speech Component in main App Component

Now comes the last part to use the created featured component in the required place. Code below describes embedding Speech Component in App Component.

Import SpeechService and required modules.

import {
    SpeechService
} from './services/speech.service';
import { Observable } from 'rxjs/Observable';

 

hidespeech will be used to store the current status of Speech Component (whether its in use or not), and completeQuery$ and searchData store the voice recorded in form Observable and String. completeQuery$ is optional (If the Speech Component is unable to track voice of speaker by any means, then it will not contain any value and hence searchData will be empty).

hidespeech: Observable<any>;
completeQuery$: Observable<any>;
searchData: String;

 

Creating speech parameter in constructor and store the current status of speech and store it into hidespeech. Based on the subscribed value of hidespeech, speech service’s stoprecord() will be called (To stop recording when the speech recognition completes). After recording stops, store the whole query in completeQuery$.

constructor (
    private speech: SpeechService
) {
    this.hidespeech = store.select(
        fromRoot.getspeechStatus);
    this.hidespeech.subscribe(hidespeech => {
        if (!hidespeech) {
            this.speech.stoprecord();
        }
    });
    this.completeQuery$ = store.select(
        fromRoot.getQuery);
    this.completeQuery$.subscribe(data => {
        this.searchData = data;
    });
}

 

Add the Speech Component in app.component.html. Now the main logic of calling Speech Component will be based on the subscribed observable value of hidespeech (If false then call Speech Component else not).

<app-speech *ngIf="hidespeech|async"></app-speech>

Using Speech Component in Home and FeedHeader Component

Import Speech Service and speech Action created above, and create hidespeech to store the current status of Speech Component.

import * as speechactions from '../../actions/speech';
import {
    SpeechService
} from '../../services/speech.service';
hidespeech: Observable<boolean>;

 

Create speech parameter of type SpeechService and store the current status of Speech Component in hidespeech. Dispatch speechactions.SearchAction (payload as true) for inferring that the Speech Component is currently in use.

constructor(
    private speech: SpeechService
) {
    this.hidespeech = store
        .select(fromRoot.getspeechStatus);
}
speechRecognition() {
    this.store.dispatch(
        new speechactions.SearchAction(true));
}

How to use the Speech Component?

Goto Loklak and click on Voice Input Icon. It will popup a screen as below.

Now, speak something to search. E.g. Google, the screen will turn into something like below with the spelled value displayed on screen.

If something goes wrong (Microphone did not work, low volume levels or unrecognisable voice), then screen will show something like:

On successful recognition of speech, the query will be set and the results will be shown as

Similar process is being followed on results page to make a search query using voice.

Resources

Adding new test cases for increasing test coverage of Loklak Server

It is a good practice to have test cases covering major portion of actual code base. The idea was same to add new test cases in Loklak Server to increase its test coverage. The results were quite amazing with a significant increase of about 3% in total test coverage of the overall project. And about 80-100% increase in the test coverage of individual files for which tests have been written.

Integration Process

For integration, a total of 6 new test cases have been written:

  1. ASCIITest
  2. GeoLocationTest
  3. CommonPatternTest
  4. InstagramProfileScraperTest
  5. EventBriteCrawlerServiceTest
  6. LocationWiseTimeServiceTest

For increasing code coverage, Java docs have been written which increase the lines of code being covered.

Implementation

Basic implementation of adding a new test case is same. Here’s is an example of EventBriteCrawlerServiceTest implementation. This can be used as a reference for adding a new test case in the project.

Prerequisite: If the test file being written tests any additional external service (e.g. EventBriteCrawlerServiceTest tests any event being created on EventBrite platform) then, a corresponding new test service or test event should be written beforehand on the given platform. For EventBriteCrawlerServiceTest, a test-event has been created.

The given below steps will be followed for creating a new test case (EventBriteCrawlerServiceTest), assuming the prerequisite step has been done:

  • A new java file is generated in test/org/loklak/api/search/ as EventBriteCrawlerServiceTest.java.
  • Define package for the test file and import EventBriteCrawlerService.java file which has to be tested along with necessary packages and methods.

package org.loklak.api.search;

import org.loklak.api.search.EventBriteCrawlerService;
import org.loklak.susi.SusiThought;
import org.junit.Test;
import org.json.JSONArray;
import org.json.JSONObject;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.assertNotNull;

 

  • Define the class and methods that need to be tested. It is a good idea to break the testing into several test methods rather than testing more features in a single method.

public class EventBriteCrawlerServiceTest {
  @Test
  public void apiPathTest() { }

  @Test
  public void eventBriteEventTest() { }

  @Test
  public void eventBriteOrganizerTest() { }

  @Test
  public void eventExtraDetailsTest() { }
}

 

  • Create an object of EventBriteCrawlerService in each test method.

EventBriteCrawlerService eventBriteCrawlerService = new EventBriteCrawlerService();

 

  • Call specific method of EventBriteCrawlerService that needs to be tested in each of the defined methods of test file and assert the actualresult to the expectedresult.

@Test
public void apiPathTest() {
    EventBriteCrawlerService eventBriteCrawlerService = 
        new EventBriteCrawlerService();

     assertEquals("/api/eventbritecrawler.json",
        eventBriteCrawlerService.getAPIPath());
}

 

  • For methods fetching an actual result (Integration tests) of event page, define and initialise an expected set of results and assert the expected result with the actual result by parsing the json result.

@Test
public void eventBriteOrganizerTest() {
    EventBriteCrawlerService eventBriteCrawlerService =
        new EventBriteCrawlerService();
    String eventTestUrl = "https://www.eventbrite.com/e/
        testevent-tickets-46017636991";
    SusiThought resultPage = eventBriteCrawlerService
        .crawlEventBrite(eventTestUrl);
    JSONArray jsonArray = resultPage.getData();
    JSONObject organizer_details =
        jsonArray.getJSONObject(1);
    String organizer_contact_info = organizer_details
        .getString("organizer_contact_info");
    String organizer_link = organizer_details
        .getString("organizer_link");
    String organizer_profile_link = organizer_details
        .getString("organizer_profile_link");
    String organizer_name = organizer_details
        .getString("organizer_name");

    assertEquals("https://www.eventbrite.com
        /e/testevent-tickets-46017636991
        #lightbox_contact",
        organizer_contact_info);
    assertEquals("https://www.eventbrite.com
        /e/testevent-tickets-46017636991
        #listing-organizer", organizer_link);
    assertEquals("", organizer_profile_link);
    assertEquals("aurabh Srivastava", organizer_name);
}

 

  • If the test file is testing the harvester, then import and add the test class in TestRunner.java file. e.g.

import org.loklak.harvester.TwitterScraperTest;

@RunWith(Suite.class)
@Suite.SuiteClasses({
  TwitterScraperTest.class
})

Testing

The last step will be to check if the test case passes with integration of other tests. The gradle will test the test case along with other tests in the project.

./gradlew build

Resources

Adding ‘Voice Search’ Feature in Loklak Search

It is beneficial to have a voice search feature embedded in a search engine like loklak.org based on PWA (Progressive Web Application) architecture. This will allow users to make query using voice on phone or computer. For integrating voice search, JavaScript Web Speech API (also known as webkitSpeechRecognition) is used which allows us to add speech recognition feature in any website or a web application.

Integration Process

For using webkitSpeechRecognition, a separate typescript service is being created along with its relevant unit test

  1. speech.service.ts
  2. speech.service.spec.ts

The main idea to implement the voice search was

  1. To create an injectable speech service.
  2. Write methods for starting and stopping voice record in it.
  3. Import and inject the created speech service into root app module.
  4. Use the created speech service into required HomeComponent and FeedHeaderComponent.

Structure of Speech Service

The speech service mainly consists of two methods

  • record() method which accepts lang as string input which specifies the language code for speech recording and returns a string Observable.

   record(lang: string): Observable<string> {
           return Observable.create(observe => {
                 const webkitSpeechRecognition }: IWindow = 
                        <IWindow>window;
                 this.recognition = new webkitSpeechRecognition();
                 this.recognition.continuous = true;
                 this.recognition.interimResults = true;
                 this.recognition.onresult = take => 
                 this.zone.run(()                            
                 observe.next(take
                 .results.item
                 (take.results.
                 length - 1).
                 item(0).
                 transcript);
                 this.recognition
                 .onerror = 
                 err => observe
                 .error(err);
                 this.recognition.onend 
                 = () => observe
                 .complete();
                 this.recognition.lang = 
                 lang;
                 this.recognition.start( 
             );
        });
     }

 

Using observe.complete() allows speech recognition action to stop when the user stops speaking.

  • stoprecord() method which is used to stop the current instance of recording.

 stoprecord() {
       if (this.recognition) {
           this.recognition.stop();
       }
   }

 

stop() method is used to stop the current instance of speech recognition.

Using Speech Service in required Component

In Loklak Search, the speech service have been included in two main components i.e. HomeComponent and FeedHeaderComponent. The basic idea of using the created speech service in these components is same.

In the TypeScript (not spec.ts) file of the two components, firstly import the SpeechService and create its object in constructor.

constructor( private speech: SpeechService ) { }

 

Secondly, define a speechRecognition() method and use the created instance or object of SpeechService in it to record speech and set the query as the recorded speech input. Here, default language has been set up as ‘en_US’ i.e. English.

 stoprecord() {
       if (this.recognition) {
           this.recognition.stop();
       }
   }

 

After user clicks on speech icon, this method will be called and the recorded speech will be set as the query and there will be router navigation to the /search page where the result will be displayed.

Resources

Installing the Loklak Search and Deploying it to Surge

The Loklak search creates a website using the Loklak server as a data source. The goal is to get a search site, that offers timeline search as well as custom media search, account and geolocation search.

In order to run the service, you can use the API of http://api.loklak.org or install your own Loklak server data storage engine. Loklak_server is a server application which collects messages from various social media tweet sources, including Twitter. The server contains a search index and a peer-to-peer index sharing interface. All messages are stored in an elasticsearch index.

The site of this repo is deployed on the GitHub gh-pages branch and automatically deployed here: http://loklak.org

In this blog, we will talk about how to install Loklak_Search locally and deploying it to Surge (Static web publishing for Front-End Developers).

How to clone the repository

Sign up / Login to GitHub and head over to the Loklak_Search repository. Then follow these steps.

  1. Go ahead and fork the repository
https://github.com/fossasia/loklak_search
  1.   Get the clone of the forked version on your local machine using
git clone https://github.com/<username>/loklak_search.git
  1.   Add upstream to synchronize repository using
git remote add upstream https://github.com/fossasia/loklak_search.git

Getting Started

The Loklak search application basically consists of the following :

  1. First, we will need to install angular-cli by using the following command:
npm install -g @angular/[email protected]

2. After installing angular-cli we need to install our required node modules, so we will do that by using the following command:

npm install

3. Deploy locally by running this

ng serve

Go to localhost:4200 where the application will be running locally.

How to Deploy Loklak Search on Surge :

Surge is the technology which publishes or generates the static web-page demo link, which makes it easier for the developer to deploy their web-app. There are a lot of benefits of using surge over generating demo link using GitHub pages.

  1. We need to install surge on our machine. Type the following in your Linux terminal:
npm install –global surge

This installs the Surge on your machine to access Surge from the command line.

  1. In your project directory just run
surge
  1. After this, it will ask you three parameters, namely
Email
Password
Domain

After specifying all these three parameters, the deployment link with the respective domain is generated.

Auto deployment of Pull Requests using Surge :

To implement the feature of auto-deployment of pull request using surge, one can follow up these steps:

  • Create a pr_deploy.sh file
  • The pr_deploy.sh file will be executed only after success of Travis CI i.e. when Travis CI passes by using command bash pr_deploy.sh
#!/usr/bin/env bash
if [ “$TRAVIS_PULL_REQUEST” == “false” ]; then
echo “Not a PR. Skipping surge deployment.”
exit 0
fi
npm i -g surge
export [email protected]
# Token of a dummy account
export SURGE_TOKEN=d1c28a7a75967cc2b4c852cca0d12206
export DEPLOY_DOMAIN=https://pr-${TRAVIS_PULL_REQUEST}-fossasia-LoklakSearch.surge.sh
surge –project ./dist –domain $DEPLOY_DOMAIN;

Here, Travis CI is first installing surge locally by npm i -g surge  and then we are exporting the environment variables SURGE_LOGIN , SURGE_TOKEN and DEPLOY_DOMAIN.

Now, execute pr_deploy.sh file from .travis.yml by using command bash pr_deploy.sh

Resources