Add RSS feed and JSON output based on type specified with query param

The idea behind writing this blog post is to discuss the method on how RSS feed and JSON output sources have been included in loklak to provide respective data sources based on the type specified with query as a parameter.

Accessing Current Query in Info-box

Accessing link to RSS feed and JSON output of loklak requires the Query to be passed as a value with parameter ‘q’ (e.g. api/search.json?q=FOSSASIA or api/search.rss?q=FOSSASIA). In order to represent links as buttons in Info-box at sidebar of loklak.org, current Query needs to be accessed/stored inside Info-box from ngrx store.

public stringQuery;
...
this.store.select(fromRoot.getQuery).subscribe(
    query => this.stringQuery = query.displayString);

 

Firstly stringQuery variable is created to store the current Query from store. As the Query can be changed in store (User might search for several Queries), storing of current Query needs to be done inside ngOnChanges().

Checking type associated with Query

There are various types associated with Query to get different type of results like ‘from:FOSSASIA will give results specifically from ‘FOSSASIA’ which will have different query parameter from other types like ‘@FOSSASIA’ or ‘#FOSSASIA’. To assign appropriate Query parameter to each of these types, we need to check the Query pattern to apply Query param based on the type.

import { hashtagRegExp, fromRegExp, mentionRegExp }
      from ‘../../utils/reg-exp’;
...

if ( hashtagRegExp.exec(this.stringQuery) !== null ) {
	// Check for hashtag this.stringQuery
	this.queryString = ‘%23 + hashtagRegExp.exec(
	this.stringQuery)[1] + '' + hashtagRegExp.exec(
	this.stringQuery)[0];
} else if ( fromRegExp.exec(this.stringQuery) !== null ) {
	// Check for from user this.stringQuery
	this.queryString = ‘from%3A’ + fromRegExp.exec(
	this.stringQuery)[1];
} else if ( mentionRegExp.exec(this.stringQuery) !== null ) {
	// Check for mention this.stringQuery
	this.queryString = ‘%40 + mentionRegExp.exec(
	this.stringQuery)[1];
} else {
	// for other queries
	this.queryString = this.stringQuery;
}

 

Note: hashtagRegExp, fromRegExp, mentionRegExp are the utility functions created to match the pattern of given string (Query) in order to classify the preceding type associated with Query. These are provided here as a reference, which can be used to add more of the types.

Passing current queryString in RSS and JSON link

General link for both RSS and JSON data remains same for each Query passed, only the type associated would be changed in the Query value. So representing the link in an anchor tag in UI would be as –

<a class=“data rss” href=“
		http://api.loklak.org/api/search.
		rss?timezoneOffset=-330&q=
		{{stringQuery}}” target=“_blank”>
</a>
<a class=”data json” href=”http://api.
		loklak.org/api/search
		.json?timezoneOffset=-330&q=
		{{stringQuery}}” target=”_blank“>
</a>

 

{{stringQuery}} is the actual query parameter to be passed to get the required results.

Testing RSS feed and JSON data

Search for a query on loklak, and click on the the RSS or JSON button below sidebar on results page and compare with the results.

RSS and JSON button should be similar to –

Resources

Continue Reading

Holding query search in Loklak

Continuously searching user’s input while the user kept typing is not good. It has certain drawbacks which result in continuous action dispatch and loading of the result page which depicts the site to be slow, but actually it is not. To prevent continuous loading and searching while user types a query in Loklak Search, it was important to keep hold on query until user completes typing and then start searching for the results. I am writing this blog post to emphasise a proper solution to constantly keep track of the query and search on an appropriate event.

Adding keypress event

First and foremost task would be to add keypress event to input tag on home and results page.

<input     
     

     (keypress)=“onEnter($event)” 
     [formControl]=“searchInputControl

/>

 

formControl will constantly keep track of the user input and will be stored in searchInputControl, while keypress event will activate onEnter() method firing an $event as an input to onEnter() method.

Adding onEnter() method

Note: Keeping track of special keypress event code 13 which is equivalent for an enter key press.

Add a new method onEnter() with event as an input which will execute its function only when the event’s keypress code is 13 (Enter key press). One point should be noted here that query to be searched should not be empty or containing leading and trailing spaces or should not contain only spaces (treated as an incomplete query to be searched).

onEnter(event: any) {  
   if (event.which === 13) {  

      if (this.searchInputControl.value.trim() !== ”) {

        // Actual work of query search will go here.

  }

}

Add query searching and migration code under onEnter()

Now comes the last part to shift query search and route migration code from setupSearchField() to the portion mentioned above in comments. We need to take one point in consideration that from now on we actually do not need to subscribe to formControl in ngOnInit() since now it depends on user query input.

Complete onEnter() in home component file

After completion, onEnter() method should be similar to:

onEnter(event: any) {
   if (event.which === 13) {
       if (this._queryControl.value.trim() !== ”) {
           this.store.dispatch(new 
               queryAction.RelocationAfterQuerySetAction());
           this.store.dispatch(new suggestAction
               .SuggestAction(this._queryControl.value.trim()));
           this.store.dispatch(new queryAction
               .InputValueChangeAction(this._queryControl.value.trim()
           ));
           this.router.navigate([`/search`],
               { queryParams: { query: this._queryControl.value.trim() }
               , skipLocationChange: true } );
           this.store.dispatch(new titleAction
           .SetTitleAction(this._queryControl.value.trim()
               +   Loklak Search’));
           this.getDataFromStore();
       }
   }
}

 

Since we are not depending on ngOnInit() therefore we also need to append getDataFromStore() to onEnter() method.

Complete onEnter() in feed-header component file

After completion, onEnter() method should be similar to:

onEnter(event: any) {
   if (event.which === 13) {
       if (this.searchInputControl.value.trim() !== '') {
           this.searchEvent.emit(this.searchInputControl
               .value.trim());
           this.setupSuggestBoxClosing();
       }
   }
}

Testing

Try typing different formats of query on home and results page of loklak.

Resources

Continue Reading

Fixing Infinite Scroll Feature for Susper using Angular

In Susper, we faced a unique problem. Every time the image tab was opened, and the user scrolled through the images, all the other tabs in the search engine, such as All, Videos etc, would stop working. They would continue to display image results as shown:

Since this problem occurred only when the infinite scroll action was called in the image tab, I diagnosed that the problem probably was in the url parameters being set.

The url parameters were set in the onScroll() function as shown:

onScroll () {
let urldata = Object.assign({}, this.searchdata);
this.getPresentPage(1);
this.resultDisplay = ‘images’;
urldata.start = (this.startindex) + urldata.rows;
urldata.fq = ‘url_file_ext_s:(png+OR+jpeg+OR+jpg+OR+gif)’;
urldata.resultDisplay = this.resultDisplay;
urldata.append = true;
urldata.nopagechange = true;
this.store.dispatch(new queryactions.QueryServerAction(urldata));
};

The parameters append and nopagechange were to ensure that the images are displayed in the same page, one after the other.
To solve this bug I first displayed the query call each time a tab is clicked on the web console.
Here I noticed that for the tab videos, nopagechange and append attributes still persisted, and had not been reset. The start offset had not been set to 0 either.
So adding these few lines before making a query call from any tab, would solve the problem.

urldata.start = 0;
urldata.nopagechange = false;
urldata.append = false;

Now the object is displayed as follows:

Now videos are displayed in the videos tab, text in the text tab and so on.
Please refer to results.component.ts for the entire code.

References:

  1. On how to dispatch queries to the store: https://gist.github.com/btroncone/a6e4347326749f938510
  2. Tutorial on the ngrx suite:http://bodiddlie.github.io/ng-2-toh-with-ngrx-suite/
Continue Reading

Implementing Sort By Date Feature In Susper

 

Susper has been given ‘Sort By Date’ feature which provides the user with latest results with the latest date. This feature enhances the search experience and helps users to find desired results more accurately. The sorting of results date wise is done by yacy backend which uses Apache Solr technology.

The idea was to create a ‘Sort By Date’ feature similar to the market leader. For example, if a user searches for keyword ‘Jaipur’ then results appear to be like this:

If a user wishes to get latest results, they can use ‘Sort By Date’ feature provided under ‘Tools’.

The above screenshot shows the sorted results.

You may however notice that results are not arranged year wise. Currently, the backend work for this is being going on Yacy and soon will be implemented on the frontend as well once backend provide us this feature.

Under ‘Tools’ we created an option for ‘Sort By Date’ simply using <li> tag.

<ul class=dropdownmenu>
  <li (click)=filterByDate()>Sort By Date</li>
</ul>

When clicked, it calls filterByDate() function to perform the following task:

filterByDate() {
  let urldata = Object.assign({}, this.searchdata);
  urldata.query = urldata.query.replace(/date, “”);
  this.store.dispatch(new queryactions.QueryServerAction(urldata));
}
Earlier we were using ‘last_modified desc’ attribute provided by Solr for sorting out dates in descending order. In June 2017, this feature was deprecated with a new update of Solr. We are using /date attribute in query for sorting out results which is being provided by Solr.

 

Continue Reading

Doing a table join in Android without using rawQuery

The Open Event Android App, downloads data from the API (about events, sessions speakers etc), and saves them locally in an SQLite database, so that the app can work even without internet connection.

Since there are multiple entities like Sessions, Speakers, Events etc, and each Session has ids of speakers, and id of it’s venue etc, we often need to use JOIN queries to join data from two tables.

 

Android has some really nice SQLite helper classes and methods. And the ones I like the most are the SQLiteDatabase.query, SQLiteDatabase.update, SQLiteDatabase.insert ones, because they take away quite a bit of pain for typing out SQL commands by hand.

But unfortunately, if you have to use a JOIN, then usually you have to go and use the SQLiteDatabase.rawQuery method and end up having to type your commands by hand.

But but but, if the two tables you are joining do not have any common column names (actually it is good design to have them so – by having all column names prefixed by tablename_ maybe), then you can hack the usual SQLiteDatabase.query() method to get a JOINed query.

Now ideally, to get the Session where speaker_id was 1, a nice looking SQL query should be like this –

SELECT * FROM speaker INNER JOIN session
ON speaker_id = session_speaker_id
WHERE speaker_id = 1

Which, in android, can be done like this –

String rawQuery = "SELECT * FROM " + SpeakerTable.TABLE_NAME + " INNER JOIN " + SessionTable.TABLE_NAME
        + " ON " + SessionTable.EXP_ID + " = " + SpeakerTable.ID
        + " WHERE " + SessionTable.ID + " = " +  id;
Cursor c = db.rawQuery(
        rawQuery,
        null
);

But of course, because of SQLite’s backward compatible support of the primitive way of querying, we turn that command into

SELECT *
FROM session, speaker
WHERE speaker_id = session_speaker_id AND speaker_id = 1

Now this we can write by hacking the terminology used by the #query() method –

Cursor c = db.query(
        SessionTable.TABLE_NAME + " , " + SpeakerTable.TABLE_NAME,
        Utils.concat(SessionTable.PROJECTION, SpeakerTable.PROJECTION),
        SessionTable.EXP_ID + " = " + SpeakerTable.ID + " AND " + SpeakerTable.ID + " = " +  id,
        null,
        null,
        null,
        null
);

To explain a bit, the first argument String tableName can take table1, table2 as well safely, The second argument takes a String array of column names, I concatenated the two projections of the two classes. and finally, put by WHERE clause into the String selection argument.

You can see the code for all database operations in the android app here  https://github.com/fossasia/open-event-android/blob/master/android/app/src/main/java/org/fossasia/openevent/dbutils/DatabaseOperations.java

Continue Reading
Close Menu