Extending the News Feature to Show results from multiple organisations

News Tab in Susper was earlier implemented to show results only from a single organisation using  site: modifier for query facet. In this blog I will discuss about how I have modified the current News Tab to show results from various News organisation like BBC, Al Jazeera, The Guardian etc.

Implementation:

Step 1:

Creating a JSON file to store organisations detail:

We need to decide from which organisations  we will fetch results using YaCy Server and then display it in Susper. We have provided a JSON file where user can add or delete News sources easily. The results will only limited to the organisations which are present in the JSON file.

Step 2:

Creating a service to fetch details from JSON file:

Now after creating the the JSON file we need a service which will fetch results from the JSON file according to our need. This service will be a simple  Angular Service having a class GetJsonService and it will access the newsFile.json and map the results in JSON format. Here is the service which does this task for us.

Step 3:

Creating a service to fetch news accordingly:

Now after fetching the JSON result we need a service to fetch the News Results from YaCy Server. I have created a separate service to do this task where I have fetched results using site: modifier from each organisation and returned the results.The code for the news.service.ts is below.

export class NewsService {
 constructor(private jsonp: Jsonp) { }
 getSearchResults(searchquery, org) {
   let searchURL = 'https://yacy.searchlab.eu/solr/select?query=';
   searchURL += searchquery.query + ' site:' + org;
   let params = new URLSearchParams();
   for (let key in searchquery) {
     if (searchquery.hasOwnProperty(key)) {
       params.set(key, searchquery[key]); } }
  //Set other parameters
   return this.jsonp
     .get(searchURL, {search: params}).map(res =>
       res.json()[0]
     ).catch(this.handleError); }

 

Step 4:

Updating the results section:

Now we have a service that gives results from a single organisation and a JSON list of organisations. Now in results.component.ts we can simply subscribe to getJsonService and in a loop we will call getNewsService by changing the organisation in every iteration. We will then check that whether we are getting the valid results or not (undefined). The results which are not valid can cause errors when we will try to read any field of an undefined variable. Then, We will simply append the 2 result items from each organisation in an empty array and later use this array to show results.

this.getJsonService.getJSON().subscribe(res => { this.newsResponse = [];
for (let i = 0; i < res.newsOrgs.length; i++) { this.getNewsService.getSearchResults(querydata,res.newsOrgs[i].provider).subscribe( response => {
    if (response.channels[0].items[0] !== undefined) {
    this.newsResponse.push(response.channels[0].items[0]); }
    if (response.channels[0].items[1] !== undefined) {
    this.newsResponse.push(response.channels[0].items[1]); } } ); }
    });

 

The newsClick() function is activated on clicking News Tab and it updates the query and its details in store.

Step 5

Displaying the results:

Now we will modify the results.component.html to show results from new newsResponse array which have 2 results each from 5 organisations.

For this we will use each item of newsResponse using *ngFor and display its title,link and description in html template. We will also use [style.color] property of our element and set the color according to theme.

<div *ngFor="let item of newsResponse" class="result"> <div class="title">
<a class="title-pointer" href="{{item.link}}" [style.color]="themeService.titleColor">{{item.title}}</a>
</div> <div class="link">
<p  [style.color]="themeService.linkColor">{{item.link}}</p>
 </div> </div>

 

Here is the view of Susper’s News Tab where we are getting results from 5 different organisations.

Resources

  1. YaCy Modifiers: http://www.yacy-websuche.de/wiki/index.php/En:SearchParameters
  1. Angular Services: https://angular.io/tutorial/toh-pt4
  2. Reading JSON data in Angular: https://stackoverflow.com/questions/43275995/angular4-how-do-access-local-json

 

Implementing news feature in loklak

The idea is to bring out an useful feature out of the enriched high quality of data provided by api.loklak.org. Through this blog post, I’ll be discussing about the implementation of a news feature similar to Google to provide latest news relevant to the query in loklak.

Creating news Component

First step in implementing news feature would be to create a news component which would be rendered on clicking of news tab.

ng g component feed/news

Creating an action & reducer for news status

This step involves creating an action & reducer function for tracking the status of news component on click of news tab. When the user will click on news tab, corresponding action would be dispatched to display the news component in feed.

import { Action } from ‘@ngrx/store’;
export const ActionTypes = {
    NEWS_STATUS: ‘[NEWS] NEWS STATUS’
};
export class NewsStatusAction implements Action {
    type = ActionTypes.NEWS_STATUS;
    constructor (public payload: boolean) { }
}
export type Actions
    = NewsStatusAction;

 

Corresponding reducer function for news status to store current news status would be:

export function reducer(state: State = initialState, 
   action: newsStatusAction.NewsStatusAction): State {
switch (action.type) {
    case newsStatusAction.ActionTypes.NEWS_STATUS: {
    const newsStatusPayload: boolean = action.payload;
    return Object.assign({}, state, {
                newsStatus: newsStatusPayload
            });
        }
        default: {
            return state;
        }
    }
}

 

I would not be getting into each line of code here, only the important portions would be discussed here.

Creating news-org to store the names of news organizations

A new file news-org would be created inside app/shared/ folder containing list of news organizations in an array newsOrgs from which we would extract and filter results.

export const newsOrgs = [
    ‘CNN’,
    ‘nytimes’
];

 

Note: This is a dynamic implementation of news service and hence, the list of news organizations can be easily modified/updated here.

We would also need to export the newsOrgs in the index file inside shared/ folder as:

export { newsOrgs } from ‘./news-org’;

Creating ngrx action, reducer & effect for news success

Now comes the main part. We would need an action to dispatch on successful response of news search and corresponding reducer to keep appending the async news results.

News action would be created following the simple ngrx composition as:

import { Action } from ‘@ngrx/store’;
import { ApiResponse } from ‘../models/api-response’;
export const ActionTypes = {
    NEWS_SEARCH_SUCCESS: ‘[NEWS] SEARCH SUCCESS’
};
export class NewsSearchSuccessAction implements Action {
    type = ActionTypes.NEWS_SEARCH_SUCCESS;
    constructor (public payload: ApiResponse) { }
}
export type Actions
    = NewsSearchSuccessAction;

 

One of the important parts of news implementation is the reducer function corresponding to the news success action. We would actually append the async results provided by news effect into the store.

import * as newsSuccessAction from ‘../actions/newsSuccess’;
import { ApiResponseResult } from ‘../models’;

export interface State {
newsResponse: ApiResponseResult[];
}
export const initialState: State = {
newsResponse: [],
};
export function reducer(state: State = initialState,
   action: newsSuccessAction.NewsSearchSuccessAction): State {
    switch (action.type) {
        case newsSuccessAction.ActionTypes.NEWS_SEARCH_SUCCESS: {
            // Appending the news result
            return Object.assign({}, state, {
                newsResponse: […action.payload.statuses]
            });
        }
        default: {
            return state;
        }
    }
}
export const getNewsResponse = (state: State) => state.newsResponse;

 

Now, we would create an effect which would actually use the search service to fetch results for the list of organizations one by one will keep dispatching a corresponding action to append the response into the store to be displayed in news feed. Instead of showing whole code for the effect, only the main logic is being displayed.

import { newsOrgs } from ‘../shared/news-org’;
...
@Injectable()
export class DisplayNewsEffects {
    @Effect({ dispatch: false })
        searchNews$: Observable<void> = this.actions$
        .pipe(
        ofType(
        newsAction.ActionTypes.NEWS_STATUS
        ),
        map((action) => {
            if (action[‘payload’]) {
                const orgs = newsOrgs;
                orgs.forEach((org) => {
                const searchServiceConfig:
                    SearchServiceConfig = 
                    new SearchServiceConfig();
                this.apiSearchService
                .fetchQuery(‘from:’
                  + org, searchServiceConfig)
                    .subscribe(response =>
                this.store$.dispatch(
                     new newsStatusAction
                     .NewsSearchSuccessAction(response)));
                });
            }
        })
    );

Filtering the news response inside news component

The basic idea is to filter the results from news response which contain the query and display them using news template on click of news tab on results page of loklak.

import { Query } from ‘../../models/query’;
import { Observable } from ‘rxjs’;
...
public query: string;
public query$: Observable<Query>;
...
ngOnInit() {
   const texts = [];
   this.store.select(fromRoot.getQuery).subscribe(res => 
       this.query = res.displayString);
   this.query$ = this.store.select(fromRoot.getQuery);
   this.store.select(fromRoot.getNewsResponse).subscribe(
       v => {
       for ( let i = 0; i < v.length; i++ ) {
           this.newsResponse.push(v[i]);
           if (v[i][‘text’].includes(this.query
               .replace(/\s/g, ”).toLowerCase())) {
               if (!texts.includes(v[i][‘text’]
                   .replace(/\s/g, ”).toLowerCase())) {
                   texts.push(v[i][‘text’]
                       .replace(/\s/g, ”).toLowerCase());
                   this.newsResponse.push(v[i]);
               }
           }
       }
   });
}

 

In above configuration, firstly we needed the current query which we extracted from ngrx store and then checking whether the text of news results contain the query string or not. If the query is present inside the text, we would add it into an array which would be used to display news feed using feed-card.

Displaying the news response

Displaying the news response is very easy, we just need to pass the news response items with index inside feed-card component which would display the news feed similar to the normal feed under the news section.

<div class=“wrapper feed-results”>
   <div *ngFor=“let item of newsResponse; let i = index “>
     <feed-card [feedItem]=“item” [feedIndex]=“i”></feed-card>
   </div>
</div>

Creating News tab to display news feed

To provide a tab to user to access news feed, we would need to create a News tab along with the other searching tools inside feed-advanced-search component as:

<button [class.selected]=“selectedTab === ‘news'” class=“tab” value=“news”
   (click)=“getFilterResults($event.currentTarget.value)”>
   News
</button>

 

Note: Complete code for news implementation can be found here and the news filter can be found here.

Testing news feature

Search a query on loklak, and then click on news tab to get the latest relevant news matching the query.

Resources

Adding News Tab in Susper

Most of the current search engines have a News tab where results are displayed which are fetched from different News Organisations. The latest results are displayed on top followed by the older ones. Current market leader, Google even uses AI and Machine Learning to analyze the contents before delivering it as results to the user. In this blog, I will describe how I have implemented a basic News Tab in Susper which show news results from dailymail.co.uk . This News tab can be improved in future to match the market leader.

Implementation of News tab in Susper:

Implementation of News tab in UI:

To implement News tab the current design pattern has been followed in Susper. Here is the code to show News tab on the results page.

<ul type="none" id="search-options">
       <li [class.active_view]="Display('news')" (click)="newsClick()">News</li>

Angular attribute binding is used to bind the active_view property with the “Display(‘news’)” function and the click event is associated with “newsClick()” function.

Here is the code for newsClick() function which filters the result using ‘site:’ parameter of yacy and then dispatches the ‘urldata’ object to ‘QueryServerAction(urldata)’ function in Query Action.

newsClick() {
   let urldata = Object.assign({}, this.searchdata);
   this.getPresentPage(1);
   this.resultDisplay = 'news';
   delete urldata.fq;
   urldata.rows = 10;
   if (urldata.query.substr(urldata.query.length - 25, 25) !== " site:www.dailymail.co.uk") {
     urldata.query += " site:www.dailymail.co.uk";
   } else {
     urldata.query += "";
   }
   urldata.resultDisplay = this.resultDisplay;
   this.store.dispatch(new queryactions.QueryServerAction(urldata));
   }

Here is the code for Query Action in query.ts file. It has two actions QueryAction and QueryServerAction

export const ActionTypes = {
 QUERYCHANGE: type('[Query] Change'),
 QUERYSERVER: type('[Query] Server'),
};
export class QueryAction implements Action {
 type = ActionTypes.QUERYCHANGE;

 constructor(public payload: any) {}
}
export class QueryServerAction implements Action {
 type = ActionTypes.QUERYSERVER;
 constructor(public payload: any) {}
}
export type Actions
 = QueryAction|QueryServerAction ;

Now the reducer takes either of the two query actions and the current state and returns the new state to the store.

Here is the code for the reducer query.ts

export const CHANGE = 'CHANGE';
export interface State {
 query: string;
 wholequery: any;
}
const initialState: State = {
 query: '',
 wholequery: {
   query: '',
   rows: 10,
   start: 0,
   mode: 'text'
 },
};
export function reducer(state: State = initialState, action: query.Actions): State {
 switch (action.type) {
   case query.ActionTypes.QUERYCHANGE: {
     const changeQuery = action.payload;
     return Object.assign({}, state, {
       query: changeQuery,
       wholequery: state.wholequery
     });
   }
   case query.ActionTypes.QUERYSERVER: {
     let serverQuery = Object.assign({}, action.payload);
     let resultCount = 10;
     if (localStorage.getItem('resultscount')) {
       resultCount = JSON.parse(localStorage.getItem('resultscount')).value || 10;
     }
     let instantsearch = JSON.parse(localStorage.getItem('instantsearch'));

     if (instantsearch && instantsearch.value) {
       resultCount = 10;
     }

     serverQuery.rows = resultCount;
     return Object.assign({}, state, {
       wholequery: serverQuery,
       query: state.query
     });
   }
   default: {
     return state;
   }
 }
}
export const getpresentquery = (state: State) => state.query;
export const getpresentwholequery = (state: State) => state.wholequery;

 

From the store the modified query is available to all other components of the appand is used by search service to get the filtered results.

The search results are then stored in observable items$ in results.components.ts and is then used in results.components.html to display results under News tab.

Here is the code to display the results.

 <div class="feed container">
     <div *ngFor="let item of items$|async" class="result">
       <div class="title">
         <a class="title-pointer" href="{{item.link}}">{{item.title}}</a>
       </div>
       <div class="link">
         <p>{{item.link}}</p>
       </div>
     </div>
   </div>

 

Resources

1.YaCy Search Parameters: http://www.yacywebsearch.net/wiki/index.php/En:SearchParameters

2.Redux in Angular: http://blog.ng-book.com/introduction-to-redux-with-typescript-and-angular-2/

3.Corresponding PR: https://github.com/fossasia/susper.com/pull/1023