Getting results for Multi-word Query and showing limited results in Knowledge Graph

Post author:praveenojha33
Post published:August 20, 2018
Post category:FOSSASIA GSoC
Post comments:0 Comments

In Susper knowledge graph,we were unable to get results for multi word query from Wikipedia API and also we were unable to decide how much information should be shown in knowledge graph which was retrieved from Wikipedia API.For example for a query donald trump we do not get information (https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=donald%20trump) . Also for searching for any query like india (https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=india) we get a lot of information. Earlier we used Angular slice pipe to display only 600 characters in knowledge graph (Infobox) but almost for all queries the sentences were terminated before it was completed. In this blog, I will describe how we solved both problems in knowledge graph.

Getting results for multi word query from Wikipedia API:

On searching a lot we found that the results were present on Wikipedia for multi word queries but these queries must have its starting letters as capital letters. For example we do not get results for queries such as donald trump

i.e https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=donald%20trump

but when we make a query for Donald Trump

i.e

https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=Donald%20Trump

But since in most of the search queries users use small letters, we need to convert the query such that each word starts with a capital letter. Since the Knowledge Graph has been implemented using ngrx pattern, this logic was easily implemented in knowledge effects. For this we will be selecting each word in query by using regular expression and then we will use toUppercase() and toLowerCase() methods of javascript and will capitalize every word of the query.

toTitleCase(str) {
    return str.replace(
        /\w\S*/g,
        function(txt) {
            return txt.charAt(0).toUpperCase() + txt.substr(1).toLowerCase();
        }
    );
}

this.knowledgeservice.getSearchResults(this.toTitleCase(querypay.query))

Like this we now get results for almost all queries.

Limiting information in Knowledge Graph without terminating the sentences:

To solve this issue, we decided to show only four lines of data retrieved from Wikipedia API

For this we have implemented a getPosition function which will take three parameters a string, a substring and a index. Here string is the whole string from which the function will return the index of index position substring.

getPosition(string, subString, index) {
    return string.split(subString, index).join(subString).length;
 }
this.description = this.description.slice(0, this.getPosition(this.description, '.', 4) + 1);

Here, We get the index of 4th ‘.’ full stop by getPosition() function and then pass it to javascript slice method to slice the string upto that position.

Using this we limited results to 4 lines without terminating each line in middle.

References

1.W3Schools Regular Expression: https://www.w3schools.com/jsref/jsref_obj_regexp.asp

2.Javascript Slice method: https://www.w3schools.com/jsref/jsref_slice_array.asp

3.Wikipedia API: https://www.mediawiki.org/wiki/API:Main_page

Showing Query Description in Infobox

Post author:praveenojha33
Post published:August 20, 2018
Post category:FOSSASIA GSoC
Post comments:0 Comments

Knowledge Graph is used to give a brief introduction about the search query instantly. But the information of knowledge graph becomes more useful when it is defined by one or two line description. Google currently shows a description of search query in its knowledge graph which makes the search results more meaningful. In this blog I will describe how I have description for query using Wikidata API implemented a similar functionality in Susper.

Steps:

1.Adding a function to fetch data from Wikidata in Service:

getQueryDescription(searchquery) {
   const params = new URLSearchParams();
   params.set('origin', '*');
   params.set('action', 'wbsearchentities');
   params.set('search', searchquery);
   params.set('language', 'en');
   params.set('format', 'json');
   const headers = new Headers({ 'Accept': 'application/json' });
   const options = new RequestOptions({ headers: headers, search: params });
   return this.http.get(this.descriptionURL, options).map(res =>
     res.json()
     ).catch(this.handleError);
 }

2.Creating an effect for query description:

To implement the logic of getting query description from the results returned by the service is made in knowledge.ts file which creates an effect for knowledge description.

Here is the effect for query description

this.knowledgeservice.getQueryDescription(querypay.query)
       .takeUntil(nextSearch$)
       .subscribe((response) => {
         if (response['search']) {
           this.store.dispatch(new knowledge.DescriptionAction(response));
           return empty();
            } else {    this.store.dispatch(new knowledge.DescriptionAction([]));
                         return empty();
                 }
});

The response is then dispatched to the corresponding knowledge action.

3.Creating an Action for Query description:

After dispatching the query description logic to the action, we must have an action which will correspond to get the query description. Here is the implementation of Query Description Action in knowledge.ts action file.

export const ActionTypes = {
 CONTENT_CHANGE: type('[Knowledge] Content Change'),
 IMAGE_CHANGE: type('[Knowledge] Image Change'),
 DESCRIPTION_CHANGE: type('[Knowledge] Description Change')
};
export class SearchContentAction implements Action {
 type = ActionTypes.CONTENT_CHANGE;
 constructor(public payload: object) {}
}
export class SearchImageAction implements Action {
 type = ActionTypes.IMAGE_CHANGE;
 constructor(public payload: object) {}
}
export class DescriptionAction implements Action {
 type = ActionTypes.DESCRIPTION_CHANGE;
 constructor(public payload: object) {}
}
export type Actions
 = SearchContentAction | SearchImageAction | DescriptionAction;

DescriptionAction is the action for Query Description and is then exported for the reducer.

4.Creating a case for query description in Knowledge reducer:

We must have a case to deal in reducer function to deal with query description action and here is the code for the case:

case knowledge.ActionTypes.DESCRIPTION_CHANGE: {
     const description_response = action.payload;
     return Object.assign({}, state, {
       content_response: state.content_response,
       image_response: state.image_response,
       description_response: description_response
     });
   }

The new state is then exported to the store:

export const getDescriptionResponse = (state: State) => state.description_response;

5.Updating the contents in store:

To update the store we just retrieve the getKnowledgeState and read its description_response and export it.

export const getDescriptionResponse = (state: State) => state.description_response;

6.Fetching Query Description from store in infobox component:

Now fetching results is very simple, we just need to select the getDescription from the store at store it in description_response$ and then we extract the query description and store in a local variable querydescription

this.description_response$ = store.select(fromRoot.getDescription);
   this.description_response$.subscribe(res => {
     if (res['search']) {
       this.querydescription = res['search'][0]['description'];
     }
   }
   );

7.Displaying query description on the result page:

Now displaying the query description on our page is very simple we just need to use our querydescription variable.

<p class="query_description">{{this.querydescription}}</p><hr width="50%" align="left" *ngIf="this.image">

Resources

Wikidata API: https://www.wikidata.org/w/api.php
Angular Services: https://angular.io/tutorial/toh-pt4
Corresponding Pull Request: https://github.com/fossasia/susper.com/pull/1115

Using Wikipedia API for knowledge graph in SUSPER

Post author:praveenojha33
Post published:July 29, 2018
Post category:FOSSASIA
Post comments:0 Comments

Knowledge Graph is way to give a brief description about search query by connecting it to a real world entity. This helps users to get information about exactly what they want. Previously Susper had a Knowledge Graph which was implemented using DBpedia API. But since DBpedia do not provide content over HTTPS connections therefore the content was blocked on susper.com and there was a need to implement the Knowledge Graph using a new API that provide contents over HTTPS. In this blog, I will describe how getting a knowledge graph was made possible using Wikipedia API.

What is Wikipedia API ?

The MediaWiki action API is a web service that provides convenient access to wiki features, data, and metadata over HTTP, via a URL usually at api.php. Clients request particular “actions” by specifying an action parameter, mainly action=query to get information.

The endpoint :

https://en.wikipedia.org/w/api.php

The format :

format=json This tells the API that we want data to be returned in JSON format.

The action :

action=query

The MediaWiki web service API implements dozens of actions and extensions implement many more; the dynamically generated API help documents all available actions on a wiki. In this case, we’re using the “query” action to get some information.

The complete API which is used in SUSPER to extract information of a query is :

https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=japan

Where titles=Search_Query, here Japan

How it is implemented in SUSPER?

For implementing it a service has been created which fetches information by setting various URL parameters. This result can be fetched by creating an instance of service and passing search query to getsearchresults(searchquery) function.

export class KnowledgeapiService {
 server = 'https://en.wikipedia.org';
 searchURL = this.server + '/w/api.php?';
 homepage = 'http://susper.com';
 logo = '../images/susper.svg';
 constructor(private http: Http,
             private jsonp: Jsonp,
             private store: Store<fromRoot.State>) {
 }
 getsearchresults(searchquery) {
   let params = new URLSearchParams();
   params.set('origin', '*');
   params.set('format', 'json');
   params.set('action', 'query');
   params.set('prop', 'extracts');
   params.set('exintro', '');
   params.set('explaintext', '');
   params.set('titles', searchquery);
   let headers = new Headers({ 'Accept': 'application/json' });
   let options = new RequestOptions({ headers: headers, search: params });
   return this.http
     .get(this.searchURL, options).map(res =>
         res.json().query.pages
     ).catch(this.handleError);
}

Since the result obtained is an observable therefore we have to subscribe for it and then extract information to local variables in infobox.component.ts file.

export class InfoboxComponent implements OnInit {
 public title: string;
 public description: string;
 query$: any;
 resultsearch = '/search';
 constructor(private knowledgeservice: KnowledgeapiService,
             private route: Router,
             private activatedroute: ActivatedRoute,
             private store: Store<fromRoot.State>,
             private ref: ChangeDetectorRef) {
   this.query$ = store.select(fromRoot.getquery);
   this.query$.subscribe( query => {
     if (query) {
       this.knowledgeservice.getsearchresults(query).subscribe(res => {
         const pageId = Object.keys(res)[0];
         if (res[pageId].extract) {
           this.title = res[pageId].title;
           this.description = res[pageId].extract;
         } else {
           this.title = '';
           this.description = '';
         }
       });
     }
   });
 }

The variable title and description are used to display results on results page.

<div *ngIf=“this.description” class=“card”>
  <div>
    <h2><b>{{this.title}}</b></h2>
    <p>{{this.description | slice:0:600}}<a href=‘https://en.wikipedia.org/wiki/{{this.title}}’>..more at Wikipedia</a></p>
  </div>
</div>

Resources

1.MediaWiki API : https://www.mediawiki.org/wiki/API:Main_page

2.Stackoverflow : https://stackoverflow.com/questions/8555320/is-there-a-clean-wikipedia-api-just-for-retrieve-content-summary

3.Angular Docs : https://angular.io/tutorial/toh-pt4

FOSSASIA Hack Trip to 32C3 Chaos Communication Congress

Post author:Mario Behling
Post published:December 30, 2015
Post category:FOSSASIA loklak

From December 27-30 the 32C3 Chaos Communication Congress took place in Hamburg, Germany. Members of the FOSSASIA community met at the event to learn about the latest OpenTech hacks. We used the opportunity to work with mentors on our coding program for kids. See photos from our gathering here.

Get together at 32C3 Hamburg

Mitch Altman at Chaos Communication Congress 32C3 with Hong Phuc Dang from FOSSASIA Hong Phuc Dang meeting Mitch Altman (Twitter)

FOSSASIA Mentors Michael Christen (Yacy), Jan Suhr (Nitrokey), Hong Phuc Dang Meetup with FOSSASIA mentors and supporters from Europe Michael Christen (Yacy.net/loklak), Jan Suhr (Nitrokey), Hong Phuc Dang (FOSSASIA), Anna (Mozilla)

OpenTech and Open Knowledge in the Mekong Delta and throughout Asia

Post author:Hong Phuc
Post published:November 12, 2012
Post category:Event FOSSASIA Open Event

The FOSSASIA organization has been very active and I would love to give you a few updates on our activities. One thing why we love to work in this community is because you get the chance to meet some of the most awesome and friendly tech people of the world. Jonas Smeedegaard from the Debian community is one of our guests who has been visiting us three times already and conducted workshops at our events and even stayed several weeks in Can Tho to train students using and developing Debian.

Jonas Smeedegaard at Workshop: Bringing Asian and International developers together

Meetups and Code Sprints

Of course we could not do all of this alone and we would like to thank some of the most outstanding folks we are working together with including HanoiLUG and Saigonlug in Vietnam, Singapore Hackerspace and the Beijinglug.

Coding Projects

We have recently involved and organized activities with the following projects and are glad to feature the following projects on this blog:

Wikipedia
OpenWrt
Crypto-Stick
LXDE
MoonOS
Android
GNOME
Debian
Fedora
Gimp
Inkscape
Linux conversion libraries
OpenStreetMap
and many more

FOSSASIA Event in Ho Chi Minh City

Internships

In 2012 we welcomed interns in Can Tho. Fifteen students participated in the program as full time interns in the office of MBM International in the biggest city in the Mekong delta. The program took place for 2 months and a core part of it was to learn how to use collaborative tools liker issue trackers and contribute to OpenStreetMap. Our students also love the Wikipedia sprints and we got a lot of positive vibes as everyone was happy to learn how to set up the Mediawiki software, that empowers the encyclopedia, share their ideas and spread Open Knowledge and Open Data about the Mekong Delta.

Event activities

We also continued to organize the FOSSASIA OpenTechSummit in 2010 and 2011 in Raffles College and Van Lang University in Ho Chi Minh City Vietnam. Plus we had a Mini-Debconf in Saigon and organized the event series OpenDesign.Asia Weeks bringing together designers and Open Source developers. We have links to the LXDE project and GNOME community and we have sent developers to the GNOME.Asia Summit 2012 in Hong Kong and supported the TYPO3 conference 2012 in Phnom Penh.

Also globally FOSSASIA starts to receive more attention. We have received the opportunity to introduce Asian projects at the Libre Graphics Meeting in Montreal (2011) and Brussels (2011), at the Linuxtag in Berlin (2012) and many other community events.

Please Join us and Get in Touch

The FOSSASIA organization supports activities and Open Tech development and Open Knowledge projects of the community active in Asia. If you are interested to cooperate for a project, need support for a development sprint or an event, please contact us on the FOSSASIA mailing list: http://groups.google.com/group/fossasia