Creating A Dockerfile For Yacy Grid MCP

The YaCy Grid is the second-generation implementation of YaCy, a peer-to-peer search engine. A YaCy Grid installation consists of a set of micro-services which communicate with each other using a common infrastructure for data persistence. The task was to deploy the second-generation of YaCy Grid. To do so, we first had created a Dockerfile. This dockerfile should start the micro services such as rabbitmq, Apache ftp and elasticsearch in one docker instance along with MCP. The microservices perform following tasks:

  • Apache ftp server for asset storage.
  • RabbitMQ message queues for the message system.
  • Elasticsearch for database operations.

To launch these microservices using Dockerfile, we referred to following documentations regarding running these services locally: https://github.com/yacy/yacy_grid_mcp/blob/master/README.md

For creating a Dockerfile we proceeded as follows:

FROM ubuntu:latest
MAINTAINER Harshit Prasad# Update
RUN apt-get update
RUN apt-get upgrade -y# add packages
# install jdk package for java
RUN apt-get install -y git openjdk-8-jdk

#install gradle required for build
RUN apt-get update && apt-get install -y software-properties-common
RUN add-apt-repository ppa:cwchien/gradle
RUN apt-get update
RUN apt-get install -y wget
RUN wget https://services.gradle.org/distributions/gradle-3.4.1-bin.zip
RUN mkdir /opt/gradle
RUN apt-get install -y unzip
RUN unzip -d /opt/gradle gradle-3.4.1-bin.zip
RUN PATH=$PATH:/opt/gradle/gradle-3.4.1/bin
ENV GRADLE_HOME=/opt/gradle/gradle-3.4.1
ENV PATH=$PATH:$GRADLE_HOME/bin
RUN gradle -v

# install apache ftp server 1.1.0
RUN wget http://www-eu.apache.org/dist/mina/ftpserver/1.1.0/dist/apache-ftpserver-1.1.0.tar.gz
RUN tar xfz apache-ftpserver-1.1.0.tar.gz

# install RabbitMQ server
RUN wget https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.6/rabbitmq-server-generic-unix-3.6.6.tar.xz
RUN tar xf rabbitmq-server-generic-unix-3.6.6.tar.xz

# install erlang language for RabbitMQ
RUN apt-get install -y erlang

# install elasticsearch
RUN wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.tar.gz
RUN sha1sum elasticsearch-5.5.0.tar.gz
RUN tar -xzf elasticsearch-5.5.0.tar.gz

# clone yacy_grid_mcp repository
RUN git clone https://github.com/nikhilrayaprolu/yacy_grid_mcp.git
WORKDIR /yacy_grid_mcp

RUN cat docker/configftp.properties > ../apacheftpserver1.1.0/res/conf/users.properties

# compile
RUN gradle build
RUN mkdir data/mcp-8100/conf/ -p
RUN cp docker/config-mcp.properties data/mcp-8100/conf/config.properties
RUN chmod +x ./docker/start.sh

# Expose web interface ports
# 2121: ftp, a FTP server to be used for mass data / file storage
# 5672: rabbitmq, a rabbitmq message queue server to be used for global messages, queues and stacks
# 9300: elastic, an elasticsearch server or main cluster address for global database storage
EXPOSE 2121 5672 9300 9200 15672 8100

# Define default command.
ENTRYPOINT [“/bin/bash”, “./docker/start.sh”]

 

We have created a start.sh file to start RabbitMQ and Apache FTP services. At the end, for compilation gradle run will be executed.

adduser –disabled-password –gecos ” r
adduser r sudo
echo ‘%sudo ALL=(ALL) NOPASSWD:ALL’ >> /etc/sudoers
chmod a+rwx /elasticsearch-5.5.0 -R
su -m r -c ‘/elasticsearch-5.5.0/bin/elasticsearch -Ecluster.name=yacygrid &’
cd /apacheftpserver1.1.0
./bin/ftpd.sh res/conf/ftpdtypical.xml &
/rabbitmq_server-3.6.6/sbin/rabbitmq-server -detached
sleep 5s;
/rabbitmq_server-3.6.6/sbin/rabbitmq-plugins enable rabbitmq_management
/rabbitmq_server3.6.6/sbin/rabbitmqctl add_user yacygrid password4account
echo [{rabbit, [{loopback_users, []}]}]. >> /rabbitmq_server-3.6.6/etc/rabbitmq/rabbitmq.config
/rabbitmq_server-3.6.6/sbin/rabbitmqctl set_permissions -p / yacygrid “.*” “.*” “.*”
cd /yacy_grid_mcp
sleep 5s;
gradle run

 

start.sh will first add username and then password. Then it will start RabbitMQ along with Apache FTP.  For username and password, we have created a separate files to configure their properties during Docker run which can be found here:

The logic behind running all the microservices in one docker instance was: creating each container for microservice and then link those containers with the help of docker-compose.yml file.

The Dockerfile which we have created was corresponding to one image. Another image was elasticsearch which was linked to this Dockerfile. The latest version of elasticsearch image was already available on their site: https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html

We configured the docker-compose.yml file according to the reference link provided above. The docker-compose file can be found here: https://github.com/yacy/yacy_grid_mcp/blob/master/docker/docker-compose.yml

The source code for the implementation of whole structure can be found here: https://github.com/yacy/yacy_grid_mcp/tree/master/docker

Resources

 

Implementation of Statistic Infobox for Susper

In Susper, we have implemented a statistic infobox to show analytics regarding Top authors, Top Providers and distribution regarding protocols and Results frequency by year.

Yacy also offers additional information for infoboxes such as files types, provider and authors. Using that information which we receive along with results we have implemented the infobox.

Implementation of Infobox:

1. For the distribution graphs, we have used angular library for chart.js https://www.npmjs.com/package/ng2-charts

2. We receive required statistics of each facet name from Yacy using the yacy search endpoint

http://yacy.searchlab.eu/solr/select?query=india&fl=last_modified&start=0&rows=15&facet=true&facet.mincount=1&facet.field=host_s&facet.field=url_protocol_s&facet.field=author_sxt&facet.field=collection_sxt&wt=yjson

Screenshot from 2017-08-15 14-10-30.png

Screenshot from 2017-08-15 14-10-16.png

We have created a statbox component to display the data related to statistic infobox at https://github.com/fossasia/susper.com/tree/master/src/app/statsbox

It takes care about rendering the statistic infobox and styling it.

Statsbox.component.ts

this.navigation$.subscribe(navigation => {
   for (let nav of navigation) {
     if (nav.displayname === 'Protocol') {
       let data = [];
       let datalabels = [];
       for (let element of nav.elements){
           datalabels.push(element.name);
           data.push(parseInt(element.count, 10));
         }
       this.barChartData[0].data = data;
       this.barChartLabels = datalabels;

     }
   }
 });
});

navigation observable gives us the latest statistics information received from the yacy and we subscribe to it and update the component variables accordingly for displaying the data.

Later these values are used by statsbox.component.html to display the statsbox.

The whole implementation of this feature can be found at pull: https://github.com/fossasia/susper.com/pull/704/

References:

1.Using Postman for analysing an API Endpoint: https://www.getpostman.com/docs

2.Using ngrx store: https://github.com/ngrx/store

Continuous Integration and Deployment of Yacy Grid

We have deployed Yacy Grid on Google cloud recently, and we have achieved this using kubernetes and Travis for auto deployment.

How we have deployed it:

Firstly, it is advised to have different containers for each service your application requires, and follow a multi container architecture. Using multi container architecture you can allocate fixed size of power to each application and also replicate individual services, whichever is required. Presently, Yacy has two main applications which are required to be deployed in separate containers – Yacy_grid_mcp and ElasticSearch.

We took the official kubernetes YAML files of ElasticSearch and followed the instructions at https://github.com/kubernetes/examples/blob/master/staging/elasticsearch/README.md for deployment of elastic search on the google cloud.

With this we are able to run pods, volumes required for elastic search and services for connecting Yacy with elastic search.

The pull request regarding deployment of separate elasticsearch component is at https://github.com/yacy/yacy_grid_mcp/pull/27/files

Below figure shows different services and external endpoints present pods use for elastic search.

Now elastic search can be accessed at 35.202.154.219:9300 and http://35.193.124.253:9200/

Continuous deployment of Yacy_grid_mcp:

Please make sure that you have created a cluster on google container engine for deploying our containers on it. Regarding starting a project and cluster please read https://cloud.google.com/container-engine/docs/

1.Initially, Travis.yml initiates and sets up the required environment for Yacy deployment by installing Google cloud cli and kubectl components.

Source code regarding the Travis setup could be found at https://github.com/yacy/yacy_grid_mcp/blob/master/.travis.yml

2.Later Travis runs the depoy_staging.sh file, which builds the docker image of yacy o the present build and pushes it to hub.docker.com

if [ "$TRAVIS_PULL_REQUEST" != "false" -o "$TRAVIS_BRANCH" != "$SOURCE_BRANCH" ]; then
    echo "Skipping deploy; The request or commit is not on master"
    exit 0
fi

set -e

docker build -t nikhilrayaprolu/yacygridmcp:$TRAVIS_COMMIT ./docker
docker login -u="$DOCKER_USERNAME" -p="$DOCKER_PASSWORD"
docker tag nikhilrayaprolu/yacygridmcp:$TRAVIS_COMMIT nikhilrayaprolu/yacygridmcp:latest
docker push nikhilrayaprolu/yacygridmcp

Later with service key, we authenticate with google cloud and set the required environments and variables

echo $GCLOUD_SERVICE   base64 --decode -i > ${HOME}/gcloud-service-key.json
gcloud auth activate-service-account --key-file ${HOME}/gcloud-service-key.json

gcloud --quiet config set project $PROJECT_NAME_STG
gcloud --quiet config set container/cluster $CLUSTER_NAME_STG
gcloud --quiet config set compute/zone ${CLOUDSDK_COMPUTE_ZONE}
gcloud --quiet container clusters get-credentials $CLUSTER_NAME_STG

And Later we push the docker image built to google cloud and deploy it

kubectl config view
kubectl config current-context

kubectl set image deployment/${KUBE_DEPLOYMENT_NAME} ${KUBE_DEPLOYMENT_CONTAINER_NAME}=nikhilrayaprolu/yacygridmcp:$TRAVIS_COMMIT

Presently Yacy runs on 5vCPUs

With the following pods and services:

Also one can use kubectl cli for getting information regarding the cluster and pods as shown below

Pull request regarding deployment of yacy on google cloud is available at: https://github.com/yacy/yacy_grid_mcp/pull/16/files

References:

1.A Medium Blog on CD to Google Container: https://medium.com/google-cloud/continuous-delivery-in-a-microservice-infrastructure-with-google-container-engine-docker-and-fb9772e81da7

2.Another Blog on CD to Google Container: https://engineering.hexacta.com/automatic-deployment-of-multiple-docker-containers-to-google-container-engine-using-travis-e5d9e191d5ad

3.Deploying ElasticSearch to Cloud using Kubernetes: https://github.com/kubernetes/examples/blob/master/staging/elasticsearch/README.md

Implementing Sort By Date Feature In Susper

 

Susper has been given ‘Sort By Date’ feature which provides the user with latest results with the latest date. This feature enhances the search experience and helps users to find desired results more accurately. The sorting of results date wise is done by yacy backend which uses Apache Solr technology.

The idea was to create a ‘Sort By Date’ feature similar to the market leader. For example, if a user searches for keyword ‘Jaipur’ then results appear to be like this:

If a user wishes to get latest results, they can use ‘Sort By Date’ feature provided under ‘Tools’.

The above screenshot shows the sorted results.

You may however notice that results are not arranged year wise. Currently, the backend work for this is being going on Yacy and soon will be implemented on the frontend as well once backend provide us this feature.

Under ‘Tools’ we created an option for ‘Sort By Date’ simply using <li> tag.

<ul class=dropdownmenu>
  <li (click)=filterByDate()>Sort By Date</li>
</ul>

When clicked, it calls filterByDate() function to perform the following task:

filterByDate() {
  let urldata = Object.assign({}, this.searchdata);
  urldata.query = urldata.query.replace(/date, “”);
  this.store.dispatch(new queryactions.QueryServerAction(urldata));
}
Earlier we were using ‘last_modified desc’ attribute provided by Solr for sorting out dates in descending order. In June 2017, this feature was deprecated with a new update of Solr. We are using /date attribute in query for sorting out results which is being provided by Solr.

 

Deploying Yacy with Docker on Different Cloud Platforms

To make deploying of yacy easier we are now supporting Docker based installation.

Following the steps below one could successfully run Yacy on docker.

  1. You can pull the image of Yacy from https://hub.docker.com/r/nikhilrayaprolu/yacygridmcp/ or buid it on your own with the docker file present at https://github.com/yacy/yacy_grid_mcp/blob/master/docker/Dockerfile

One could pull the docker image using command:

docker pull nikhilrayaprolu/yacygridmcp

 

2) Once you have an image of yacygridmcp you can run it by typing

docker run <image_name>

 

You can access the yacygridmcp endpoint at localhost:8100

Installation of Yacy on cloud servers:

Installing Yacy and all microservices with just one command:

  • One can also download,build and run Yacy and all its microservices (presently supported are yacy_grid_crawler, yacy_grid_loader, yacy_grid_ui, yacy_grid_parser, and yacy_grid_mcp )
  • To build all these microservices in one command, run this bash script productiondeployment.sh
    • `bash productiondeployment.sh build` will install all required dependencies and build microservices by cloning them from github repositories.
    • `bash productiondeployment.sh run` will run all services and starts them.
    • Right now all repositories are cloned into ~/yacy and you can make customisations and your own changes to this code and build your own customised yacy.

The related PRs of this work are https://github.com/yacy/yacy_grid_mcp/pull/21 and https://github.com/yacy/yacy_grid_mcp/pull/20 and https://github.com/yacy/yacy_grid_mcp/pull/13

Resources:

Implementation of Speech UI in Susper

Recently, we have implemented a speech recognition feature in Susper where user could search by voice but it does not have an attractive UI. Google has a good user experience while recording the voice. We have implemented a similar Speech UI in Susper,

How we have implemented this?

  1. First we made a component speechtotext. It takes care of all the styling and functional changes of the speech UI and rendering the speech and any instructions required for the user. https://github.com/fossasia/susper.com/tree/master/src/app/speechtotext
  2. Initially when user clicks on the microphone in the search bar, it triggers the speechRecognition()

searchbar.component.html

<div class="input-group-btn">
 <button class="btn btn-default" id="speech-button" type="submit">
   <img src="../../assets/images/microphone.png" class="microphone" (click)="speechRecognition()"/>
 </button>

searchbar.component.ts

speechRecognition() {
 this.store.dispatch(new speechactions.SearchAction(true));
}

3) This dispatches an action speechaction.SearchAction(true), the app.component.ts is subscribed to this action and whenever this action is triggered the app component will open the speechtotext component.

Speechtotext.component.ts

Speech to text component on getting initialised calls the speech service’s record function which activates standard browser’s speech API

constructor(private speech: SpeechService) {
 
 this.speechRecognition();
}
speechRecognition() {
 this.speech.record('en_US').subscribe(voice => this.onquery(voice));
}

On recording the user’s voice and converting it to text, the text is sent to the onquery method as input and the recognised text is sent to other components through ngrx store.

onquery(event: any) {
 this.resettimer();
 this.store.dispatch(new queryactions.QueryServerAction({ 'query': event, start: 0, rows: 10, search: true }));
 this.message = event;
}

We have some UI text transitions where the user is shown with messages like ‘Listening…’ ,‘Speak Now’ and ‘Please check your microphone’ which are handle by creating a timer observable in angular.

ngOnInit() {
 this.timer = Observable.timer(1500, 2000);
 this.subscription = this.timer.subscribe(t => {
   this.ticks = t;

   if (t === 1) {
     this.message = "Listening...";
   }
   if (t === 4) {
     this.message = "Please check your microphone and audio levels.";
     this.miccolor = '#C2C2C2';
   }
   if (t === 6) {
     this.subscription.unsubscribe();
     this.store.dispatch(new speechactions.SearchAction(false));
   }
 });
}

The related PR regarding speech to text is at https://github.com/fossasia/susper.com/issues/624 .

With this now we have achieved a good UI for handling requests on Speech.

Resources:

Adding tip to drop downs in Susper using CSS in Angular

To create simple drop downs using twitter bootstrap, it is fairly easy for developers. The issue faced in Susper, however, was to add a tip on the top over such dropdowns similar to Google:

This is how it looks finally, in Susper, with a tip over the standard rectangular drop-down:

This is how it was done:

  1. First, make sure you have designed your drop-down according to your requirements, added the desired height, width and padding. These were the specifications used in Susper’s drop-down.

.dropdown-menu{
height: 500px;
width: 327px;
padding: 28px;
}
  1. Next add the following code to your drop-down class css:

.dropdown-menu:before {
position: absolute;
top: -7px;
right: 19px;
display: inlineblock;
border-right: 7px solid transparent;
border-bottom: 7px solid #ccc;
border-left: 7px solid transparent;
border-bottom-color: rgba(0, 0, 0, 0.2);
content: ;
}
.dropdown-menu:after {
position: absolute;
top: -5px;
right: 20px;
display: inlineblock;
border-right: 6px solid transparent;
border-bottom: 6px solid #ffffff;
border-left: 6px solid transparent;
content: ;
}

In css, :before inserts the style before any other html, whereas :after inserts the style after the html is loaded. Some of the parameters are explained here:

  • Top: can be used to change the position of the menu tip vertically, according to the position of your button and menu.
  • Right: can be used to change the position of the menu tip horizontally, so that it can be positioned used below the menu icon.
  • Position : absolute is used to make sure all our values are absolute and not relative to the higher div hierarchically
  • Border: All border attributes are used to specify border thickness, color and transparency before and after, which collectively gives the effect of a tip for the drop down.
  • Content : This value is set to a blank string ‘’, because otherwise none of our changes will be visible, since the divs will have no space allocated to them.

Resources

Implementation of Customizable Instant Search on Susper using Local Storage

Results on Susper could be instantly displayed as user types in a query. This was a strict feature till some time before, where the user doesn’t have customizable option to choose. But now one could turn off and on this feature.

To turn on and off this feature visit ‘Search Settings’ on Susper. This will be the link to it: http://susper.com/preferences and you will find different options to choose from.

How did we implement this feature?

Searchsettings.component.html:

<div>
 <h4><strong>Susper Instant Predictions</strong></h4>
 <p>When should we show you results as you type?</p>
 <input name="options" [(ngModel)]="instantresults" disabled value="#" type="radio" id="op1"><label for="op1">Only when my computer is fast enough</label><br>
 <input name="options" [(ngModel)]="instantresults" [value]="true" type="radio" id="op2"><label for="op2">Always show instant results</label><br>
 <input name="options" [(ngModel)]="instantresults" [value]="false" type="radio" id="op3"><label for="op3">Never show instant results</label><br>
</div>

User is displayed with options to choose from regarding instant search.when the user selects a new option, his selection is stored in the instantresults variable in search settings component using ngModel.

Searchsettings.component.ts:

Later when user clicks on save button the instantresults object is stored into localStorage of the browser

onSave() {
 if (this.instantresults) {
   localStorage.setItem('instantsearch', JSON.stringify({value: true}));
 } else {
   localStorage.setItem('instantsearch', JSON.stringify({ value: false }));
   localStorage.setItem('resultscount', JSON.stringify({ value: this.resultCount }));
 }
 this.router.navigate(['/']);
}

 

Later this value is retrieved from the localStorage function whenever a user enters a query in search bar component and search is made according to user’s preference.

Searchbar.component.ts

Later this value is retrieved from the localStorage function whenever a user enters a query in search bar component and search is made according to user’s preference.

onquery(event: any) {
 this.store.dispatch(new query.QueryAction(event));
 let instantsearch = JSON.parse(localStorage.getItem('instantsearch'));

 if (instantsearch && instantsearch.value) {
   this.store.dispatch(new queryactions.QueryServerAction({'query': event, start: this.searchdata.start, rows: this.searchdata.rows}));
   this.displayStatus = 'showbox';
   this.hidebox(event);
 } else {
   if (event.which === 13) {
     this.store.dispatch(new queryactions.QueryServerAction({'query': event, start: this.searchdata.start, rows: this.searchdata.rows}));
     this.displayStatus = 'showbox';
     this.hidebox(event);
   }
 }
}

 

Interaction of different components here:

  1. First we set the instantresults object in Local Storage from search settings component.
  2. Later this value is retrieved and used by search bar component using localstorage.get() method to decide whether to display results instantly or not.

Below, Gif shows how you could use this feature in Susper to customise the instant results in your browser.

References:

 

Using Hidden Attribute for Angular in Susper

In Angular, we can use the hidden attribute, to hide and show different components of the page. This blog explains what the hidden attribute is, how it works and how to use it for some common tasks.
In Susper, we used the [hidden] attribute for two kinds of tasks.

  1. To hide components of the page until all the search results load.
  2. To hide components of the page, if they were meant to appear only in particular cases (say only the first page of the search results etc).

Let us now see how we apply this in a html file.
Use the [hidden] attribute for the component, to specify a flag variable responsible for hiding it.
When this variable is set to true or 1, the component is hidden otherwise it is shown.
Here is an example of how the [hidden] attribute is used:

<app-infobox [hidden]=”hidefooter class=“infobox col-md-4” *ngIf=“Display(‘all’)”></app-infobox>

Note that [hidden] in a way simply sets the css of the component as { display: none }, whereas in *ngIf, the component is not loaded in the DOM.
So, in this case unless Display(‘all’) returns true the component is not even loaded to the DOM but if [hidden] is set to true, then the component is still present, only not displayed.
In the typescript files, here is how the two tasks are performed:
To hide components of the page, until all the search results load.

this.querychange$ = store.select(fromRoot.getquery);
this.querychange$.subscribe(res => {
this.hidefooter = 1;

this.responseTime$ = store.select(fromRoot.getResponseTime);
this.responseTime$.subscribe(responsetime => {
this.hidefooter = 0;

The component is hidden when the query request is just sent. It is then kept hidden until the results for the previously sent query are available.

2. To hide components of the page, if they were meant to appear only in particular cases.
For example, if you wish to show a component like Autocorrect only when you are on the first page of the search results, here is how you can do it:

if (this.presentPage === 1) {
this.hideAutoCorrect = 0;
} else {
this.hideAutoCorrect = 1;
}

This should hopefully give you a good idea on how to use the hidden attribute. These resources can be referred to for more information.

Crawl Job Feature For Susper To Index Websites

The Yacy backend provides search results for Susper using a web crawler (or) spider to crawl and index data from the internet. They also require some minimum input from the user.

As stated by Michael Christen (@Orbiter) “a web index is created by loading a lot of web pages first, then parsing the content and placing the result into a search index. The question is: how to get a large list of URLs? This is solved by a crawler: we start with a single web page, extract all links, then load these links and go on. The root of such a process is the ‘Crawl Start’.”

Yacy has a web crawler module that can be accessed from here: http://yacy.searchlab.eu/CrawlStartExpert.html. As we would like to have a fully supported front end for Yacy, we also introduced a crawler in Susper. Using crawler one could tell Yacy what process to do and how to crawl a URL to index search results on Yacy server. To support the indexing of web pages with the help of Yacy server, we had implemented a ‘Crawl Job’ feature in Susper.

1)Visit http://susper.com/crawlstartexpert and give information regarding the sites you want Susper to crawl.Currently, the crawler accepts an input of URLs or a file containing URLs. You could customise crawling process by tweaking crawl parameters like crawling depth, maximum pages per domain, filters, excluding media etc.

2) Once crawl parameters are set, click on ‘Start New Crawl Job’ to start the crawling process.

3) It will raise a basic authentication pop-up. After filling, the user will receive a success alert and will be redirected back to home page.

The process of crawl job on Yacy server will get started according to crawling parameters.

Implementation of Crawler on Susper:

We have created a separate component and service in Susper for Crawler

Source code can be found at:

When the user initiates the crawl job by pressing the start button, it calls startCrawlJob() function from the component and this indeed calls the CrawlStart service.We send crawlvalues to the service and subscribe, to the return object confirming whether the crawl job has started or not.

crawlstart.component.ts:-

startCrawlJob() {
 this.crawlstartservice.startCrawlJob(this.crawlvalues).subscribe(res => {
   alert('Started Crawl Job');
   this.router.navigate(['/']);
 }, (err) => {
   if (err === 'Unauthorized') {
     alert("Authentication Error");
   }
 });
};

 

After calling startCrawlJob() function from the service file, the service file creates a URLSearchParams object to create parameters for each key in input and send it to Yacy server through JSONP request.

crawlstart.service.ts

startCrawlJob(crawlvalues) {
 let params = new URLSearchParams();
 for (let key in crawlvalues) {
   if (crawlvalues.hasOwnProperty(key)) {
     params.set(key, crawlvalues[key]);
   }

 }
 params.set('callback', 'JSONP_CALLBACK');


 let options = new RequestOptions({ search: params });
 return this.jsonp
   .get('http://yacy.searchlab.eu/Crawler_p.json', options).map(res => {
     res.json();
   });

}

Resources: