Fixing Infinite Scroll Feature for Susper using Angular

In Susper, we faced a unique problem. Every time the image tab was opened, and the user scrolled through the images, all the other tabs in the search engine, such as All, Videos etc, would stop working. They would continue to display image results as shown:

Since this problem occurred only when the infinite scroll action was called in the image tab, I diagnosed that the problem probably was in the url parameters being set.

The url parameters were set in the onScroll() function as shown:

onScroll () {
let urldata = Object.assign({}, this.searchdata);
this.getPresentPage(1);
this.resultDisplay = ‘images’;
urldata.start = (this.startindex) + urldata.rows;
urldata.fq = ‘url_file_ext_s:(png+OR+jpeg+OR+jpg+OR+gif)’;
urldata.resultDisplay = this.resultDisplay;
urldata.append = true;
urldata.nopagechange = true;
this.store.dispatch(new queryactions.QueryServerAction(urldata));
};

The parameters append and nopagechange were to ensure that the images are displayed in the same page, one after the other.
To solve this bug I first displayed the query call each time a tab is clicked on the web console.
Here I noticed that for the tab videos, nopagechange and append attributes still persisted, and had not been reset. The start offset had not been set to 0 either.
So adding these few lines before making a query call from any tab, would solve the problem.

urldata.start = 0;
urldata.nopagechange = false;
urldata.append = false;

Now the object is displayed as follows:

Now videos are displayed in the videos tab, text in the text tab and so on.
Please refer to results.component.ts for the entire code.

References:

  1. On how to dispatch queries to the store: https://gist.github.com/btroncone/a6e4347326749f938510
  2. Tutorial on the ngrx suite:http://bodiddlie.github.io/ng-2-toh-with-ngrx-suite/

Creating an Infobox for Mobile View Using Angular in Susper

In Susper, the Information and Analytics boxes disappeared for widths smaller than 1100px, since they were too big to fit in the existing page layout.
In Laptop view:

In Mobile view:

Hence we decided to design a new Info-cum-Analytics box for mobile devices, where the Analytics are displayed only if the ‘Show Analytics’ button is clicked and they are hidden on clicking the ‘Hide Analytics’ button.

The following is the html code for the Infobox:

<div class=“combo-box”>
<appinfobox></app-infobox>
<button class=“btn” id=“toggle-button” (click)=“BoxToggle()” type=“button” datatoggle=“collapse” datatarget=“#statbox” ariaexpanded=“false” ariacontrols=“collapseExample”>
{{boxMessage}} Analytics
</button>
<appstatsbox class=“collapse” id=“statbox”></app-statsbox>
</div>

We can make the following observations:

  • The combo box has both the Info and the Stats box, in addition to a button, which toggles the display status of the Analytics box.
  • To display the appropriate message on the button, we have a special function called BoxToggle() which sets the value of boxMessage()
  • The data-toggle attribute of the button has been set to collapse and the data-target is statbox
  • The statsbox has the class collapse, so that it collapses on clicking on the Toggle button.

The typescript code is as follows:

BoxToggle() {
if (this.boxMessage === ‘Show’) {
this.boxMessage = ‘Hide’;
} else {
this.boxMessage = ‘Show’;
}
}

This code thus effectively toggles between the messages Show and Hide, if the button is clicked when the message says Show, it changes to Hide and vice-versa

The CSS code:

@media screen and (min-width:768px) {
div.combo-box {
display: none;
}
}

The css code ensures that this combo-box is displayed only for widths less than 768px (only mobile and tablet screens).

To view the entire code you can check results.component.html and results.component.css in the Susper repository.

References

  1. W3 Schools for Collapsible buttons and their implementation: https://www.w3schools.com/bootstrap/bootstrap_collapse.asp
  2. Official Bootstrap documentation for collapsible buttons: https://v4-alpha.getbootstrap.com/components/collapse/

Implementation of Image Viewer in Susper

We have implemented image viewer in Susper similar to Google.

Before when a user clicks on a thumbnail the images are opened in a separate page, but we want to replace this with an image viewer similar to Google.

Implementation Logic:

1. Thumbnails for images in susper are arranged as shown in the above picture.

2. When a user clicks on an image a hidden empty div(image viewer) of the last image in a row is opened.

3. The clicked image is then rendered in the image viewer (hidden div of the last element in a row).

4. Again clicking on the same image closes the opened image viewer.

5. If a second image is clicked then, if an image is in the same row, it is rendered inside the same image viewer. else if the image is in another row, this closes the previous image viewer and renders the image in a new image viewer (hidden div of the last element of the row)

6. Since image viewer is strictly the hidden empty div of the last element in a row when it is expanded it occupies the position of the next row, moving them further down similar to what we want.

Implementation Code

results.component.html

<div *ngFor="let item of items;let i = index">
 <div class="item">
   <img src="{{item.link}}" height="200px" (click)="expandImage(i)" [ngClass]="'image'+i">
 </div>
 <div class=" item image-viewer" *ngIf="expand && expandedrow === i">
   <span class="helper"></span> <img [src]="items[expandedkey].link" height="200px" style="vertical-align: middle;">
 </div>

</div>

Each thumbnail image will have a <div class=” item image-viewer” which is in hidden state initially.

Whenever a user clicks on a thumbnail that triggers expandImage(i)

results.component.ts

expandImage(key) {
 if (key === this.expandedkey    this.expand === false) {
   this.expand = !this.expand;
 }
 this.expandedkey = key;
 let i = key;
 let previouselementleft = 0;
 while ( $('.image' + i) && $('.image' + i).offset().left > previouselementleft) {
   this.expandedrow = i;
   previouselementleft = $('.image' + i).offset().left;
   i = i + 1;

The expandImage() function takes the unique key and finds which image is the last element is the last image in the whole row, and on finding the last image, expands the image viewer of the last element and renders the selected image in the image viewer.

The source code for the whole implementation of image viewer could be seen at pull: https://github.com/fossasia/susper.com/pull/687/files

Resources:

  1. Selecting elements in Jquery: https://learn.jquery.com/using-jquery-core/selecting-elements/

 

Creating A Better Responsive Design In Susper

A lot of work has been done on making Susper, a wonderful search-engine and still more work have to be done on it. To become a good competitor in the market, one should make their website UI design such that:

  • It should be eye-catching for the users on the first-time visit to the website.
  • It should be easy to use with simple UI features rather than having more complex UI features.

We have been more oriented towards the material design. We have used Bootstrap technology for designing UI. Earlier, we proposed an idea of creating a UI using Angular Material v2 but it was dropped due to time limitations and other issue priorities.

To make Susper a better competitor in the market, we made sure it should be responsive as well on the following devices:

  • Mobile screen devices:
    • 320px – Smaller screen size.
    • 375px – Medium screen size.
    • 425px – Larger screen size.
  • Tablets:
    • 768px – default screen size for tablets.
  • Laptops:
    • 1024px – Smaller screen size.
    • 1440px – Larger screen size.
  • 4K:
    • 2560px – Default screen size.

We targeted these devices using @media queries in CSS3. For e.g. if I want to make a site responsive for the mobile devices, I will be using:

@media screen and (minwidth: 320px) and (maxwidth: 425px) {
  // do something
}

 

Here, min-width: 320px means that the screen size should be greater than and equal to 320px and max-width: 425px means that the screen size should be less than and equal to 425px.

It is not necessary to use only these dimensions. Suppose if there is break in UI design between 320px and 425px then, one can add that screen size using @media query. In this case, nested @media queries play a quite good role.

@media screen and (minwidth: 320px) and (maxwidth: 425px) {
  // do something
  // let’s say, break in UI design is observed at 375px
  // add nested @media query
  @media screen and (minwidth: 375px) {
    // do something
  }
}

 

We’re still improving our CSS code at present following this grid pattern. One can check UI code at Susper repository hosted on GitHub: https://github.com/fossasia/susper.com

We have also used a lot of breakpoints which are not nested. But it’s good practice to break points in nested form. This will be solved while improving our CSS code.

Here are some screenshots of the current responsiveness of Susper:

  • Mobile screen devices:
  • Tablet devices:

  • Laptops:
  • 4K display:

Resources:

Creating A Dockerfile For Yacy Grid MCP

The YaCy Grid is the second-generation implementation of YaCy, a peer-to-peer search engine. A YaCy Grid installation consists of a set of micro-services which communicate with each other using a common infrastructure for data persistence. The task was to deploy the second-generation of YaCy Grid. To do so, we first had created a Dockerfile. This dockerfile should start the micro services such as rabbitmq, Apache ftp and elasticsearch in one docker instance along with MCP. The microservices perform following tasks:

  • Apache ftp server for asset storage.
  • RabbitMQ message queues for the message system.
  • Elasticsearch for database operations.

To launch these microservices using Dockerfile, we referred to following documentations regarding running these services locally: https://github.com/yacy/yacy_grid_mcp/blob/master/README.md

For creating a Dockerfile we proceeded as follows:

FROM ubuntu:latest
MAINTAINER Harshit Prasad# Update
RUN apt-get update
RUN apt-get upgrade -y# add packages
# install jdk package for java
RUN apt-get install -y git openjdk-8-jdk

#install gradle required for build
RUN apt-get update && apt-get install -y software-properties-common
RUN add-apt-repository ppa:cwchien/gradle
RUN apt-get update
RUN apt-get install -y wget
RUN wget https://services.gradle.org/distributions/gradle-3.4.1-bin.zip
RUN mkdir /opt/gradle
RUN apt-get install -y unzip
RUN unzip -d /opt/gradle gradle-3.4.1-bin.zip
RUN PATH=$PATH:/opt/gradle/gradle-3.4.1/bin
ENV GRADLE_HOME=/opt/gradle/gradle-3.4.1
ENV PATH=$PATH:$GRADLE_HOME/bin
RUN gradle -v

# install apache ftp server 1.1.0
RUN wget http://www-eu.apache.org/dist/mina/ftpserver/1.1.0/dist/apache-ftpserver-1.1.0.tar.gz
RUN tar xfz apache-ftpserver-1.1.0.tar.gz

# install RabbitMQ server
RUN wget https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.6/rabbitmq-server-generic-unix-3.6.6.tar.xz
RUN tar xf rabbitmq-server-generic-unix-3.6.6.tar.xz

# install erlang language for RabbitMQ
RUN apt-get install -y erlang

# install elasticsearch
RUN wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.tar.gz
RUN sha1sum elasticsearch-5.5.0.tar.gz
RUN tar -xzf elasticsearch-5.5.0.tar.gz

# clone yacy_grid_mcp repository
RUN git clone https://github.com/nikhilrayaprolu/yacy_grid_mcp.git
WORKDIR /yacy_grid_mcp

RUN cat docker/configftp.properties > ../apacheftpserver1.1.0/res/conf/users.properties

# compile
RUN gradle build
RUN mkdir data/mcp-8100/conf/ -p
RUN cp docker/config-mcp.properties data/mcp-8100/conf/config.properties
RUN chmod +x ./docker/start.sh

# Expose web interface ports
# 2121: ftp, a FTP server to be used for mass data / file storage
# 5672: rabbitmq, a rabbitmq message queue server to be used for global messages, queues and stacks
# 9300: elastic, an elasticsearch server or main cluster address for global database storage
EXPOSE 2121 5672 9300 9200 15672 8100

# Define default command.
ENTRYPOINT [“/bin/bash”, “./docker/start.sh”]

 

We have created a start.sh file to start RabbitMQ and Apache FTP services. At the end, for compilation gradle run will be executed.

adduser –disabled-password –gecos ” r
adduser r sudo
echo ‘%sudo ALL=(ALL) NOPASSWD:ALL’ >> /etc/sudoers
chmod a+rwx /elasticsearch-5.5.0 -R
su -m r -c ‘/elasticsearch-5.5.0/bin/elasticsearch -Ecluster.name=yacygrid &’
cd /apacheftpserver1.1.0
./bin/ftpd.sh res/conf/ftpdtypical.xml &
/rabbitmq_server-3.6.6/sbin/rabbitmq-server -detached
sleep 5s;
/rabbitmq_server-3.6.6/sbin/rabbitmq-plugins enable rabbitmq_management
/rabbitmq_server3.6.6/sbin/rabbitmqctl add_user yacygrid password4account
echo [{rabbit, [{loopback_users, []}]}]. >> /rabbitmq_server-3.6.6/etc/rabbitmq/rabbitmq.config
/rabbitmq_server-3.6.6/sbin/rabbitmqctl set_permissions -p / yacygrid “.*” “.*” “.*”
cd /yacy_grid_mcp
sleep 5s;
gradle run

 

start.sh will first add username and then password. Then it will start RabbitMQ along with Apache FTP.  For username and password, we have created a separate files to configure their properties during Docker run which can be found here:

The logic behind running all the microservices in one docker instance was: creating each container for microservice and then link those containers with the help of docker-compose.yml file.

The Dockerfile which we have created was corresponding to one image. Another image was elasticsearch which was linked to this Dockerfile. The latest version of elasticsearch image was already available on their site: https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html

We configured the docker-compose.yml file according to the reference link provided above. The docker-compose file can be found here: https://github.com/yacy/yacy_grid_mcp/blob/master/docker/docker-compose.yml

The source code for the implementation of whole structure can be found here: https://github.com/yacy/yacy_grid_mcp/tree/master/docker

Resources

 

Implementing Advanced Search Feature In Susper

Susper has been provided ‘Advanced Search’ feature which provides the user a great experience to search for desired results. Advanced search has been implemented in such a way it shows top authors, top providers, and distribution regarding protocols. Users can choose any of these options to get best results.

We receive data of each facet name from Yacy using yacy search endpoint. More about yacy search endpoint can be found here:  http://yacy.searchlab.eu/solr/select?query=india&fl=last_modified&start=0&rows=15&facet=true&facet.mincount=1&facet.field=host_s&facet.field=url_protocol_s&facet.field=author_sxt&facet.field=collection_sxt&wt=yjson

For implementing this feature, we created Actions and Reducers using concepts of Redux. The implemented actions can be found here: https://github.com/fossasia/susper.com/blob/master/src/app/actions/search.ts

Actions have been implemented because these actually represent some kind of event. For e.g. like the beginning of an API call here.

We also have created an interface for search action which can be found here under reducers as filename index.ts: https://github.com/fossasia/susper.com/blob/master/src/app/reducers/index.ts

Reducers are a pure type of function that takes the previous state and an action and returns the next state. We have used Redux to implement actions and reducers for the advanced search.

For advanced search, the reducer file can be found here: https://github.com/fossasia/susper.com/blob/master/src/app/reducers/search.ts

The main logic has been implemented under advancedsearch.component.ts:

export class AdvancedsearchComponent implements OnInit {
  querylook = {}; // array of urls
  navigation$: Observable<any>;
  selectedelements: Array<any> = []; // selected urls by user
changeurl
(modifier, element) {
// based on query urls are fetched
// if an url is selected by user, it is decoded
  this.querylook[‘query’] = this.querylook[‘query’] + ‘+’ + decodeURIComponent(modifier);
  this.selectedelements.push(element);
// according to selected urls
// results are loaded from yacy
  this.route.navigate([‘/search’], {queryParams: this.querylook});
}

// same method is implemented for removing an url
removeurl(modifier) {
  this.querylook[‘query’] = this.querylook[‘query’].replace(‘+’ + decodeURIComponent(modifier), );

  this.route.navigate([‘/search’], {queryParams: this.querylook});
}

 

The changeurl() function replaces the query with a query and selected URL and searches for the results only from the URL provider. The removeurl() function removes URL from the query and works as a normal search, searching for the results from all providers.

The source code for the implementation of advanced search feature can be found here: https://github.com/fossasia/susper.com/tree/master/src/app/advancedsearch

Resources

Implementation of Statistic Infobox for Susper

In Susper, we have implemented a statistic infobox to show analytics regarding Top authors, Top Providers and distribution regarding protocols and Results frequency by year.

Yacy also offers additional information for infoboxes such as files types, provider and authors. Using that information which we receive along with results we have implemented the infobox.

Implementation of Infobox:

1. For the distribution graphs, we have used angular library for chart.js https://www.npmjs.com/package/ng2-charts

2. We receive required statistics of each facet name from Yacy using the yacy search endpoint

http://yacy.searchlab.eu/solr/select?query=india&fl=last_modified&start=0&rows=15&facet=true&facet.mincount=1&facet.field=host_s&facet.field=url_protocol_s&facet.field=author_sxt&facet.field=collection_sxt&wt=yjson

Screenshot from 2017-08-15 14-10-30.png

Screenshot from 2017-08-15 14-10-16.png

We have created a statbox component to display the data related to statistic infobox at https://github.com/fossasia/susper.com/tree/master/src/app/statsbox

It takes care about rendering the statistic infobox and styling it.

Statsbox.component.ts

this.navigation$.subscribe(navigation => {
   for (let nav of navigation) {
     if (nav.displayname === 'Protocol') {
       let data = [];
       let datalabels = [];
       for (let element of nav.elements){
           datalabels.push(element.name);
           data.push(parseInt(element.count, 10));
         }
       this.barChartData[0].data = data;
       this.barChartLabels = datalabels;

     }
   }
 });
});

navigation observable gives us the latest statistics information received from the yacy and we subscribe to it and update the component variables accordingly for displaying the data.

Later these values are used by statsbox.component.html to display the statsbox.

The whole implementation of this feature can be found at pull: https://github.com/fossasia/susper.com/pull/704/

References:

1.Using Postman for analysing an API Endpoint: https://www.getpostman.com/docs

2.Using ngrx store: https://github.com/ngrx/store

Continuous Integration and Deployment of Yacy Grid

We have deployed Yacy Grid on Google cloud recently, and we have achieved this using kubernetes and Travis for auto deployment.

How we have deployed it:

Firstly, it is advised to have different containers for each service your application requires, and follow a multi container architecture. Using multi container architecture you can allocate fixed size of power to each application and also replicate individual services, whichever is required. Presently, Yacy has two main applications which are required to be deployed in separate containers – Yacy_grid_mcp and ElasticSearch.

We took the official kubernetes YAML files of ElasticSearch and followed the instructions at https://github.com/kubernetes/examples/blob/master/staging/elasticsearch/README.md for deployment of elastic search on the google cloud.

With this we are able to run pods, volumes required for elastic search and services for connecting Yacy with elastic search.

The pull request regarding deployment of separate elasticsearch component is at https://github.com/yacy/yacy_grid_mcp/pull/27/files

Below figure shows different services and external endpoints present pods use for elastic search.

Now elastic search can be accessed at 35.202.154.219:9300 and http://35.193.124.253:9200/

Continuous deployment of Yacy_grid_mcp:

Please make sure that you have created a cluster on google container engine for deploying our containers on it. Regarding starting a project and cluster please read https://cloud.google.com/container-engine/docs/

1.Initially, Travis.yml initiates and sets up the required environment for Yacy deployment by installing Google cloud cli and kubectl components.

Source code regarding the Travis setup could be found at https://github.com/yacy/yacy_grid_mcp/blob/master/.travis.yml

2.Later Travis runs the depoy_staging.sh file, which builds the docker image of yacy o the present build and pushes it to hub.docker.com

if [ "$TRAVIS_PULL_REQUEST" != "false" -o "$TRAVIS_BRANCH" != "$SOURCE_BRANCH" ]; then
    echo "Skipping deploy; The request or commit is not on master"
    exit 0
fi

set -e

docker build -t nikhilrayaprolu/yacygridmcp:$TRAVIS_COMMIT ./docker
docker login -u="$DOCKER_USERNAME" -p="$DOCKER_PASSWORD"
docker tag nikhilrayaprolu/yacygridmcp:$TRAVIS_COMMIT nikhilrayaprolu/yacygridmcp:latest
docker push nikhilrayaprolu/yacygridmcp

Later with service key, we authenticate with google cloud and set the required environments and variables

echo $GCLOUD_SERVICE   base64 --decode -i > ${HOME}/gcloud-service-key.json
gcloud auth activate-service-account --key-file ${HOME}/gcloud-service-key.json

gcloud --quiet config set project $PROJECT_NAME_STG
gcloud --quiet config set container/cluster $CLUSTER_NAME_STG
gcloud --quiet config set compute/zone ${CLOUDSDK_COMPUTE_ZONE}
gcloud --quiet container clusters get-credentials $CLUSTER_NAME_STG

And Later we push the docker image built to google cloud and deploy it

kubectl config view
kubectl config current-context

kubectl set image deployment/${KUBE_DEPLOYMENT_NAME} ${KUBE_DEPLOYMENT_CONTAINER_NAME}=nikhilrayaprolu/yacygridmcp:$TRAVIS_COMMIT

Presently Yacy runs on 5vCPUs

With the following pods and services:

Also one can use kubectl cli for getting information regarding the cluster and pods as shown below

Pull request regarding deployment of yacy on google cloud is available at: https://github.com/yacy/yacy_grid_mcp/pull/16/files

References:

1.A Medium Blog on CD to Google Container: https://medium.com/google-cloud/continuous-delivery-in-a-microservice-infrastructure-with-google-container-engine-docker-and-fb9772e81da7

2.Another Blog on CD to Google Container: https://engineering.hexacta.com/automatic-deployment-of-multiple-docker-containers-to-google-container-engine-using-travis-e5d9e191d5ad

3.Deploying ElasticSearch to Cloud using Kubernetes: https://github.com/kubernetes/examples/blob/master/staging/elasticsearch/README.md

Writing Browser Specific CSS for Susper in Angular

In Susper, we were facing a unique problem for Information box and Analytics box alignment.
At a width of around 1290 px, the boxes fit perfectly in Firefox as shown:

However, they were slipping to the next line in Google Chrome browsers for the same dimension(1290 px)

The solution to this issue was to write browser-specific CSS.
The two most commonly used browser specific tags are

  1. @-moz-document url-prefix() { }: This tag is used to target the Mozilla Firefox browser in particular. Anything written within the curly braces will not apply to any other browser.
  2. @media screen and (-webkit-min-device-pixel-ratio:0) { }: This tag is used to target all browsers that support webkit such as Chrome, Safari etc.

For our problem, we need to use @media screen and (-webkit-min-device-pixel-ratio:0) { }
This was how the code was written for both the components (Information box and Analytics box). Please refer to infobox.component.css and statsbox.component.css for the entire code.

@media screen and (-webkit-min-device-pixel-ratio:0) {
@media screen and (max-width: 1300px) {
.card {
width: 366px;
}
}
}
@media screen and (max-width: 1280px) {
.card {
width: 366px;
}

As a result of this snippet of code, we see the following effects:

  • In Chrome, the boxes change to a smaller width at 1300px itself, thus preventing it from slipping to the next line
  • In Firefox, the boxes change to a smaller width only at 1280px, and not at 1300px, thus achieving the exact design we envisioned.

This is how the display finally looks in Chrome:

References:

  1. Stack overflow on specific CSS tags for Chrome: https://stackoverflow.com/questions/9328832/how-to-apply-specific-css-rules-to-chrome-only
  2. Stack overflow on specific CSS tags for Firefox: https://stackoverflow.com/questions/952861/targeting-only-firefox-with-css

Making a Sticky Top Navigation bar for Susper using Angular

A lot of websites, require a top navigation bar that sticks to the top, irrespective of the screen dimension size. This blog deals with how the top navigation bar was made sticky in Susper.

  1. Using the correct Bootstrap classes. Notice the code enveloping the navigation bar.

<nav class=“top-nav navbar navbar-static-top navbar-default”>

class=“container-fluid”>

class=“navbar-header” id=“navcontainer”>

</div>
</div>
</nav>

Points to note:

  • Using navbar and navbar-default creates a standard gray navigation bar.
  • Using navbar-static-top makes the navbar stick only to the top of the page and disappear on scrolling down.
  • Using container-fluid creates a container for the contents of the navbar with wide margins
  1.  Now we also need to write some personalized CSS code. Notice the classes navcontainer and  top-nav. This is the CSS code for these classes:

.top-nav{
margin-bottom: 0;
}
#navcontainer {
height: 65px;
width: 100vw;
}#navcontainer ul {
margin: 0;
padding: 0;
list-style-type: none;
}

Points to note:

  • Margin and padding can be set according to how the navbar should look. Click here to know the difference between margins and padding.
  • The height has been customized to 65px in Susper, with a width of 100vw(entire viewpost width).
  1. Lastly, if your navigation bar is inside the body tag, remember that by default, body has a top margin of 57 px. As a result you may see an extra white space on top of your navigation bar. To remove this:
  • Move the navigation bar code out of the body tag. If you can’t then,
  • Place your navigation bar in a container ( resultContainer on the Susper result page) and write this in your CSS file.

.resultContainer{
margin-top: -57px;
}

References: