While adding video sub-tabs to the loklak search, we faced an issue regarding how could we possibly embed YouTube videos returned by the Loklak search.json API and avoid security risks. Either way was to add iframely APIs or use automatic sanitization. I made reference to Angular Documentation to get this approach for DOM Sanitization. The main issue faced when using an iframe element to display videos in Loklak Search is that the web application became vulnerable to cross-site scripting (XSS) attack .

I would be explaining in this blog about I implemented lazy loading in loklak search and used automatic sanitization to make video resource trusted and safe to be displayed on loklak search. This is the error we got when <iframe> was used in loklak search without sanitizing it.

How to display Youtube videos and avoid security risks?

These steps were used in loklak search and show how to get embeddable youtube links and bypass Security risks by marking resource URL as trusted.

• Importing DomSanitizer into the component from the @angular/platform-browser module and then injecting into the constructor of the component.

This step will inject Domsantizer and provides some bypassSecurityTrust APIs.

• Calling a method during the initialization of the component and passing all the video URLs as a parameter which needs to bypass security

As, the Angular Documentation states ,it is better to call it during initialization so that  bypasses security as early as possible and it is declared that no security risks are created by its use at an early stage.

• Make youtube links embeddable and bypass security to mark video URLs as safe and trusted.

This method performs two functions:-

• Firstly, for each link, it tries matches the URLs of the youtube videos to standard formats of the youtube URLs using regular expressions and then extracts the unique video ID and brings it to an embeddable format for the iframe.

• Secondly, it also makes the sanitizer to bypass security and trusts the Resource URL.

• Now links are ready to be used as the value for the source attribute for the iframe element.

Done! This makes our application risk-free for the users, Youtube Videos can now be displayed safely using iframe tag.

Resources

• https://angular.io/api/platform-browser/DomSanitizer
• https://netbasal.com/angular-2-security-the-domsanitizer-service-2202c83bd90
• https://stackoverflow.com/questions/39438039/correct-way-provide-domsanitizer-to-component-with-angular-2-rc6

I was working on one of the issues which was to render top hashtags of the last 24 hours on the home-page of loklak search. The loklak search.json API took just milliseconds to return the response. However, it took about 3 minutes to render the HTML. Generally, API response acts as a Rate Determining Step but here, template rendering process was slow so, that optimization became crucial.

This blog will explain how I improved slow template rendering problem using some APIs of Angular. I would be explaining common mistakes we often make and solution about how to quickly render HTML after data is received from the APIs.

Previous Code

In this code, we have made an API request with query ‘since:day’ on the initialization of the Home page component. Next, we have subscribed to the store which will return an observable of hashtags. Then, we are subscribing to the observable of hashtags and extract the data from it.

In the HTML file, now we are presenting hashtags using *ngFor and *ngIf property.

This code working is fine. The only fault is that the template is not aware if some input is received or some state property has changed. Therefore, we need a trigger to do detect the change and render the HTML.

TypeScript File

export class HomeComponent implements OnInit, OnDestroy {
    public apiResponseHashtags$: Observable<Array<{ tag: string, count: number }>>;
    public apiResponseHashtags: <Array<{ tag: string, count: number }>>;

constructor(private store: Store<fromRoot.State>)
ngOnInit() {
this.getTopHashtags();
this.getDataFromStore();}
}

    private getTopHashtags() {
    this.store.dispatch(new queryAction.RelocationAfterQuerySetAction());
this.store.dispatch(new queryAction.InputValueChangeAction('since:day'));
    }

    private getDataFromStore() {
this.apiResponseHashtags$ =  this.store.select(fromRoot.getApiResponseTags);
    this.apiResponseHashtags$.subscribe((data) => {
this.apiResponseHashtags = data;
});
    }
}

HTML File

<div class="top-hashtags">
    <div *ngIf="apiResponseHashtags.length !== 0">
        <span *ngFor ="let item of apiResponseHashtags">
            <a [routerLink]="['/search']" [queryParams]="{ query : '#' + item.tag }">#{{item.tag}}</a>&nbsp;
        </span>
    </div>
</div>

Solution

These are the following changes that need to be introduced for quick HTML rendering.

• Here, we will be using ‘Change Detection Strategy: OnPush’ while defining component for pushing changes in the template. Using ‘OnPush’ will make changes to the template as soon as some input is received.

• Instead of subscribing to the observables of hashtags, we need to use the async pipe to extract data from observable in the HTML file. This is because, in the previous code, we were using subscribed value for hashtags. Now, the observable of hashtags will work like an input for the template and  ‘Change Detection Strategy: OnPush’ will act as a trigger.

TypeScript file

@Component({
    selector: 'app-home',
    templateUrl: './home.component.html',
    styleUrls: ['./home.component.scss'],
    changeDetection: ChangeDetectionStrategy.OnPush
})
export class HomeComponent implements OnInit, OnDestroy {
public apiResponseHashtags$: Observable<Array<{ tag: string, count: number }>>;

constructor(private store: Store<fromRoot.State>)
ngOnInit() {
this.getTopHashtags();
this.getDataFromStore();}
}


private getTopHashtags() {
    this.store.dispatch(new queryAction.RelocationAfterQuerySetAction());
this.store.dispatch(new queryAction.InputValueChangeAction('since:day'));
    }

    private getDataFromStore() {
this.apiResponseHashtags$ = this.store.select(fromRoot.getApiResponseTags);
    }
}

HTML File

<div class="top-hashtags">
    <div *ngIf="(apiResponseHashtags$ | async).length !== 0">
        <span *ngFor ="let item of (apiResponseHashtags$ | async)">
            <a [routerLink]="['/search']" [queryParams]="{ query : '#' + item.tag }">#{{item.tag}}</a>&nbsp;
        </span>
    </div>
</div>

Conclusion

Whenever the template data depends on the API response or user interaction, it is beneficial to use ChangeDetectionStrategy  and reduce template rendering time. Moreover, instead of subscribing to observables of data received from the API, we will be using an async pipe to make observable as an input for the template.

Resources

• https://angular-2-training-book.rangle.io/handout/change-detection/change_detection_strategy_onpush.html
• https://stackoverflow.com/questions/39795634/angular-2-change-detection-and-changedetectionstrategy-onpush/39802466#39802466
• https://www.youtube.com/watch?v=X0DLP_rktsc

Single page applications like Loklak Search, are the ones which only have a single physical HTML page but has many different logical pages. This is achieved with the help of JavaScript, and angular provides strong abstraction to help the developers write the logical pages and Angular Router manages which page to display when a particular view is demanded. The Angular Router manages the URL state and the state of corresponding components.

Overview:

Now it’s important to understand how the routing works. In every web application structure, there is an index.html file, for SPA this is the only one single physical HTML page. This page is served when the base url is requested, ie. the web server gives this file to the client when the client requests for the base url of the application, in our case http://loklak.net/, This index.html page contains all the JS and CSS required by the application to work, which are loaded, as normal HTTP requests. When the javascript is loaded, the control of the application is governed by Angular which manages all the components, views and routes. When a user demands some route, the angular updates the view with the required components that are required to show the contents of that route.

It’s important to note here that no further simple requests are made by the client for any resource, angular is the one which controls all such requests. When we define routes in angular, say /search, we tell it what is the component which should be shown to the user when the route is /search, whenever the user navigates to such route angular shows the appropriate component, there is no request made by the client to the server for any such routes.

In Fact, the server doesn’t even know of such route, as they are defined in angular itself not, on the server side. The important fact is now what happens when the client refreshes the page when route /search is the URL, now the request is sent to the server which doesn’t know which page to serve when such request comes, so it will throw a 404: Page Not Found error.

Solution:

The solution to this situation is pretty straight forward, the server instead of throwing the 404 page, returns with the same index.html page. This makes the problem go away entirely, the server responds with the same page always, this page has the javascript to load and give control to angular, which when loaded, resolves which page to show according to the URL. This is a clean simple solution to the SPA routing problem. But this only works when you have access to Server Code, and you can request the server to respond with the same page whatever be the URL.

The problem with gh-pages:

But Loklak Search is deployed on gh-pages, and thus we don’t have any access to the server side code, to display the page of our requirements. GitHub pages just serve the HTML file which matches the corresponding URL, plain simple server. Now if on gh-pages we request for /search route it will respond with a 404 Error, as it is unable to find the search.html file in the directory.

How it is achieved in Loklak Search?

Loklak search is also deployed on gh-pages, to have a workaround to this problem we take advantage of the page served by gh-pages when the requested page is not found. When 404 error occur gh-pages responds with the 404.html file if it exists in the repository, else it responds with a default 404 page. We use this 404.html page to be able to load our main index.html page.

Here we have our 404.html file which is served by gh-pages whenever some url which is not unknown is requested

• Here we first store the actual location which was requested by the user in the session storage.
• Then we do a Redirective Refresh using the meta tag, to the root page ie. to load index.html which is known to gh-pages and is served.

https://gist.github.com/hemantjadon/9581dd0b4907c567b2a90eb949c5cbbc.js

<!-- 404.html -->
<!doctype html>
<html>
   <head>
  <!-- This stores the URL the user was attempting to go to in sessionStorage, and then redirects all 404 responses to the app’s index.html page -->
      <scrpt>
         sessionStorage.redirect = location.href;
      </scrpt>
         <meta http-equiv="refresh" content="0;URL='http://loklak.net/'">
      </meta>
   </head>
   <body>
   </body>
</html>

And then in our index.html after loading the angular and all the resources, we run a function, it takes out the redirect key stored in session storage, and compare it to the current location.

If the locations don’t match, we replace the history state with the with the redirect and this time as angular is already loaded it knows which page to show to the user.

<!-- index.html -->

...

...

<scrpt>

   (function(){
     var redirect = sessionStorage.redirect;
     delete sessionStorage.redirect;
     if (redirect && redirect != location.href) {
        history.replaceState(null, null, redirect);
     }
   })();

</scrpt>

...

...

Resources and links

• Loklak Search (Repo): https://github.com/fossasia/loklak_search
• Loklak Search (404.html): Link to file
• Blog for deploying SPA: Blog by Evan Salter

Data is always more interesting to look at when it is visualised by means of graphs and various plots rather than just viewing data as raw json or xml. There are various open source libraries available for visualising data. The one which I will be using today is called morris.js. Although it does not provide a lot of plots but it is extremely simple to use and can be of great help when quick data visualisation is required.

This blog shows one of the many ways in which data obtained from loklak service can be visualised using morris.js. This is same library which I am using presently to build MultiLinePlotter app whicch will soon be displayed on Loklak app store.

What is morris.js?

It is an open source library for drawing graphs using javascript. Presently it provides APIs to draw line charts, area charts, bar charts and donut charts. To know more about morris.js please visit this link.

Loklak aggregation service

A loklak search result can also contain field aggregations if they are requested in the api call. Fields to be aggregated must be listed in the fields attribute in the api request. The following url is an example for requesting aggregation on hashtags and mentions:

http://api.loklak.org/api/search.json?q=spacex%20since:2015-04-01%20until:2015-04-06&source=cache&count=0&fields=mentions,hashtags&limit=6

‘created_at’ is a special field which will return a date histogram if mentioned in the fields attribute. To know more about the loklak aggregation service and the loklak project in general please visit this link.

Getting started

An example of morris.js can be found here. Feel free to browse this code to see morris.js in working.

So what we will be doing today is plotting a simple bar chart using morris.js and the data returned by loklak aggregation service. For this we will be using the following api call :

https://api.loklak.org/api/search.json?callback=JSON_CALLBACK&q=fossasia%20since:2015-12-10&source=cache&count=0&fields=created_at

The above request will return an date histogram for the query fossasia, that is, date vs number of tweets containing the term ‘fossasia’.

So the first thing that we need to do is create a directory for this project and create a file named index.html in it. This is the only file we will be requiring.

Next we need to add morris.js and its dependencies:

<link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/morris.js/0.5.1/morris.css">
    <script type="text/javascript" src="//ajax.googleapis.com/ajax/libs/jquery/1.9.0/jquery.min.js"></script>
    <script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/raphael/2.1.0/raphael-min.js"></script>
    <script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/morris.js/0.5.1/morris.min.js"></script>

This will add the css and js script files for morris.js and its dependencies : Jquery and Raphael.
Next we need to add the div element that will contain our chart.

<div id="myfirstchart" style="height: 380px;margin-top:50px;overflow-x: auto;"></div>

The most important thing to note over here is that the div must have an id so that we can refer it later from our js code.

Next comes the most important and interesting part : adding the js code that will fetch the data and plot the bar chart.
The first thing that the script will do is make an ajax call to the above mentioned url and get the data.This is done using the following code :

$.ajax({
          url: "https://api.loklak.org/api/search.json?callback=JSON_CALLBACK&q=fossasia%20since:2015-12-10&source=cache&count=0&fields=created_at",
          jsonp: "callback",
          dataType: "jsonp",
          success: function( response ) {
              data=[];
              for (var date in response.aggregations.created_at) {
                  data.push({day: date, tweets: response.aggregations.created_at[date]});
              }
              console.log(data);
              plotData(data);
          }
      });

It is to be noted here that a callback function is provided in the url and data type is set to jsonp and not json.This is done to avoid cross origin resource sharing (CORS) issue so that our browser does not block the ajax request.
Once our request is successful we populate the data array from the response returned to us.
We iterate over all the key value pairs present in the created_at object and add them as individual objects in the data array. This will be required in the next part.

Next we have to plot the data acquired by us. For this we will be using plotData(data) function as written below:

function plotData(data) {
          Morris.Bar({
              element: 'myfirstchart',
            data: data,
            xkey: 'day',
       ykeys: ['tweets'],
            labels: ['tweets']
          });
      }

We send this function the data array we created earlier. Inside this function we create a Morris.Bar object which takes an object as parameter. The object has the following properties:

‘element’ : This takes the id of the div where we want to load the chart.

‘data’ : This is the data array we created earlier and passed to this function. It is the data to plot.It is an array of objects containing the x and y  attributes.

‘xkey’ : A string containing the name of the attribute that contains X labels.

‘ykeys’ : A list of strings containing names of attributes that contain Y values (one for each series of data to be plotted).

‘labels’ : A list of strings containing labels for the data series to be plotted (corresponding to the values in the ykeys option).

That’s it. These are the basic parameters required to plot a bar chart using morris.js. Simple enough, isn’t it!!

Here is the complete js code:

<script type="text/javascript">
      $.ajax({
          url: "https://api.loklak.org/api/search.json?callback=JSON_CALLBACK&q=fossasia%20since:2015-12-10&source=cache&count=0&fields=created_at",
          jsonp: "callback",
          dataType: "jsonp",
          success: function( response ) {
              data=[];
              for (var date in response.aggregations.created_at) {
                  data.push({day: date, tweets: response.aggregations.created_at[date]});
              }
              console.log(data);
              plotData(data);
          }
      });
      function plotData(data) {
          Morris.Bar({
              element: 'myfirstchart',
              data: data,
              xkey: 'day',
              ykeys: ['tweets'],
              labels: ['tweets']
          });
      }
    </script>

Next, open terminal and navigate to your project directory containing the index.html file, then execute the following command:

Python -m SimpleHTTPServer

As the command suggests, it starts a simple HTTP server at port 8000.

So next ,as you might have already guessed, fire up your favourite browser and visit http://127.0.0.1:8000 and you will be able to see the chart present below.

Python’s simple HTTP server comes in handy if you are developing a frontend app and you want to test it on real small screen devices. For example if you want to test it on your mobile then just connect your computer and your mobile to your router, start python http server on your computer, find out your computer’s ip address (ifconfig) and visit http://<computer’s ip address>:8000 on your mobile. You will be able to view your app on your mobile. If you don’t use a router then create_ap may help.

Important Resources

• Find more information on morris.js here
• Visit Loklak apps GitHub page here
• You can find a similar implementation of the above tutorial here