Displaying Youtube Videos using iframe in loklak search and Preventing Cross-site Scripting (XSS) Risks

While adding video sub-tabs to the loklak search, we faced an issue regarding how could we possibly embed YouTube videos returned by the Loklak search.json API and avoid security risks. Either way was to add iframely APIs or use automatic sanitization. I made reference to Angular Documentation to get this approach for DOM Sanitization. The main issue faced when using an iframe element to display videos in Loklak Search is that the web application became vulnerable to cross-site scripting (XSS) attack .

I would be explaining in this blog about I implemented lazy loading in loklak search and used automatic sanitization to make video resource trusted and safe to be displayed on loklak search. This is the error we got when <iframe> was used in loklak search without sanitizing it.

ERROR Error: unsafe value used in a resource URL context (see http://g.co/ng/security#xss)
Stack trace:
DomSanitizerImpl.prototype.sanitize@http://localhost:4200/vendor.bundle.js:30605:23 [angular]
setElementProperty@http://localhost:4200/vendor.bundle.js:9990:58 [angular]
checkAndUpdateElementValue@http://localhost:4200/vendor.bundle.js:9910:13 [angular]
checkAndUpdateElementInline@http://localhost:4200/vendor.bundle.js:9844:24 [angular]
checkAndUpdateNodeInline@http://localhost:4200/vendor.bundle.js:12535:23 [angular]
checkAndUpdateNode@http://localhost:4200/vendor.bundle.js:12510:16 [angular]
debugCheckAndUpdateNode@http://localhost:4200/vendor.bundle.js:13139:38 [angular]
debugCheckRenderNodeFn@http://localhost:4200/vendor.bundle.js:13118:13 [angular]
View_FeedCardComponent_6/<@ng:///FeedModule/FeedCardComponent.ngfactory.js:134:5 [angular]

 

How to display Youtube videos and avoid security risks?

These steps were used in loklak search and show how to get embeddable youtube links and bypass Security risks by marking resource URL as trusted.

  • Importing DomSanitizer into the component from the @angular/platform-browser module and then injecting into the constructor of the component.

import { DomSanitizer } from ‘@angular/platform-browser’;

export class FeedCardComponent implements OnInit {

constructor(private sanitizer: DomSanitizer) {}

}

 

This step will inject Domsantizer and provides some bypassSecurityTrust APIs.

  • Calling a method during the initialization of the component and passing all the video URLs as a parameter which needs to bypass security

ngOnInit() {
this.sanitizeAndEmbedURLs(this.videosURL);
}

 

As, the Angular Documentation states ,it is better to call it during initialization so that  bypasses security as early as possible and it is declared that no security risks are created by its use at an early stage.

  • Make youtube links embeddable and bypass security to mark video URLs as safe and trusted.

private sanitizeAndEmbedURLs(links) {
links.forEach((link, i) => {
let videoid = links[i].match(/(?:https?:\/{2})?(?:w{3}\.)?youtu(?:be)?\.(?:com|be)(?:\/watch\?v=|\/)([^\s&]+)/);
if (videoid !== null) {
links[i] = ‘http://www.youtube.com/embed/’ + videoid[1];
}
links[i] = this.sanitizer.bypassSecurityTrustResourceUrl(links[i]);
})
}

 

This method performs two functions:-

  • Firstly, for each link, it tries matches the URLs of the youtube videos to standard formats of the youtube URLs using regular expressions and then extracts the unique video ID and brings it to an embeddable format for the iframe.

http://www.youtube.com/embed/videoID

 

  • Secondly, it also makes the sanitizer to bypass security and trusts the Resource URL.

 

  • Now links are ready to be used as the value for the source attribute for the iframe element.

<iframe [src]=“videosURL[i]” width=“100%”allowfullscreen></iframe>

 

Done! This makes our application risk-free for the users, Youtube Videos can now be displayed safely using iframe tag.

 

Resources

Continue Reading

Using Open Layers 3 to Render a loklak Emoji Heatmap

In Emoji Heatmapper App I am implementing a heatmap with the help of Open Layers 3. Open Layers 3 contains a really handy class, ol.layer.Heatmap. In this blog post am going to tell you how the heatmap actually works in the backend.

A heatmap is an impressive way to visualize data. For a given matrix of data in which each value is represented by a color. The heatmap implementation is usually expensive in computation terms: For each grid’s pixel we need to compute its color from a set of known values. It is not a feasible method to be implemented on the client side because map rendering would take so much of time.

Open Layers 3 contains an easy-to-use class called ol.layer.Heatmap, which allows to render vector data as a heatmap. So how is this implemented?

The ol.layer.Heatmap layer uses a smart estimation to the design which produces relatively good results and which is also fast. The steps can be outlined as:

  • A gradient of colors is created as an image.
  • Each value is rendered in a canvas as a blurred point using the default radius and gradient. This produces a canvas where the blurred points may overlap each other and create more fuzzy zones.
  • Finally, an image is obtained from the canvas. The color is obtained from the previous image and the obtained color value may vary from 0 to 255.

Example usage of ol.layer.Heatmap class:

var heatMap = new ol.layer.Heatmap({
  source: vector,
  blur: parseInt(15, 10),
  radius: parseInt(5, 10),
  opacity: 0.9,
  gradient: ['#0000ff', '#f00', '#f00', '#ff0', '#f00']
});

 

The colored image is then rendered in the map canvas, obtaining a nice effect suited to be used for density maps. The ol.layer.Heatmap offers some properties we can use to visualize the map in a better way. The properties include blur, radius, gradient, shadow and weight. This can be configured as per feature, according to the level of importance to each feature determining in more or less measure of the final color we want.

Fig: Default colors of gradient property

Fig: Used gradient property for different colors

Resources

Continue Reading

Sentiment Data in Emoji-Heatmapper loklak App

Analysing emojis can uncover meaning and sentiment in ways regular text analytics cannot. So this was the main idea to introduce sentiment data into the Emoji-Heatmapper app. LokLak Search API has features such as classification and categorization of tweets. The emotions, for instance, can be joy, anticipation, sad etc.

So, in the Emoji-heatmapper app, I am displaying the occurrence of emojis on the map according to the location traced and also the sentiment related to the emoji i.e., search query as follows:

How to get the sentiment data:

One should simply enter the emoji into the search box for the results. The following code shows part of the LokLak Search API results (JSONObject):

      "hashtags_count": 0,
      "classifier_emotion": "anger",
      "classifier_emotion_probability": 
      1.2842921170985733E-9,
      "classifier_language": "english",
      "classifier_language_probability": 
      1.4594549568869297E-8,
      "without_l_len": 49,
      "without_lu_len": 49,
      "without_luh_len": 49,

I am using the above field name ”classifier_emotion” to display the results.

Till here getting the data relevant to query part is done. Next, the classifier_emotion of each tweet containing the query is collected into an array and sorted to get a unique list.

var emotion = [];
var emotions_array = [];

for (var i = 0; i  <  tweets.statuses.length; i++) {
    if (tweets.statuses[i].classifier_emotion) {
        emotion = tweets.statuses[i].classifier_emotion;
        emotions_array.push(emotion);
    }

    emotions_array.sort();
    emotion_array = jQuery.unique( emotions_array );

Loading the Sentiment data onto the Screen

When the query has a single emotion, or if multiple emotions or no emotions. These use cases/situations are displayed as follows:

Fig: Single Emotion

Fig: Multiple Emotions

Fig: No Emotions data

The code which creates the data dynamically on the output screen is as follows:

//Loading the sentiment
$(document).ready(function() {
    var listItems= "";
    if (emotions_array.length == 0) {
        listItems = "No Sentiment data is available for " + query
    }
    if (emotion_array.length == 1) {
        listItems += "<h3> Sentiment of " + query + " is ";
    } else if (emotion_array.length > 1) {
        listItems += "<h3> Sentiments of " + query + " are ";
    }

    var emotion_data = emotion_array.join(", ") + "."
    listItems += emotion_data + "</h3>"

    $("#sentiment").html(listItems);
});

Conclusion

The Emoji-Heatmapper app displays the sentiment data of the query being searched for which populates data dynamically using LokLak Search API.

Resources

Continue Reading

Map Responses from SUSI Server

Giving user responses in Maps is a very common reply of various assistants. Android and iOS clients of SUSI.AI can Render their own custom map views but there is a challenge in doing that in Bots and other clients which can only render Images.

The MapServlet in SUSI-Server is a servlet that takes the constraints of a map as input parameters and returns PNG Image of the rendered map.

This is a simple HttpServlet. Since it does not requires any auth or user rights we do not extend AbstactAPIHandler in this class.

 
public class MapServlet extends HttpServlet {

This is to method which runs whenever the Servlet receives a GET Query. After getting the query the function call another function “process(…)” which processes the input parameters.

 
@Override
protected void doGet(HttpServletRequest request, HttpServletResponse          response) throws ServletException, IOException {
Query post = RemoteAccess.evaluate(request);

This line is the for DDOS Protection.  It checks if the user query rate is too high and rate limit his queries. If Server suspects that the post is a DDOS then it returns a 503 error.

if (post.isDoS_blackout()) {
  response.sendError(503, "your request frequency is too high"); return;
} 
process(request, response, post);

The process method get the zoom, latitude, longitude, width and the height of the map and checks for the errors in them.

if (map.getHeight() > height || map.getWidth() > width) {
            BufferedImage bi = map.getImage();

            int xoff = (map.getWidth() - width) / 2;
            int yoff = (map.getHeight() - height) / 2;

            bi = bi.getSubimage(xoff, yoff, width, height);
            map = new RasterPlotter(width, height, RasterPlotter.DrawMode.MODE_REPLACE, "FFFFFF");
            map.insertBitmap(bi, 0, 0);
        }

Then we compute our image using RasterPlotter.

map.setDrawMode(DrawMode.MODE_SUB);
map.setColor(0xffffff);
if (text.length() > 0) PrintTool.print(map, 6, 12, 0, uppercase ? text.toUpperCase() : text, -1, false, 100);
PrintTool.print(map, map.getWidth() - 6, map.getHeight() - 6, 0, "MADE WITH LOKLAK.ORG", 1, false, 50);

Here we calculate the height of the map. If a user passed a height and width in get parameters, we scale the map according to that but if not its 256×256.

    int mx = (int) (map.getWidth() * (lon - west_lon) / (east_lon - west_lon));
    int my = (int) (map.getHeight() * (lat - north_lat) / (south_lat - north_lat));

// the marker has a height of 40 pixel and a width of 25 pixel
    final BufferedImage logo = ImageIO.read(FileSystems.getDefault().getPath("html").resolve("artwork").resolve("marker-red.png").toFile());
    map.insertBitmap(logo, Math.min(map.getWidth() - 25, Math.max(0, mx - 12)), Math.min(map.getHeight() - 40, Math.max(0, my - 40)), FilterMode.FILTER_ANTIALIASING);

We use the code below to set some text on the map for example the name of the place or the city etc.

map.setDrawMode(DrawMode.MODE_SUB);
map.setColor(0xffffff);
if (text.length() > 0) PrintTool.print(map, 6, 12, 0, uppercase ? text.toUpperCase() : text, -1, false, 100);
PrintTool.print(map, map.getWidth() - 6, map.getHeight() - 6, 0, "MADE WITH LOKLAK.ORG", 1, false, 50);

Finally we draw a marker on map

    int mx = (int) (map.getWidth() * (lon - west_lon) / (east_lon - west_lon));
    int my = (int) (map.getHeight() * (lat - north_lat) / (south_lat - north_lat));
      
    final BufferedImage logo = ImageIO.read(FileSystems.getDefault().getPath("html").resolve("artwork").resolve("marker-red.png").toFile());
    map.insertBitmap(logo, Math.min(map.getWidth() - 25, Math.max(0, mx - 12)), Math.min(map.getHeight() - 40, Math.max(0, my - 40)), FilterMode.FILTER_ANTIALIASING);

  At last we set the copyright message of OPENSTREETMAP at the bottom left.

PrintTool.print(map, 6, map.getHeight() - 6, 0, "(C) OPENSTREETMAP CONTRIBUTORS", -1, false, 100);

At the end we write the image and set the headers for Cross Origin Access.

    response.addHeader("Access-Control-Allow-Origin", "*");
        RemoteAccess.writeImage(fileType, response, post, map);
        post.finalize();
    }
}

 The servlet can be tested locally at

http://localhost:4000/vis/map.png?text=Test&mlat=1.28373&mlon=103.84379&zoom=18

Or on the live SUSI Server
http://api.susi.ai/vis/map.png?text=Test&mlat=1.28373&mlon=103.84379&zoom=18

Output:

References:

https://www.openstreetmap.org/

http://wiki.openstreetmap.org/wiki/Java_Access_Example

https://github.com/fossasia/susi_server/

Continue Reading

Solving slow template rendering in loklak search

I was working on one of the issues which was to render top hashtags of the last 24 hours on the home-page of loklak search. The loklak search.json API took just milliseconds to return the response. However, it took about 3 minutes to render the HTML. Generally, API response acts as a Rate Determining Step but here, template rendering process was slow so, that optimization became crucial.

This blog will explain how I improved slow template rendering problem using some APIs of Angular. I would be explaining common mistakes we often make and solution about how to quickly render HTML after data is received from the APIs.

Previous Code

In this code, we have made an API request with query ‘since:day’ on the initialization of the Home page component. Next, we have subscribed to the store which will return an observable of hashtags. Then, we are subscribing to the observable of hashtags and extract the data from it.

In the HTML file, now we are presenting hashtags using *ngFor and *ngIf property.

This code working is fine. The only fault is that the template is not aware if some input is received or some state property has changed. Therefore, we need a trigger to do detect the change and render the HTML.

TypeScript File

export class HomeComponent implements OnInit, OnDestroy {
    public apiResponseHashtags$: Observable<Array<{ tag: string, count: number }>>;
    public apiResponseHashtags: <Array<{ tag: string, count: number }>>;

constructor(private store: Store<fromRoot.State>)
ngOnInit() {
this.getTopHashtags();
this.getDataFromStore();}
}

    private getTopHashtags() {
    this.store.dispatch(new queryAction.RelocationAfterQuerySetAction());
this.store.dispatch(new queryAction.InputValueChangeAction('since:day'));
    }

    private getDataFromStore() {
this.apiResponseHashtags$ =  this.store.select(fromRoot.getApiResponseTags);
    this.apiResponseHashtags$.subscribe((data) => {
this.apiResponseHashtags = data;
});
    }
}

HTML File

<div class="top-hashtags">
    <div *ngIf="apiResponseHashtags.length !== 0">
        <span *ngFor ="let item of apiResponseHashtags">
            <a [routerLink]="['/search']" [queryParams]="{ query : '#' + item.tag }">#{{item.tag}}</a>&nbsp;
        </span>
    </div>
</div>

Solution

These are the following changes that need to be introduced for quick HTML rendering.

  • Here, we will be using ‘Change Detection Strategy: OnPushwhile defining component for pushing changes in the template. Using ‘OnPush’ will make changes to the template as soon as some input is received.

 

  • Instead of subscribing to the observables of hashtags, we need to use the async pipe to extract data from observable in the HTML file. This is because, in the previous code, we were using subscribed value for hashtags. Now, the observable of hashtags will work like an input for the template and  ‘Change Detection Strategy: OnPush’ will act as a trigger.

TypeScript file

@Component({
    selector: 'app-home',
    templateUrl: './home.component.html',
    styleUrls: ['./home.component.scss'],
    changeDetection: ChangeDetectionStrategy.OnPush
})
export class HomeComponent implements OnInit, OnDestroy {
public apiResponseHashtags$: Observable<Array<{ tag: string, count: number }>>;

constructor(private store: Store<fromRoot.State>)
ngOnInit() {
this.getTopHashtags();
this.getDataFromStore();}
}


private getTopHashtags() {
    this.store.dispatch(new queryAction.RelocationAfterQuerySetAction());
this.store.dispatch(new queryAction.InputValueChangeAction('since:day'));
    }

    private getDataFromStore() {
this.apiResponseHashtags$ = this.store.select(fromRoot.getApiResponseTags);
    }
}

HTML File

<div class="top-hashtags">
    <div *ngIf="(apiResponseHashtags$ | async).length !== 0">
        <span *ngFor ="let item of (apiResponseHashtags$ | async)">
            <a [routerLink]="['/search']" [queryParams]="{ query : '#' + item.tag }">#{{item.tag}}</a>&nbsp;
        </span>
    </div>
</div>

Conclusion

Whenever the template data depends on the API response or user interaction, it is beneficial to use ChangeDetectionStrategy  and reduce template rendering time. Moreover, instead of subscribing to observables of data received from the API, we will be using an async pipe to make observable as an input for the template.

Resources

  • https://angular-2-training-book.rangle.io/handout/change-detection/change_detection_strategy_onpush.html
  • https://stackoverflow.com/questions/39795634/angular-2-change-detection-and-changedetectionstrategy-onpush/39802466#39802466
  • https://www.youtube.com/watch?v=X0DLP_rktsc

 

 

 

Continue Reading

Routing in Loklak Search SPA Hosted on GitHub-Pages

Single page applications like Loklak Search, are the ones which only have a single physical HTML page but has many different logical pages. This is achieved with the help of JavaScript, and angular provides strong abstraction to help the developers write the logical pages and Angular Router manages which page to display when a particular view is demanded. The Angular Router manages the URL state and the state of corresponding components.

Overview:

Now it’s important to understand how the routing works. In every web application structure, there is an index.html file, for SPA this is the only one single physical HTML page. This page is served when the base url is requested, ie. the web server gives this file to the client when the client requests for the base url of the application, in our case http://loklak.net/, This index.html page contains all the JS and CSS required by the application to work, which are loaded, as normal HTTP requests. When the javascript is loaded, the control of the application is governed by Angular which manages all the components, views and routes. When a user demands some route, the angular updates the view with the required components that are required to show the contents of that route.

 

It’s important to note here that no further simple requests are made by the client for any resource, angular is the one which controls all such requests. When we define routes in angular, say /search, we tell it what is the component which should be shown to the user when the route is /search, whenever the user navigates to such route angular shows the appropriate component, there is no request made by the client to the server for any such routes.

 

In Fact, the server doesn’t even know of such route, as they are defined in angular itself not, on the server side. The important fact is now what happens when the client refreshes the page when route /search is the URL, now the request is sent to the server which doesn’t know which page to serve when such request comes, so it will throw a 404: Page Not Found error.

Solution:

The solution to this situation is pretty straight forward, the server instead of throwing the 404 page, returns with the same index.html page. This makes the problem go away entirely, the server responds with the same page always, this page has the javascript to load and give control to angular, which when loaded, resolves which page to show according to the URL. This is a clean simple solution to the SPA routing problem. But this only works when you have access to Server Code, and you can request the server to respond with the same page whatever be the URL.

The problem with gh-pages:

But Loklak Search is deployed on gh-pages, and thus we don’t have any access to the server side code, to display the page of our requirements. GitHub pages just serve the HTML file which matches the corresponding URL, plain simple server. Now if on gh-pages we request for /search route it will respond with a 404 Error, as it is unable to find the search.html file in the directory.

How it is achieved in Loklak Search?

Loklak search is also deployed on gh-pages, to have a workaround to this problem we take advantage of the page served by gh-pages when the requested page is not found. When 404 error occur gh-pages responds with the 404.html file if it exists in the repository, else it responds with a default 404 page. We use this 404.html page to be able to load our main index.html page.

 

Here we have our 404.html file which is served by gh-pages whenever some url which is not unknown is requested

  • Here we first store the actual location which was requested by the user in the session storage.
  • Then we do a Redirective Refresh using the meta tag, to the root page ie. to load index.html which is known to gh-pages and is served.

https://gist.github.com/hemantjadon/9581dd0b4907c567b2a90eb949c5cbbc.js

<!-- 404.html -->
<!doctype html>
<html>
   <head>
  <!-- This stores the URL the user was attempting to go to in sessionStorage, and then redirects all 404 responses to the app’s index.html page -->
      <scrpt>
         sessionStorage.redirect = location.href;
      </scrpt>
         <meta http-equiv="refresh" content="0;URL='http://loklak.net/'">
      </meta>
   </head>
   <body>
   </body>
</html>

And then in our index.html after loading the angular and all the resources, we run a function, it takes out the redirect key stored in session storage, and compare it to the current location.

If the locations don’t match, we replace the history state with the with the redirect and this time as angular is already loaded it knows which page to show to the user.

<!-- index.html -->

...

...

<scrpt>

   (function(){
     var redirect = sessionStorage.redirect;
     delete sessionStorage.redirect;
     if (redirect && redirect != location.href) {
        history.replaceState(null, null, redirect);
     }
   })();

</scrpt>

...

...

Resources and links

Continue Reading

Visualising Data acquired by loklak Aggregation API

Data is always more interesting to look at when it is visualised by means of graphs and various plots rather than just viewing data as raw json or xml. There are various open source libraries available for visualising data. The one which I will be using today is called morris.js. Although it does not provide a lot of plots but it is extremely simple to use and can be of great help when quick data visualisation is required.

This blog shows one of the many ways in which data obtained from loklak service can be visualised using morris.js. This is same library which I am using presently to build MultiLinePlotter app whicch will soon be displayed on Loklak app store.

What is morris.js?

It is an open source library for drawing graphs using javascript. Presently it provides APIs to draw line charts, area charts, bar charts and donut charts. To know more about morris.js please visit this link.

Loklak aggregation service

A loklak search result can also contain field aggregations if they are requested in the api call. Fields to be aggregated must be listed in the fields attribute in the api request. The following url is an example for requesting aggregation on hashtags and mentions:

http://api.loklak.org/api/search.json?q=spacex%20since:2015-04-01%20until:2015-04-06&source=cache&count=0&fields=mentions,hashtags&limit=6

‘created_at’ is a special field which will return a date histogram if mentioned in the fields attribute. To know more about the loklak aggregation service and the loklak project in general please visit this link.

Getting started

An example of morris.js can be found here. Feel free to browse this code to see morris.js in working.

So what we will be doing today is plotting a simple bar chart using morris.js and the data returned by loklak aggregation service. For this we will be using the following api call :

https://api.loklak.org/api/search.json?callback=JSON_CALLBACK&q=fossasia%20since:2015-12-10&source=cache&count=0&fields=created_at

The above request will return an date histogram for the query fossasia, that is, date vs number of tweets containing the term ‘fossasia’.

So the first thing that we need to do is create a directory for this project and create a file named index.html in it. This is the only file we will be requiring.

Next we need to add morris.js and its dependencies:

<link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/morris.js/0.5.1/morris.css">
    <script type="text/javascript" src="//ajax.googleapis.com/ajax/libs/jquery/1.9.0/jquery.min.js"></script>
    <script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/raphael/2.1.0/raphael-min.js"></script>
    <script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/morris.js/0.5.1/morris.min.js"></script>

This will add the css and js script files for morris.js and its dependencies : Jquery and Raphael.
Next we need to add the div element that will contain our chart.

<div id="myfirstchart" style="height: 380px;margin-top:50px;overflow-x: auto;"></div>

The most important thing to note over here is that the div must have an id so that we can refer it later from our js code.

Next comes the most important and interesting part : adding the js code that will fetch the data and plot the bar chart.
The first thing that the script will do is make an ajax call to the above mentioned url and get the data.This is done using the following code :

$.ajax({
          url: "https://api.loklak.org/api/search.json?callback=JSON_CALLBACK&q=fossasia%20since:2015-12-10&source=cache&count=0&fields=created_at",
          jsonp: "callback",
          dataType: "jsonp",
          success: function( response ) {
              data=[];
              for (var date in response.aggregations.created_at) {
                  data.push({day: date, tweets: response.aggregations.created_at[date]});
              }
              console.log(data);
              plotData(data);
          }
      });

It is to be noted here that a callback function is provided in the url and data type is set to jsonp and not json.This is done to avoid cross origin resource sharing (CORS) issue so that our browser does not block the ajax request.
Once our request is successful we populate the data array from the response returned to us.
We iterate over all the key value pairs present in the created_at object and add them as individual objects in the data array. This will be required in the next part.

Next we have to plot the data acquired by us. For this we will be using plotData(data) function as written below:

function plotData(data) {
          Morris.Bar({
              element: 'myfirstchart',
            data: data,
            xkey: 'day',
       ykeys: ['tweets'],
            labels: ['tweets']
          });
      }

We send this function the data array we created earlier. Inside this function we create a Morris.Bar object which takes an object as parameter. The object has the following properties:

‘element’ : This takes the id of the div where we want to load the chart.

‘data’ : This is the data array we created earlier and passed to this function. It is the data to plot.It is an array of objects containing the x and y  attributes.

xkey’ : A string containing the name of the attribute that contains X labels.

‘ykeys’ : A list of strings containing names of attributes that contain Y values (one for each series of data to be plotted).

‘labels’ : A list of strings containing labels for the data series to be plotted (corresponding to the values in the ykeys option).

That’s it. These are the basic parameters required to plot a bar chart using morris.js. Simple enough, isn’t it!!

Here is the complete js code:

<script type="text/javascript">
      $.ajax({
          url: "https://api.loklak.org/api/search.json?callback=JSON_CALLBACK&q=fossasia%20since:2015-12-10&source=cache&count=0&fields=created_at",
          jsonp: "callback",
          dataType: "jsonp",
          success: function( response ) {
              data=[];
              for (var date in response.aggregations.created_at) {
                  data.push({day: date, tweets: response.aggregations.created_at[date]});
              }
              console.log(data);
              plotData(data);
          }
      });
      function plotData(data) {
          Morris.Bar({
              element: 'myfirstchart',
              data: data,
              xkey: 'day',
              ykeys: ['tweets'],
              labels: ['tweets']
          });
      }
    </script>

Next, open terminal and navigate to your project directory containing the index.html file, then execute the following command:

Python -m SimpleHTTPServer

As the command suggests, it starts a simple HTTP server at port 8000.

So next ,as you might have already guessed, fire up your favourite browser and visit http://127.0.0.1:8000 and you will be able to see the chart present below.

Python’s simple HTTP server comes in handy if you are developing a frontend app and you want to test it on real small screen devices. For example if you want to test it on your mobile then just connect your computer and your mobile to your router, start python http server on your computer, find out your computer’s ip address (ifconfig) and visit http://<computer’s ip address>:8000 on your mobile. You will be able to view your app on your mobile. If you don’t use a router then create_ap may help.

Important Resources

  • Find more information on morris.js here
  • Visit Loklak apps GitHub page here
  • You can find a similar implementation of the above tutorial here
Continue Reading

Creating Middle Section of Listing Page for loklak Apps Store

The Loklak applications website that is apps.loklak.org now has a fully functional store listing page where users and developers can find various information about the app they have selected including information about getting started with the app, use of the app and other relevant information. Developers can showcase their apps and promote their apps with promo image and preview images.

Before beginning with the actual topic let me provide a brief overview of the various components on the page. The page consists of a left sidebar which contains various categories for app filtering. There is a right column showing suggested apps so that users can easily view apps which are similar to the one they have chosen. Finally there is a middle section containing app promo image, app title, short description, author’s name, a carousel showing preview images, a getting started section, an app use section and an other relevant information section and finally some additional information.

The main feature and requirement of the middle section is that it needs to be dynamic, that is, it should dynamically show information for a given selected app. No content should be hardcoded.

Now apps.loklak.org is entirely a front end project, there is no centralised data base from where the content can be loaded. How to solve this problem? From where to get the data related to each app. Well, here comes in the app.json file present in each app. Each application present on apps.loklak.org contains an app.json file which contains some metadata about the apps. Now the app.json can be modified by the developer of the app to contain relevant information and references to assets which will be used for app store listing.

Getting started with the middle section page

At first, let us have a look at a sample app.json file.

{
  "@context":"http://schema.org",
  "@type":"SoftwareApplication",
  "permissions":"/api/search.json",
  "name":"sentimentVisualizer",
  "headline":"Sentiment Visualizer",
  "alternativeHeadline":"Tool for visualizing the sentiment of a tweet",
  "applicationCategory":"Visualizer",
  "applicationSubCategory":"Text Retrieval",
  "operatingSystem":"http://loklak.org",
  "promoImage":"promo.png",
  "appImages":["disp1.png", "disp2.png", "disp3.png"],
  "getStarted":"getStarted.md",
  "appUse":"appUse.md",
  "oneLineDescription":"Application to visualise sentiment related to tweets",
  "appSource": "https://github.com/fossasia/apps.loklak.org/tree/master/sentimentVisualizer",
  "contributors": [{"name": "Damini Satya", "url": "https://github.com/daminisatya"}],
  "techStack": ["HTML", "CSS", "AngularJs", "Bootstrap", "Loklak API"],
  "license": {"name": "LGPL 2.1", "url": "https://www.gnu.org/licenses/old-licenses/lgpl-2.1"},
  "version": "1.0",
  "updated": "June 14,2017",
  "author":{
    "@type":"Person",
    "name":"Damini Satya",
    "email":"[email protected]",
    "url":"https://github.com/daminisatya",
    "sameAs":"https://github.com/daminisatya"
  }
}

Each of the apps must have an app.json like the above one. As it can be seen, it contains all the necessary information for sore listing, like app name, version, oneLineDescription, etc. It also contains paths to important assets.

Before proceeding we need to know the name of the app which has been selected. This is obtained from the url as shown below.

var addr = window.location + "";
if (addr.indexOf("?") !== -1) {
    $scope.appName = addr.split("?")[1].split("=")[1];
}

Once we get the app name, next thing which we need to do is load the contents of app.json.
This is done using AngularJs’s http service.

$http.get($scope.appName + "/app.json").success(function (data) {
        $scope.appData = data;
        $scope.setupCarousel();
        $scope.getStarted();
        $scope.appUse();
        $scope.others();
});

A http ajax call is made to app.json of the corresponding app and the entire data is stored in appData.

Before moving to the subsequent function calls let us take a look at the corresponding html code which creates the top section.

<div class="app-image-and-info">
              <div class="app-image animated fadeInDown">
                <img ng-src="../{{selectedApp.name}}/promo.png" class="img-rounded image-loaded img-responsive">
              </div>
              <div class="app-info">
                <h2 class="app-name"> {{selectedApp.name}} </h2>
                <h4 class="app-headline"> {{appData.headline}} </h4>
                <h5 class="author"> <a href={{appData.author.url}} class="author-link">
                  by {{appData.author.name}} </a> </h5>
                <div class="short-desc"> {{appData.oneLineDescription}} </div>
                <a href="../{{selectedApp.name}}" class="try-now"> Try Now </a>
              </div>
            </div>

It inserts the promo image, app name, one line description, author name and link and a ‘Try now’ button to try the live app.

In the last JS code snippet (last to last snippet) we saw some function calls. The first one is setupCarousel.

$scope.setupCarousel = function () {
        var items = "";
        var active = "";
        for (var i = 0; i < $scope.appData.appImages.length; i++) {
            var image = $scope.appData.appImages[i];
            active = i == 0 ? " active" : "";
            var item = "";
            item = "
+ active + "'> + $scope.selectedApp.name + "/" + image + "'>
"
; items += item; } $(document).ready(function () { $(".carousel-inner").html(items); }); }

This function iterates over all the preview image paths and creates a html string for the image carousel. Finally it inserts the html into the corresponding div.

The next three functions loads the text content from the respective file paths via ajax calls.
Let us see one of them as the other two are almost similar.

$scope.getStarted = function () {
        $http.get($scope.appName + "/" +$scope.appData.getStarted).success(function (data) {
            $(document).ready(function () {
                $(".get-started-md").html(converter.makeHtml(data));
            });
        });
}

Since this text contents will be used for showcasing purpose, raw text would look somewhat dull and boring. Developer’s should be able to customise them. But gain, we cannot go for css styling as the content is dynamically loaded, a single page is being used for all apps. What if developer is able to use markdown in their text? This will definitely solve the problem as developers will be able to format and style their content. So how do we render markdown to html? Well showdown.js allows us to do so. It dynamically renders markdown to html. This library has been used here to convert the markdown to html and finally insert it into the page using jquery.

After these three sections, there is a section showing some additional data like app version, last updated, contributors list, technology used, license, author’s website and mail etc. All these informations are fetched from app.json and displayed via AngularJS variables. The functional store listing page can be seen here

Important resources

Continue Reading

Creating a store listing page for Loklak Apps Store

The various apps built with the loklak api is presently hosted at apps.loklak.org. On visiting the site, the users are presented with an app wall from where users can select any app and the user will be taken to the live version of the app. However there is no store listing page available like other app sites (for ex. Google playstore) where information about the app like how to use the app, what the app does, getting started with the app can be found.

The first part of my GSOC project is to create such an app store listing page where users/developers can find  all relevant information about the selected app.

Proposed design of the store listing page

The store listing page will be designed in three column format. There will be a left sidebar which will contain a list of categories for navigation, the category to which the selected app belongs will be highlighted. There will be a right side bar which will contain a list of similar apps for suggestion. The middle section will contain all the necessary information about the selected application, promo image and app preview images. Apart from this the page will contain a navbar for basic navigation.

Getting started with the store listing page implementation

Let’s get started with the left sidebar. Target is to make the entire page as dynamic as possible. Minimum information will be hard coded in html  and js code. Maximum data will be fetched from external json files using ajax requests. In this way if we need to make any change to data, it can be done by simply changing the data in the json files, the html and js code can be left intact.

First the store listing page has to know which app has been selected by the user. For this the app name is sent as an url parameter to the store listing page. The store listing page itself will have url like this http://apps.loklak.org/details.html?q=<app_name>. From here we can easily get the app name as shown below.

var addr = window.location + "";
if (addr.indexOf("?") !== -1) {
    $scope.appName = addr.split("?")[1].split("=")[1];
}

The category names, icon, and color are loaded from apps.json file. The apps.json file is one of the most important files in the repository which manages app and site meta data. It contains list of category objects each of which contains category name, category style and category icon reference.

$http.get('apps.json').success(function (data) {
        $scope.categories = data.categories;
        $scope.apps = data.apps;
        $scope.categories.unshift({"name": "All","image":"all.png","style" :
            {"background-color": "#ED3B3B"}});
        $scope.getSelectedApp();
        $scope.getSimilarApps();
    });

Using the above code we get all the categories and apps and bind them with the scope variable so that we can access them from the corresponding HTML.

Next we need to actually setup the UI for the sidebar using the data we have acquired. Well here comes in AngularJs’s one of the most useful feature – two way data binding. Using two way data binding we can easily access the category list from HTML code, iterate over it and set up the UI.

<div class="category-body">
              <ul class="menu-list">
                <li class="menu-list-item" ng-repeat="category in categories track by $index">
                  <a href="/#{{category.name.split(' ').join('')}}" class="menu-item" id="{{category.name | nospace}}"
                    ng-class="{selected: category.name.split(' ').join('') ==
                      selectedApp.applicationCategory.split(' ').join('')}">
                    <span class="category-image" ng-style = {{category.style}}>
                      <img ng-src="images/sidebar-images/{{category.image}}">
                    </span>
                    <span class="category-name"> {{category.name}} </span>
                  </a>
                </li>
              </ul>
            </div>

As it can be seen from the above code ng-class is being used to conditionally add a class named ‘selected’ for highlighting the category to which the selected app belongs. Each icon is given its own custom background color using ng-style, and the source is set using ng-src. Clicking on an category will take the user to the main page (app wall) and the user will be presented with only those apps which belong to the category which the user previously selected.

Creating a suggestion list

Next step is creating the right side column. It will contain all the other apps which are similar to the one selected by the user. For this we need to filter out all the apps which belongs to the same category to which the user selected app belongs. This can be done easily by using the list of apps which we have got from apps.json file. We can simply iterate over all the apps and filter the apps from the required category into another list.

$scope.getSimilarApps = function() {
        $scope.apps.forEach(function (item) {
            if (item.applicationCategory === $scope.selectedApp.applicationCategory
                && item.name !== $scope.selectedApp.name) {
                $scope.similarApps.push(item);
            }
        });
    }

This small function does the task of filtering and populates the similarApps list.

Next, once again we need to setup the UI using the data AngularJs acquired. For this we again iterate over the similarApps list and this time we set up cards to hold the app image and name.

<div class="similar-apps">
            <div class="similar-apps-header">
              <h4> Similar apps </h4>
            </div>
            <div class="similar-apps-list">
              <div ng-repeat="app in similarApps" class="card fadeIn">
                <a href="../details.html?q={{app.name}}">
                  <img ng-src="../{{app.name}}/screenshot.png" class="similar-app-image" alt="App image">
                  <div class="card-container">
                    <h4>{{app.headline}}</h4>
                  </div>
                </a>
              </div>
            </div>
          </div>

On clicking on any of the card, the store listing page of the corresponding app will be shown.

Important resources

  •  Find more information of AngularJS and AngularJS services here.
Continue Reading

Editing files and piped data with “sed” in Loklak server

What is sed ?

“sed” is used in “Loklak Server” one of the most popular projects of FOSSASIA. “sed” is acronym for “Stream Editor” used for filtering and transforming text, as mentioned in the manual page of sed. Stream can be a file or input from a pipeline or even standard input. Regular expressions are used to filter the text and transformation are carried out using sed commands, either inline or from a file. So, most of the time writing a single line does the work of text substitution, removal or to obtaining a value from a text file.

Basic Syntax of “sed”

$sed [options]... {inline commands or file having sed commands} [input_file]...

Loklak Server uses a config.properties file – contains key-value pairs – which is the basis of the server as it contains configuration values, used by the server during runtime for various operations.

Let’s go through a simple sed example that prints line containing word “https” at the beginning in the config.properties file.

$sed -n '/^https/p' config.properties

Here “-n” option suppresses automatically printing of pattern space (pattern space is where each line is put that is to be processed by sed). Without “-n” option sed will print the whole file.

Now, the regular expression part,  “/^https” matches all the lines that has “https” at the start of line and “/p” is print command to print the output in console. Finally we provide the filename i.e. config.properties. If filename is not provided then sed waits for input from standard input.

Use cases of “sed” in Loklak Server

  • Displaying proper port number in message while starting or installing Loklak Server

The default port of loklak server is port number 9000, but it can be started in any non-occupied port by using “-p” flag with bin/start.sh and bin/installation.sh like

$ bin/installation.sh -p 8888

starts installation of Loklak Server in port 8888. To display the proper localhost address so that user can open it in a browser the port number in shortlink.urlstub parameter in config.properties needs to be changed. This is carried out by the function change_shortlink_urlstub in bin/utility.sh. The function is defined as

Now let’s try to understand what the sed command is doing.

“-i” option is used for in-place editing of the specified file i.e. config.properties in conf directory.

s” is substitute command of sed. The regular expression can be divided into two parts, between “/”:

  1. \(shortlink\.urlstub=http:.*:\)\(.*\) this is used to find the match in a line.
  2. \1′”$1″ is used to substitute the matched string in part 1.

The regular expressions can be split into groups so that operations can be performed on each group separately. A group is enclosed between “\(“ and “\)”. In our 1st part of regular expressions, there are two groups.

Dissecting the first group i.e. \(shortlink\.urlstub=http:.*:\):

  • shortlink\.urlstub=http:” will match the expression “shortlink.urlstub=http:”, here “\” is used as an escape sequence as “.” in regex represents any character.
  • “.*:”, “.” represents any character and “*” represents 0 or more characters of the previous character. So, it will basically match any expression. But, it ends with a “:”, which means any expression that ends with a “:”. Thus it matches the expression “//localhost:”.

So, the first group collectively matches the expression “shortlink.urlstub=http://localhost:”.

As described above second group i.e. \(.*\) will match any expression, here matches “9000”.

Now coming to the 2nd part of regular expression i.e. \1′”$1″:

  • “\1” represents the match of the first group in 1st part i.e. “shortlink.urlstub=http://localhost:”.
  • “$1” is the value of the first parameter provided to our function “change_shortlink_urlstub” which is an unused port where we want to run Loklak Server.

So 2nd part picks up the match from the first group and concatenates with the first parameter of the function. Assuming the first parameter to the function is “8888”, the expression for 2nd part becomes “shortlink.urlstub=http://localhost:8888” which replaces “shortlink.urlstub=http://localhost:9000”.

So a correct localhost address is displayed in the console while starting or installing Loklak Server.

  • Extracting value of a key from config.properties

“grep” and “sed” are used to extract the values of key from config.properties in bash scripts e.g. extracting default port, value of “port.http” in bin/start.sh, value of “shortlink.urlstub” in bin/start.sh and bin/installation.sh.

$ grep -iw 'port.http' conf/config.properties | sed 's/^[^=]*=//'

Here grep is used to filter a line and pass the filtered line to sed by piping the output. “i” flag is for ignoring case sensitivity and “w” flag is used for matching of word only. The output of

$ grep -iw 'port.http' conf/config.properties
port.http=9000

The aim is to get “9000”, the value of “port.http”. The approach used here is to substitute “port.http=” in the output of grep command above with a string of zero characters, that way only “9000” remains.

Let’s deconstruct the “sed” part, s/^[^=]*=//:

s” command of sed is used for substitution.  Here 1st part is “^[^=]*=” and 2nd part is nothing, as no characters are enclosed within “//”.

  • “^” means to match at the start.
  • [^characters] represents not to consider a set of characters. For matching a set of characters “^” is not used inside square brackets, [characters]. Here [^=] means not to include equal to – “=” symbol – and “*” after that makes it to match characters that are not “=”.

So “^[^=]*=” matches a sequence of characters that doesn’t start with “=” and followed by “=”. Thus the matched expression in this case is “http.port=” (“h” is not “=” and it ends with “=”), which is substituted by a string with zero characters which leaves “9000”.  Finally, “9000” is assigned to the required variable is bash script.

Continue Reading
Close Menu