Visualising Tweet Statistics in MultiLinePlotter App for Loklak Apps

MultiLinePlotter app is now a part of Loklak apps site. This app can be used to compare aggregations of tweets containing a particular query word and visualise the data for better comparison. Recently there has been a new addition to the app. A feature for showing tweet statistics like the maximum number of tweets (along with date) containing the given query word and the average number of tweets over a period of time. Such statistics is visualised for all the query words for better comparison.

Related issue: https://github.com/fossasia/apps.loklak.org/issues/236

Obtaining Maximum number of tweets and average number of tweets

Before visualising the statistics we need to obtain them. For this we simply need to process the aggregations returned by the Loklak API. Let us start with maximum number of tweets containing the given keyword. What we actually require is what is the maximum number of tweets that were posted and contained the user given keyword and on which date the number was maximum. For this we can use a function which will iterate over all the aggregations and return the largest along with date.

$scope.getMaxTweetNumAndDate = function(aggregations) {
        var maxTweetDate = null;
        var maxTweetNum = -1;

        for (date in aggregations) {
            if (aggregations[date] > maxTweetNum) {
                maxTweetNum = aggregations[date];
                maxTweetDate = date;
            }
        }

        return {date: maxTweetDate, count: maxTweetNum};
    }

The above function maintains two variables, one for maximum number of tweets and another for date. We iterate over all the aggregations and for each aggregation we compare the number of tweets with the value stored in the maxTweetNum variable. If the current value is more than the value stored in that variable then we simply update it and keep track of the date. Finally we return an object containing both maximum number of tweets and the corresponding date.Next we need to obtain average number of tweets. We can do this by summing up all the tweet frequencies and dividing it by number of aggregations.

$scope.getAverageTweetNum = function(aggregations) {
        var avg = 0;
        var sum = 0;

        for (date in aggregations) {
            sum += aggregations[date];
        }

        return parseInt(sum / Object.keys(aggregations).length);
    }

The above function calculates average number of tweets in the way mentioned before the snippet.

Next for every tweet we need to store these values in a format which can easily be understood by morris.js. For this we use a list and store the statistics values for individual query words as objects and later pass it as a parameter to morris.

var maxStat = $scope.getMaxTweetNumAndDate(aggregations);
        var avg = $scope.getAverageTweetNum(aggregations);

        $scope.tweetStat.push({
            tweet: $scope.tweet,
            maxTweetCount: maxStat.count,
            maxTweetOn: maxStat.date,
            averageTweetsPerDay: avg,
            aggregationsLength: Object.keys(aggregations).length
        });

We maintain a list called tweetStat and the list contains objects which stores the query word and the corresponding values.

Apart from plotting these statistics, the app also displays the statistics when user clicks on an individual treat present in the search record section. For this we filter tweetStat list mentioned above and get the required object corresponding to the query word the user selected bind it to angular scope. Next we display it using HTML.

<div class="tweet-stat max-tweet">
                  <div class="stat-label"> <h4>Maximum number of tweets containing '{{modalHeading}}' :</h4></div>
                  <div class="stat-value"> <strong>{{selectedTweetStat.maxTweetCount}}</strong> tweets on
                    <strong>{{selectedTweetStat.maxTweetOn}}</strong>
                  </div>
                </div>

Finally we need to plot the statistics. For this we use a function called plotStatGraph dedicated only for plotting statistics graph. We pass the tweetStat list as a parameter to morris and configure all the other parameters.

$scope.plotStatGraph = function() {
        $scope.plotStat = new Morris.Bar({
            element: 'graph',
            data: $scope.tweetStat,
            xkey: 'tweet',
            ykeys: ['maxTweetCount', 'averageTweetsPerDay'],
            labels: ['Maximum no. of tweets : ', 'Average no. of tweets/day'],
            parseTime: false,
            hideHover: 'auto',
            resize: true,
            stacked: true,
            barSizeRatio: 0.40
        });
        $scope.graphLoading = false;
    }

But now we have two graphs. One for showing variations in aggregation and the other for showing statistics. How do we manage them? Somehow we need to show them in the same page as this is a single page app. Also we need to avoid vertical scrolling as it would degrade both UI and UX. So we need to implement a switching mechanism. The user should be able to switch between the two graph views as per their wish. How to achieve that? Well, for this we maintain a global variable which will keep track of the current plot type. If the current graph type is aggregations then we call the function to plot aggregations otherwise we call the above mentioned function to plot statistics.

$scope.plotData = function() {
        $(".plot-data").html("");
        if ($scope.currentGraphType === "aggregations") {
            $scope.plotAggregationGraph();
        } else {
            $scope.plotStatGraph();
        }
    }

Lastly we integrate this state variable (currentGraphType) with the UI so that users can easily toggle between graph views with just a click.

<div class="switch" ng-click="toggle()">
                <span ng-if="queryRecords.length !== 0" class="glyphicon glyphicon-stats"></span>
              </div>

Important resources

Developing MultiLinePlotter App for Loklak

MultiLinePlotter is a web application which uses Loklak API under the hood to plot multiple tweet aggregations related to different user provided query words in the same graph. The user can give several query words and multiple lines for different queries will be plotted in the same graph. In this way, users will be able to compare tweet distribution for various keywords and visualise the comparison. All the searched queries are shown under the search record section. Clicking on a record causes a dialogue box to pop up where the individual tweets related to the query word is displayed. Users can also remove a series from the plot dynamically by just pressing the Remove button beside the query word in record section. The app is presently hosted on Loklak apps site.

Related issue – https://github.com/fossasia/apps.loklak.org/issues/225

Getting started with the app

Let us delve into the working of the app. The app uses Loklak aggregation API to get the data.

A call to the API looks something like this:

http://api.loklak.org/api/search.json?q=fossasia&source=cache&count=0&fields=created_at

A small snippet of the aggregation returned by the above API request is shown below.

"aggregations": {"created_at": {
    "2017-07-03": 3,
    "2017-07-04": 9,
    "2017-07-05": 12,
    "2017-07-06": 8,
}}

The API provides a nice date v/s number of tweets aggregation. Now we need to plot this. For plotting Morris.js has been used. It is a lightweight javascript library for visualising data.

One of the main features of this app is addition and removal of multiple series from the graph dynamically. How do we achieve that? Well, this can be achieved by manipulating the morris.js data list whenever a new query is made. Let us understand this in steps.

At first, the data is fetched using angular HTTP service.

$http.jsonp('http://api.loklak.org/api/search.json?callback=JSON_CALLBACK',
            {params: {q: $scope.tweet, source: 'cache', count: '0', fields: 'created_at'}})
                .then(function (response) {
                    $scope.getData(response.data.aggregations.created_at);
                    $scope.plotData();
                    $scope.queryRecords.push($scope.tweet);
                });

Once we get the data, getData function is called and the aggregation data is passed to it. The query word is also stored in queryRecords list for future use.

In order to plot a line graph morris.js requires a data object which will contain the required values for a series. Given below is an example of such a data object.

data: [
    { x: '2006', a: 100, b: 90 },
    { x: '2007', a: 75,  b: 65 },
    { x: '2008', a: 50,  b: 40 },
    { x: '2009', a: 75,  b: 65 },
],

For every ‘x’, ‘a’ and ‘b’ will be plotted. Thus two lines will be drawn. Our app will also maintain a data list like the one shown above, however, in our case, the data objects will have a variable number of keys. One key will determine the ‘x’ value and other keys will determine the ordinates (number of tweets).

All the data objects present in the data list needs to be updated whenever a new search is done.

The getData function does this for us.

var value = $scope.tweet;
        for (date in aggregations) {
            var present = false;
            for (var i = 0; i < $scope.data.length; i++) {
                var item = $scope.data[i];
                if (item['day'] === date) {
                    item[[value]] = aggregations[date];
                    $scope.data[i] = item
                    present = true;
                    break;
                }
            }
            if (!present) {
                $scope.data.push({day: date, [value]: aggregations[date]});
            }
        }


The for loop in the above code snippet updates the global data list used by morris.js. It simply iterates over the dates in the aggregation, extracts the object corresponding to a particular date, adds the new query word as a key and, the number of tweets on that date as the value.If a date is not already present in the list, then it inserts a new object corresponding to the date and query word. Once our data list is updated, we are ready to redraw the graph with the updated data. This is done using plotData function. The plotData function simply checks the user selected graph type. If the selected type is aggregations then it calls plotAggregationGraph() to redraw the aggregations plot.

$scope.remove = function(record) {
        $scope.queryRecords = $scope.queryRecords.filter(function(e) {
            return e !== record });

        $scope.data.forEach(function(item) {
            delete item[record];
        });

        $scope.data = $scope.data.filter(function(item) {
            return Object.keys(item).length !== 1;
        });

        $scope.ykeys = $scope.ykeys.filter(function(item) {
            return item !== record;
        });

        $scope.labels = $scope.labels.filter(function(item) {
            return item !== record;
        });

        $scope.plotData();
}

The above function simply scans the data list, filters the objects which contains selected record as a key and removes them using filter method of javascript arrays. It also removes the corresponding labels and entries from labels and ykeys arrays. Finally, it once again calls plotData function to redraw the plot.

Given below is a sample plot generated by this app with the query words – google, android, microsoft, samsung.

 

Conclusion

This blog post explained how multiple series are plotted dynamically in the MultiLinePlotter app. Apart from aggregations plot it also plots tweet statistics like maximum tweets and average tweets containing a query word and visualises them using stacked bar chart. I will be discussing about them in my subsequent blogs.

Important resources

Visualising Data acquired by loklak Aggregation API

Data is always more interesting to look at when it is visualised by means of graphs and various plots rather than just viewing data as raw json or xml. There are various open source libraries available for visualising data. The one which I will be using today is called morris.js. Although it does not provide a lot of plots but it is extremely simple to use and can be of great help when quick data visualisation is required.

This blog shows one of the many ways in which data obtained from loklak service can be visualised using morris.js. This is same library which I am using presently to build MultiLinePlotter app whicch will soon be displayed on Loklak app store.

What is morris.js?

It is an open source library for drawing graphs using javascript. Presently it provides APIs to draw line charts, area charts, bar charts and donut charts. To know more about morris.js please visit this link.

Loklak aggregation service

A loklak search result can also contain field aggregations if they are requested in the api call. Fields to be aggregated must be listed in the fields attribute in the api request. The following url is an example for requesting aggregation on hashtags and mentions:

http://api.loklak.org/api/search.json?q=spacex%20since:2015-04-01%20until:2015-04-06&source=cache&count=0&fields=mentions,hashtags&limit=6

‘created_at’ is a special field which will return a date histogram if mentioned in the fields attribute. To know more about the loklak aggregation service and the loklak project in general please visit this link.

Getting started

An example of morris.js can be found here. Feel free to browse this code to see morris.js in working.

So what we will be doing today is plotting a simple bar chart using morris.js and the data returned by loklak aggregation service. For this we will be using the following api call :

https://api.loklak.org/api/search.json?callback=JSON_CALLBACK&q=fossasia%20since:2015-12-10&source=cache&count=0&fields=created_at

The above request will return an date histogram for the query fossasia, that is, date vs number of tweets containing the term ‘fossasia’.

So the first thing that we need to do is create a directory for this project and create a file named index.html in it. This is the only file we will be requiring.

Next we need to add morris.js and its dependencies:

<link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/morris.js/0.5.1/morris.css">
    <script type="text/javascript" src="//ajax.googleapis.com/ajax/libs/jquery/1.9.0/jquery.min.js"></script>
    <script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/raphael/2.1.0/raphael-min.js"></script>
    <script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/morris.js/0.5.1/morris.min.js"></script>

This will add the css and js script files for morris.js and its dependencies : Jquery and Raphael.
Next we need to add the div element that will contain our chart.

<div id="myfirstchart" style="height: 380px;margin-top:50px;overflow-x: auto;"></div>

The most important thing to note over here is that the div must have an id so that we can refer it later from our js code.

Next comes the most important and interesting part : adding the js code that will fetch the data and plot the bar chart.
The first thing that the script will do is make an ajax call to the above mentioned url and get the data.This is done using the following code :

$.ajax({
          url: "https://api.loklak.org/api/search.json?callback=JSON_CALLBACK&q=fossasia%20since:2015-12-10&source=cache&count=0&fields=created_at",
          jsonp: "callback",
          dataType: "jsonp",
          success: function( response ) {
              data=[];
              for (var date in response.aggregations.created_at) {
                  data.push({day: date, tweets: response.aggregations.created_at[date]});
              }
              console.log(data);
              plotData(data);
          }
      });

It is to be noted here that a callback function is provided in the url and data type is set to jsonp and not json.This is done to avoid cross origin resource sharing (CORS) issue so that our browser does not block the ajax request.
Once our request is successful we populate the data array from the response returned to us.
We iterate over all the key value pairs present in the created_at object and add them as individual objects in the data array. This will be required in the next part.

Next we have to plot the data acquired by us. For this we will be using plotData(data) function as written below:

function plotData(data) {
          Morris.Bar({
              element: 'myfirstchart',
            data: data,
            xkey: 'day',
       ykeys: ['tweets'],
            labels: ['tweets']
          });
      }

We send this function the data array we created earlier. Inside this function we create a Morris.Bar object which takes an object as parameter. The object has the following properties:

‘element’ : This takes the id of the div where we want to load the chart.

‘data’ : This is the data array we created earlier and passed to this function. It is the data to plot.It is an array of objects containing the x and y  attributes.

xkey’ : A string containing the name of the attribute that contains X labels.

‘ykeys’ : A list of strings containing names of attributes that contain Y values (one for each series of data to be plotted).

‘labels’ : A list of strings containing labels for the data series to be plotted (corresponding to the values in the ykeys option).

That’s it. These are the basic parameters required to plot a bar chart using morris.js. Simple enough, isn’t it!!

Here is the complete js code:

<script type="text/javascript">
      $.ajax({
          url: "https://api.loklak.org/api/search.json?callback=JSON_CALLBACK&q=fossasia%20since:2015-12-10&source=cache&count=0&fields=created_at",
          jsonp: "callback",
          dataType: "jsonp",
          success: function( response ) {
              data=[];
              for (var date in response.aggregations.created_at) {
                  data.push({day: date, tweets: response.aggregations.created_at[date]});
              }
              console.log(data);
              plotData(data);
          }
      });
      function plotData(data) {
          Morris.Bar({
              element: 'myfirstchart',
              data: data,
              xkey: 'day',
              ykeys: ['tweets'],
              labels: ['tweets']
          });
      }
    </script>

Next, open terminal and navigate to your project directory containing the index.html file, then execute the following command:

Python -m SimpleHTTPServer

As the command suggests, it starts a simple HTTP server at port 8000.

So next ,as you might have already guessed, fire up your favourite browser and visit http://127.0.0.1:8000 and you will be able to see the chart present below.

Python’s simple HTTP server comes in handy if you are developing a frontend app and you want to test it on real small screen devices. For example if you want to test it on your mobile then just connect your computer and your mobile to your router, start python http server on your computer, find out your computer’s ip address (ifconfig) and visit http://<computer’s ip address>:8000 on your mobile. You will be able to view your app on your mobile. If you don’t use a router then create_ap may help.

Important Resources

  • Find more information on morris.js here
  • Visit Loklak apps GitHub page here
  • You can find a similar implementation of the above tutorial here

Creating Middle Section of Listing Page for loklak Apps Store

The Loklak applications website that is apps.loklak.org now has a fully functional store listing page where users and developers can find various information about the app they have selected including information about getting started with the app, use of the app and other relevant information. Developers can showcase their apps and promote their apps with promo image and preview images.

Before beginning with the actual topic let me provide a brief overview of the various components on the page. The page consists of a left sidebar which contains various categories for app filtering. There is a right column showing suggested apps so that users can easily view apps which are similar to the one they have chosen. Finally there is a middle section containing app promo image, app title, short description, author’s name, a carousel showing preview images, a getting started section, an app use section and an other relevant information section and finally some additional information.

The main feature and requirement of the middle section is that it needs to be dynamic, that is, it should dynamically show information for a given selected app. No content should be hardcoded.

Now apps.loklak.org is entirely a front end project, there is no centralised data base from where the content can be loaded. How to solve this problem? From where to get the data related to each app. Well, here comes in the app.json file present in each app. Each application present on apps.loklak.org contains an app.json file which contains some metadata about the apps. Now the app.json can be modified by the developer of the app to contain relevant information and references to assets which will be used for app store listing.

Getting started with the middle section page

At first, let us have a look at a sample app.json file.

{
  "@context":"http://schema.org",
  "@type":"SoftwareApplication",
  "permissions":"/api/search.json",
  "name":"sentimentVisualizer",
  "headline":"Sentiment Visualizer",
  "alternativeHeadline":"Tool for visualizing the sentiment of a tweet",
  "applicationCategory":"Visualizer",
  "applicationSubCategory":"Text Retrieval",
  "operatingSystem":"http://loklak.org",
  "promoImage":"promo.png",
  "appImages":["disp1.png", "disp2.png", "disp3.png"],
  "getStarted":"getStarted.md",
  "appUse":"appUse.md",
  "oneLineDescription":"Application to visualise sentiment related to tweets",
  "appSource": "https://github.com/fossasia/apps.loklak.org/tree/master/sentimentVisualizer",
  "contributors": [{"name": "Damini Satya", "url": "https://github.com/daminisatya"}],
  "techStack": ["HTML", "CSS", "AngularJs", "Bootstrap", "Loklak API"],
  "license": {"name": "LGPL 2.1", "url": "https://www.gnu.org/licenses/old-licenses/lgpl-2.1"},
  "version": "1.0",
  "updated": "June 14,2017",
  "author":{
    "@type":"Person",
    "name":"Damini Satya",
    "email":"[email protected]",
    "url":"https://github.com/daminisatya",
    "sameAs":"https://github.com/daminisatya"
  }
}

Each of the apps must have an app.json like the above one. As it can be seen, it contains all the necessary information for sore listing, like app name, version, oneLineDescription, etc. It also contains paths to important assets.

Before proceeding we need to know the name of the app which has been selected. This is obtained from the url as shown below.

var addr = window.location + "";
if (addr.indexOf("?") !== -1) {
    $scope.appName = addr.split("?")[1].split("=")[1];
}

Once we get the app name, next thing which we need to do is load the contents of app.json.
This is done using AngularJs’s http service.

$http.get($scope.appName + "/app.json").success(function (data) {
        $scope.appData = data;
        $scope.setupCarousel();
        $scope.getStarted();
        $scope.appUse();
        $scope.others();
});

A http ajax call is made to app.json of the corresponding app and the entire data is stored in appData.

Before moving to the subsequent function calls let us take a look at the corresponding html code which creates the top section.

<div class="app-image-and-info">
              <div class="app-image animated fadeInDown">
                <img ng-src="../{{selectedApp.name}}/promo.png" class="img-rounded image-loaded img-responsive">
              </div>
              <div class="app-info">
                <h2 class="app-name"> {{selectedApp.name}} </h2>
                <h4 class="app-headline"> {{appData.headline}} </h4>
                <h5 class="author"> <a href={{appData.author.url}} class="author-link">
                  by {{appData.author.name}} </a> </h5>
                <div class="short-desc"> {{appData.oneLineDescription}} </div>
                <a href="../{{selectedApp.name}}" class="try-now"> Try Now </a>
              </div>
            </div>

It inserts the promo image, app name, one line description, author name and link and a ‘Try now’ button to try the live app.

In the last JS code snippet (last to last snippet) we saw some function calls. The first one is setupCarousel.

$scope.setupCarousel = function () {
        var items = "";
        var active = "";
        for (var i = 0; i < $scope.appData.appImages.length; i++) {
            var image = $scope.appData.appImages[i];
            active = i == 0 ? " active" : "";
            var item = "";
            item = "
+ active + "'> + $scope.selectedApp.name + "/" + image + "'>
"
; items += item; } $(document).ready(function () { $(".carousel-inner").html(items); }); }

This function iterates over all the preview image paths and creates a html string for the image carousel. Finally it inserts the html into the corresponding div.

The next three functions loads the text content from the respective file paths via ajax calls.
Let us see one of them as the other two are almost similar.

$scope.getStarted = function () {
        $http.get($scope.appName + "/" +$scope.appData.getStarted).success(function (data) {
            $(document).ready(function () {
                $(".get-started-md").html(converter.makeHtml(data));
            });
        });
}

Since this text contents will be used for showcasing purpose, raw text would look somewhat dull and boring. Developer’s should be able to customise them. But gain, we cannot go for css styling as the content is dynamically loaded, a single page is being used for all apps. What if developer is able to use markdown in their text? This will definitely solve the problem as developers will be able to format and style their content. So how do we render markdown to html? Well showdown.js allows us to do so. It dynamically renders markdown to html. This library has been used here to convert the markdown to html and finally insert it into the page using jquery.

After these three sections, there is a section showing some additional data like app version, last updated, contributors list, technology used, license, author’s website and mail etc. All these informations are fetched from app.json and displayed via AngularJS variables. The functional store listing page can be seen here

Important resources

Creating a store listing page for Loklak Apps Store

The various apps built with the loklak api is presently hosted at apps.loklak.org. On visiting the site, the users are presented with an app wall from where users can select any app and the user will be taken to the live version of the app. However there is no store listing page available like other app sites (for ex. Google playstore) where information about the app like how to use the app, what the app does, getting started with the app can be found.

The first part of my GSOC project is to create such an app store listing page where users/developers can find  all relevant information about the selected app.

Proposed design of the store listing page

The store listing page will be designed in three column format. There will be a left sidebar which will contain a list of categories for navigation, the category to which the selected app belongs will be highlighted. There will be a right side bar which will contain a list of similar apps for suggestion. The middle section will contain all the necessary information about the selected application, promo image and app preview images. Apart from this the page will contain a navbar for basic navigation.

Getting started with the store listing page implementation

Let’s get started with the left sidebar. Target is to make the entire page as dynamic as possible. Minimum information will be hard coded in html  and js code. Maximum data will be fetched from external json files using ajax requests. In this way if we need to make any change to data, it can be done by simply changing the data in the json files, the html and js code can be left intact.

First the store listing page has to know which app has been selected by the user. For this the app name is sent as an url parameter to the store listing page. The store listing page itself will have url like this http://apps.loklak.org/details.html?q=<app_name>. From here we can easily get the app name as shown below.

var addr = window.location + "";
if (addr.indexOf("?") !== -1) {
    $scope.appName = addr.split("?")[1].split("=")[1];
}

The category names, icon, and color are loaded from apps.json file. The apps.json file is one of the most important files in the repository which manages app and site meta data. It contains list of category objects each of which contains category name, category style and category icon reference.

$http.get('apps.json').success(function (data) {
        $scope.categories = data.categories;
        $scope.apps = data.apps;
        $scope.categories.unshift({"name": "All","image":"all.png","style" :
            {"background-color": "#ED3B3B"}});
        $scope.getSelectedApp();
        $scope.getSimilarApps();
    });

Using the above code we get all the categories and apps and bind them with the scope variable so that we can access them from the corresponding HTML.

Next we need to actually setup the UI for the sidebar using the data we have acquired. Well here comes in AngularJs’s one of the most useful feature – two way data binding. Using two way data binding we can easily access the category list from HTML code, iterate over it and set up the UI.

<div class="category-body">
              <ul class="menu-list">
                <li class="menu-list-item" ng-repeat="category in categories track by $index">
                  <a href="/#{{category.name.split(' ').join('')}}" class="menu-item" id="{{category.name | nospace}}"
                    ng-class="{selected: category.name.split(' ').join('') ==
                      selectedApp.applicationCategory.split(' ').join('')}">
                    <span class="category-image" ng-style = {{category.style}}>
                      <img ng-src="images/sidebar-images/{{category.image}}">
                    </span>
                    <span class="category-name"> {{category.name}} </span>
                  </a>
                </li>
              </ul>
            </div>

As it can be seen from the above code ng-class is being used to conditionally add a class named ‘selected’ for highlighting the category to which the selected app belongs. Each icon is given its own custom background color using ng-style, and the source is set using ng-src. Clicking on an category will take the user to the main page (app wall) and the user will be presented with only those apps which belong to the category which the user previously selected.

Creating a suggestion list

Next step is creating the right side column. It will contain all the other apps which are similar to the one selected by the user. For this we need to filter out all the apps which belongs to the same category to which the user selected app belongs. This can be done easily by using the list of apps which we have got from apps.json file. We can simply iterate over all the apps and filter the apps from the required category into another list.

$scope.getSimilarApps = function() {
        $scope.apps.forEach(function (item) {
            if (item.applicationCategory === $scope.selectedApp.applicationCategory
                && item.name !== $scope.selectedApp.name) {
                $scope.similarApps.push(item);
            }
        });
    }

This small function does the task of filtering and populates the similarApps list.

Next, once again we need to setup the UI using the data AngularJs acquired. For this we again iterate over the similarApps list and this time we set up cards to hold the app image and name.

<div class="similar-apps">
            <div class="similar-apps-header">
              <h4> Similar apps </h4>
            </div>
            <div class="similar-apps-list">
              <div ng-repeat="app in similarApps" class="card fadeIn">
                <a href="../details.html?q={{app.name}}">
                  <img ng-src="../{{app.name}}/screenshot.png" class="similar-app-image" alt="App image">
                  <div class="card-container">
                    <h4>{{app.headline}}</h4>
                  </div>
                </a>
              </div>
            </div>
          </div>

On clicking on any of the card, the store listing page of the corresponding app will be shown.

Important resources

  •  Find more information of AngularJS and AngularJS services here.

Creating a Script to Review loklak Apps

The Loklak applications site now has a functional store listing page where developers can showcase their apps and users and other developers can get all sorts of information about an app. However, for an app to be showcased properly on store listing page, the app must contain a properly configured app.json file and some necessary assets. But while creating an app a developer might miss out some vital information from app.json or forget to provide some of the required assets, which he will later come to know from his co-developers or reviewers.

Now this will cause inconvenience, both for the developer and reviewer. So to overcome this apps.loklak.org has now got a new script to review a given app. It checks whether the necessary fields are present in app.json or not. If present then it checks whether the fields are empty or not. If any invalid or missing information problem is encountered then it is reported to the developer along with information on what should be the actual case. If the app passes all the checks then the developer is informed that his app is ready to be published.

In order to use this script all the developer needs to do is open his app directory in terminal and execute the following command

../bin/review.sh

This will present the developer with all the necessary informations.

How the script works?

Now let us delve into the working of the script. The initial call to the shell script review.sh calls a python script review.py. This python script performs all the checks on the app and displays necessary informations.

The first thing the script does is, it checks whether there is an app.json present in the app directory or not, if yes then the process continues otherwise it ends immediately with an error.

if "app.json" not in dir_contents:
    print_info("Please include a well configured app.json")
    print_problem("review failed")
    exit(0)

Next it performs one of the most important checks. It verifies whether the name of the app and app directory name is same or not. If it is same then there is no problem else it shows the corresponding error message.

app_dir_name = os.getcwd().split("/")[-1]
    if app_dir_name != app_json.get("name"):
      print_problem("app directory name and name mentioned in app.json are different")
      print_info("app directory name and name mentioned in app.json must be same")
      problems_no += 1

Next the script checks whether there is an index.html present or not. Index.html is a must as it serves as the entry point of the app.

if “index.html” not in dir_contents:
print_problem(“index.html is missing”)
print_info(“app must contain index.html”)
problems_no += 1

After this the script checks whether a number of key fields are present or not. These field includes applicationCategory, oneLineDescription, author. If they are present, it checks whether the fields are empty or not. If the fields are either absent or empty, error is shown to the developer.

if app_json.get("applicationCategory") == None:
    print_problem("key applicationCategory missing in app.json")
    print_info("app.json must contain key applicationCategory with category as value")
    problems_no += 1
  else:
    if app_json.get("applicationCategory") == "":
      print_problem("applicationCategory cannot be an empty string")
      problems_no += 1
  
  if app_json.get("oneLineDescription") == None:
    print_problem("key oneLineDescription missing in app.json")
    print_info("app.json must contain key oneLineDescription with a one line description of the app")
    problems_no += 1
  else:
    if app_json.get("oneLineDescription") == "":
      print_problem("oneLineDescription cannot be an empty string")
      problems_no += 1
  
  if app_json.get("author") == None:
    print_problem("author object missing in app.json")
    print_info("app.json must contain author object containing information about author")
    problems_no += 1
  else:
    author = app_json.get("author")
    if author.get("name") == None:
      print_problem("name of author is mssing in author object")
      print_info("name of author must be mentioned in author object")
      problems_no += 1
    else:
      if author.get("name") == "":
        print_problem("author name cannot be an empty string")
        problems_no += 1

After the above checks are performed, it checks whether the fields corresponding to the image assets are present or not and whether the assets being referred in app.json exist in the directory. If not, then the developer is informed about the same.

if app_json.get("promoImage") == None:
    print_problem("key promoImage missing in app.json")
    print_info("app.json must contain key promoImage with path of promo image as value")
    problems_no += 1
  else:
    if os.path.isfile(app_json.get("promoImage")) == False:
      if app_json.get("promoImage") == "":
        print_problem("promoImage cannot be an empty string")
      else:
        print_problem(app_json.get("promoImage") + " does not exists")
      problems_no += 1
  
  if app_json.get("appImages") == None:
    print_problem("key appImages missing in app.json")
    print_info("app.json must contain key appImages with paths of preview images as value")
    problems_no += 1
  else:
    for image in app_json.get("appImages"):
      if os.path.isfile(image) == False:
        if image == "":
          print_problem("appImages cannot contain empty strings")
        else:
          print_problem(image +" does not exists")
        problems_no += 1

Finally after all the checks are performed, the developer is informed whether his or her app is ready to be published or not. The number of errors are also presented along with all other details.

if problems_no > 0:
    print_problem("Number of problems detected : " + str(problems_no))
    print_problem("App cannot be published")
    print_info("Please check https://github.com/fossasia/apps.loklak.org/blob/master/docs/tutorial.md")
  else:
    print_success("App is ready for publishing")
    print_info("Check https://github.com/fossasia/apps.loklak.org/blob/master/docs/tutorial.md to publish your app")

Here is an sample output for a faulty app

Here is an sample output for app which is ready to go.

The script is presently functional and can be used to review your apps before sending it for being published on apps.loklak.org. However there are a number of things which are presently not checked like version, getStarted.md, appUse.md, etc. These checks will be included soon.

Creating a command line tool to initiate loklak app development

There are various apps presently hosted on apps.loklak.org, which shows several interesting and awesome stuff that can be done using the loklak API. Now the question is how to create a loklak app? Well previously there were two ways to create a loklak app, you could either create an app from scratch creating all the necessary files including index.html (app entry point), app.json (required for maintaining app metadata and store listing), getStarted.md, appUse.md and other necessary files, or you could use the boilerplate app which provides you with a ready to go app template which you have to edit to create your app.

Recently there has been a new addition to the repository, a command line tool called loklakinit, built with python which will automate the process of initiating a loklak app on the go. The tool creates the entire loklak app directory structure including app.json with all the necessary details (as provided by the user) and defaults, index.html, a css and js subdirectory with a style.css and script.js in the respective folders, getStarted.md file, appUse.md file and others.md file with proper references in app.json.

All that the developer needs to do is, execute the following from the root of the apps.loklak.org repository.

bin/loklakinit.sh

This will start the process and initiate the app.

Creating loklakinit tool

Now let us delve into the code. So what actually the script does? How it works? Well the tool acts very much like the popular npm init command. At first the script creates a dictionary which stores the defaults for app.json. Next the script takes a number of inputs from the user and predicts some default values, if the user refuses to provide any input for a particular parameter then the default value for that parameter is used.

app_json = collections.OrderedDict()

app_json["@context"] = "http://schema.org"
app_json["@type"] = "SoftwareApplication"
app_json["permissions"] = "/api/search.json"
app_json["name"] = "myloklakapp"
app_json["headline"] = "My first loklak app"
app_json["alternativeHeadline"] = app_json["headline"]
app_json["applicationCategory"] = "Misc"
app_json["applicationSubCategory"] = ""
app_json["operatingSystem"] = "http://loklak.org"
app_json["promoImage"] = "promo.png"
app_json["appImages"] = ""
app_json["oneLineDescription"] = ""
app_json["getStarted"] = "getStarted.md"
app_json["appUse"] = "appUse.md"
app_json["others"] = "others.md"

author = collections.OrderedDict()
author["@type"] = "Person"
author["name"] = ""
author["email"] = ""
author["url"] = ""
author["sameAs"] = "" 

app_json["author"] = author

The first part of the script inserts some default values into the app_json dictionary. Now what is an OrderedDict? Well, it is nothing but a python dictionary in which the order in which the keys are inserted is maintained. A ordinary python dictionary does not maintain the order of the keys.

while True :

  app_name = raw_input("name (" + app_json["name"] + ") : ")
  if app_name :
    app_json["name"] = app_name

  app_context = raw_input("@context (" + app_json["@context"] + ") : ")
  if app_context :
    app_json["@context"] = app_context

  app_type = raw_input("@type (" + app_json["@type"] + ") : ")
  if app_type :
    app_json["@type"] = app_type

  app_permissions = raw_input("permissions (" + app_json["permissions"] + ") : ")
  if app_permissions :
    app_json["permissions"] = app_permissions.split(",")

  app_headline = raw_input("headline (" + app_json["headline"] + ") : ")
  if app_headline :
    app_json["headline"] = app_headline
    app_json["alternativeHeadline"] = app_headline


  app_alternative_headline = raw_input("alternative headline (" + app_json["alternativeHeadline"] + ") : ")
  if app_alternative_headline :
    app_json["alternativeHeadline"] = app_alternative_headline

  app_category = raw_input("application category (" + app_json["applicationCategory"] + ") : ")
  if app_category :
    app_json["applicationCategory"] = app_category

  app_sub_category = raw_input("sub category : ")
  if app_sub_category :
    app_json["applicationSubCategory"] = app_sub_category

  app_os = raw_input("application operating system (" + app_json["operatingSystem"] + ") : ")
  if app_os :
    app_json["operatingSystem"] = app_os

  app_promo_image = raw_input("promo image (" + app_json["promoImage"] + ") : ")
  if app_promo_image :
    app_json["promoImage"] = app_promo_image

  app_images = raw_input("app preview images (values can be separted by comma) : ")
  if app_images :
    app_json["appImages"] = app_images.split(",")

  app_description = raw_input("one line description : ")
  if app_description :
    app_json["oneLineDescription"] = app_description

  author["name"] = raw_input("author name : ")

  author_type = raw_input("author type (" + author["@type"] + ") : ")
  if author_type :
    author["@type"] = author_type

  author["email"] = raw_input("email : ")

  author["url"] = raw_input("url : ")
  author["sameAs"] = author["url"]

  author["sameAs"] = raw_input("same as : ")

  app_json["author"] = author

  print json.dumps(app_json, indent=2)

  confirm = raw_input("confirm (yes) : ")

  if confirm.lower() != "no" :
break

The next part of the script asks for various inputs from the developer which he or she can either provide or skip, if an input skipped then the default value is used. However for some parameters like author name, url etc, no default value is used, the developer must provide those values later by editing the app.json file. Once all the inputs are taken, the script outputs the json object created, if the output is alright then the developer may press enter and the process will continue, otherwise the developer may enter ‘no’ and once again the script will start taking inputs.

Once the final app_json is created, next part is creating the actual directory structure with the default files. This is done as shown below.

os.mkdir(app_json["name"])
os.chdir(app_json["name"])
os.mkdir("css")
os.mkdir("js")
index_html = open("index.html", 'w')
index_html.write("<!-- entry point of the app, on launching the appliccation" +
                  "this page will be displayed -->")
index_html.close()

app_json_file = open("app.json", 'w')
app_json_file.write(json.dumps(app_json, indent=2))
app_json_file.close()

get_started_md = open("getStarted.md", 'w')
get_started_md.write("<!-- getting started with the app -->")
get_started_md.close()

app_use_md = open("appUse.md", 'w')
app_use_md.write("<!-- use of this app -->")
app_use_md.close()

others_md = open("others.md", 'w')
others_md.write("<!-- other relevant information about this app -->")
others_md.close()

style_css = open("css/style.css", 'w')
style_css.write("/* css to style the app */")
style_css.close()

script_js = open("js/script.js", 'w')
script_js.write("// javascript logic for the app")
script_js.close()

print "done"

A new directory is created in the root of the repository, the name of the directory is same as the app name. Python’s os module is used to create the directory and subdirectories. Inside the directory, app.json file is created in which app_json dictionary is dumped as a json object.
After this the other default files are created. These include index.html, js/script.js, css/style.css
getStarted.md, appUse.md and others.md. After everything is done script exits and the developer is all set to start with his development.

Finally loklakinit.sh is used to call loklakinit.py

#!/usr/bin/env bash

python bin/loklakinit.py

This allows users to initiate the script by just executing bin/loklakinit.sh from root of the repository. The entire source code for this script can be found here (python code) and here (bash script)

Enhancing Github profile scraper service in loklak

The Github profile scraper is one of the several scrapers present in the loklak project, some of the other scrapers being Quora profile scraper, WordPress profile scraper, Instagram profile scraper, etc.Github profile scraper scrapes the profile of a given github user and provides the data in json format. The scraped data contains user_id, followers, users following the given user, starred repositories of an user, basic information like user’s name, bio and much more. It uses the popular Java Jsoup library for scraping.

The entire source code for the scraper can be found here.

Changes made

The scraper previously provided very limited information like, name of the user, description, starred repository url, followers url, following url, user_id, etc. One of the major problem was the api accepted only the profile name as a parameter and it returned the entire data, that is, the scraper scraped all the data even if the user did not ask for it. Moreover the service provided certain urls as data for example starred repository url, followers url, following url instead of providing the actual data present at those urls.

The scraper contained only one big function where all the scraping was performed. The code was not modular.

The scraper has been enhanced in the ways mentioned below.

  • The entire code has been refactored.Code scraping user specific data has been separated from code scraping organization specific data. Also separate method has been created for accessing the Github API.
  • Apart from profile parameter, the api now accepts another parameter called terms. It is a list of fields the user wants information on. In this way the scraper scrapes only that much data which is required by the user. This allows better response time and prevents unnecessary scraping.
  • The scraper now provides more information like user gists, subscriptions, events, received_events and repositories.

Code refactoring

The code has been refactored to smaller methods. ‘getDataFromApi’ method has been designed to access Github API. All the other methods which want to make request to api.github.com/ now calls ‘getDatFromApi’ method with required parameter. The method is shown below.

private static JSONArray getDataFromApi(String url) {
       URI uri = null;
       try {
           uri = new URI(url);
       } catch (URISyntaxException e1) {
           e1.printStackTrace();
       }

       JSONTokener tokener = null;
       try {
           tokener = new JSONTokener(uri.toURL().openStream());
       } catch (Exception e1) {
           e1.printStackTrace();
       }
       JSONArray arr = new JSONArray(tokener);
       return arr;
   }

 

For example if we want to make a request to the endpoint https://api.github.com/users/djmgit/followers then we can use the above method and we will get a JSONArray in return.

All the code which scrapes user related data has been moved to ‘scrapeGithubUser’ method. This method scrapes basic user information like full name of the user, bio, user name, atom feed link, and location.

String fullName = html.getElementsByAttributeValueContaining("class", "vcard-fullname").text();
githubProfile.put("full_name", fullName);
 
String userName = html.getElementsByAttributeValueContaining("class", "vcard-username").text();
githubProfile.put("user_name", userName);
 
String bio = html.getElementsByAttributeValueContaining("class", "user-profile-bio").text();
githubProfile.put("bio", bio);
 
String atomFeedLink = html.getElementsByAttributeValueContaining("type", "application/atom+xml").attr("href");
githubProfile.put("atom_feed_link", "https://github.com" + atomFeedLink);
 
String worksFor = html.getElementsByAttributeValueContaining("itemprop", "worksFor").text();
githubProfile.put("works_for", worksFor);
 
String homeLocation = html.getElementsByAttributeValueContaining("itemprop", "homeLocation").attr("title");
githubProfile.put("home_location", homeLocation);

 

Next it returns other informations like starred repositories information, followers information, following information and information about the organizations the user belongs to but before fetching such data it checks whether such data is really needed. In this way unnecessary api calls are prevented.

if (terms.contains("starred") || terms.contains("all")) {
            String starredUrl = GITHUB_API_BASE + profile + STARRED_ENDPOINT;
            JSONArray starredData = getDataFromApi(starredUrl);
            githubProfile.put("starred_data", starredData);
 
            int starred = Integer.parseInt(html.getElementsByAttributeValue("class", "Counter").get(1).text());
            githubProfile.put("starred", starred);
        }
 
        if (terms.contains("follows") || terms.contains("all")) {
            String followersUrl = GITHUB_API_BASE + profile + FOLLOWERS_ENDPOINT;
            JSONArray followersData = getDataFromApi(followersUrl);
            githubProfile.put("followers_data", followersData);
 
            int followers = Integer.parseInt(html.getElementsByAttributeValue("class", "Counter").get(2).text());
            githubProfile.put("followers", followers);
        }
 
        if (terms.contains("following") || terms.contains("all")) {
            String followingUrl = GITHUB_API_BASE + profile + FOLLOWING_ENDPOINT;
            JSONArray followingData = getDataFromApi(followingUrl);
            githubProfile.put("following_data", followingData);
 
            int following = Integer.parseInt(html.getElementsByAttributeValue("class", "Counter").get(3).text());
            githubProfile.put("following", following);
        }
 
        if (terms.contains("organizations") || terms.contains("all")) {
            JSONArray organizations = new JSONArray();
            Elements orgs = html.getElementsByAttributeValue("itemprop", "follows");
            for (Element e : orgs) {
                JSONObject obj = new JSONObject();
 
                String label = e.attr("aria-label");
                obj.put("label", label);
 
                String link = e.attr("href");
                obj.put("link", "https://github.com" + link);
 
                String imgLink = e.children().attr("src");
                obj.put("img_link", imgLink);
 
                String imgAlt = e.children().attr("alt");
                obj.put("img_Alt", imgAlt);
 
                organizations.put(obj);
            }
            githubProfile.put("organizations", organizations);
        }

 

Similarly ‘scrapeGithubOrg’ is used to scrape information related to Github organization.

API usage

The API can be used in the ways shown below.

  • api/githubprofilescraper.json?profile=<profile_name> : This is equivalent to api/githubprofilescraper.json?profile=<profile_name>&terms=all :This will return all the data the scraper is currently capable of collecting.
  • api/githubprofilescraper.json?profile=<profile_name>&terms=<list_containing_required_fields> : This will return the basic information about a profile and data on selective fields as mentioned by the user.

For example – api/githubprofilescraper.json?profile=djmgit&terms=followers,following,organizations

The above request will return data about followers, following, and organizations apart from basic profile information. Thus the scraper will scrape only those data that is required by the user.

Future scope

‘scrapeGithubUser’ is still lengthy and it can be further broken down into smaller methods. The part of the method which scrapes basic user information like name, bio, etc can be moved to a separate method. This will further increase code readability and maintainability.

Now we have a much more functional and enhanced Github profile scraper which provides more data than the previous one.

Displaying error notifications in whatsTrending? app

The issue I am solving in the whatsTrending app is to display error notifications when the date fields and the count field are not validated and when a user enters invalid data. Specifically we want to display error notifications for junk values and dates with formats other than YYYY-MM-DD and any other invalid data in the whatsTrending app’s filter option.

The whatsTrending app is a web app that shows the top trending hashtags of twitter messages in a given date range using tweets collected by the loklak search engine. Users can also limit the number of top hash tags they want to see and use filters with start and end dates.

App to know trending hashtags on twitter

What is the problem? The date fields and the count field are not validated which means junk values and date with formats other than YYYY-MM-DD do not show any error.

So how can the problem be solved? Well the format (pattern) of the date can be verified by regular expression. A regular expression describes a pattern in a given text.So the format checking problem can be described as finding the pattern YYYY-MM-DD in the input date where Y, M and D are numbers.The Regex should specify that the pattern should be present at the beginning of the text.

More detailed information about regex can be found here.

The regex for this pattern is :

/^\d{4}-\d{2}-\d{2}$/

The pattern says there should be 4 numbers followed by ‘-’ then two numbers then again ‘-’ and then again two numbers.

This can be implemented the following way :

$scope.isValidDate = function(dateString) {
        var regEx = /^\d{4}-\d{2}-\d{2}$/;
        if (dateString.match(regEx) === null) {
            return false;
        }

        dateComp = dateString.split('-');
        var i=0;
        for (i=0; i<dateComp.length; i++) { dateComp[i] = parseInt(dateComp[i]); } if (dateComp.length > 3) {
            return false;
        }

        if (dateComp[1] > 12 || dateComp[1] <= 0) { return false; } if (dateComp[2] > 31 || dateComp[2] <= 0) { return false; } if (((dateComp[1] === 4) || (dateComp[1] === 6) || (dateComp[1] === 9) || (dateComp[1] === 11)) && (dateComp[2] > 30)) {
            return false;
        }

        if (dateComp[1] ===2) {
            if (((dateComp[0] % 4 === 0) && (dateComp[0] % 100 !== 0)) || (dateComp[0] % 400 === 0)) {
                if (dateComp[2] > 29) {
                    return false;
                }
            } else {
                if (dateComp[2] > 28) {
                    return false;
                }
            }
        }

        return true;
    }

So the first part of the code checks for the above mentioned pattern in the input. If not found it returns false.If found then we split the entire date into a list containing year, month and day and the remaining part if any is removed.Each component is converted to integer.Then further validation is done on the month and day as can be seen from the code above.The range of the month and date is checked.Also leap year checking is done.

In the same way the count field is also validated. The regex for this field is much simpler. We just need to check that the input consists only of numbers and nothing else.
So the regex for this is :

 /^[0-9]+$/

This means repetition of digits in the range 0-9.We search for this pattern in the text. If found we return true else false.The function for this is as follows:

$scope.isNumber = function(numString) {
        var regEx = /^[0-9]+$/;
        return String(numString).match(regEx) != null;
    }

Next we need to call these function and see if their is any error. If there is an error we need to display it.This can be done using a modal. Bootstrap has got an inbuilt modal which can be invoked using javascript.

Showing error using modal

So at first we need to define the modal and its content (empty if necessary as in this case)using HTML.The HTML code for this can be found here.

A small yet nice tutorial on Bootstrap modal can be found here
Next we need to set the content of the modal and invoke it from our JS file on encountering an error.

$scope.displayErrorModal = function(val, type) {
        if (type === 0) {
            if (!$scope.isValidDate(val)) {
                $scope.loading = false;
                $('.modal-body').html('Please enter valid date in YYYY-MM-DD format'); 
                $('#myModal').modal('show');                 
                return false; 
            } 
         } else { 
             if (!$scope.isNumber(val)) { 
                 $scope.loading = false; 
                 $('.modal-body').html('Please enter a valid number'); 
                 $('#myModal').modal('show');                    
                 return false; 
             } 
         } 
         return true; 
}

The above function accepts a parameter val and another parameter type.The parameter type tells what validation needs to be performed, date validation or number validation and calls previous two methods accordingly and passes val which is the value to validated.If any of the validation fails then it sets the content of the modal using : $(‘.modal-body’).html(“your content”) and then invokes it using : $(‘#modalID’).modal(‘show’). This displays a nice modal on the page and the user is notified about the error.

So this is it for this post.Thanks for reading it.My next post will be on fixing the design of the boilerplate app.