LoklakApps – blog.fossasia.org

Adding additional information to store listing page of Loklak apps site

Post author:Deepjyoti Mondal
Post published:September 3, 2017
Post category:FOSSASIA loklak
Post comments:0 Comments

Loklak apps site has now got a completely functional store listing page where users can find all relevant information about the app which they want to view. The page has a left side bar which shows various categories to switch between, a right sidebar for suggesting similar kind of apps to users and a middle section to provide users with various important informations about the app like getting started, use of app, promo images, preview images, test link and various other details. In this blog I will be describing how the bottom section of the middle column has been created (related issue: #209). The bottom section The bottom section provides various informations like updated, version, app source, developer information, contributors, technology stack, license. All these informations has to be dynamically loaded for each selected app. As I had previously mentioned here, no HTML content can be hard coded in the store listing page. So how do we show the above mentioned informations for the different apps? Well, for this we will once again use the app.json of the corresponding app like we had done for the middle section here. At first, for a given app we need to define some extra fields in the app.json file as shown below. "appSource": "https://github.com/fossasia/apps.loklak.org/tree/master/MultiLinePlotter", "contributors": [{"name": "djmgit", "url": "http://djmgit.github.io/"}], "techStack": ["HTML", "CSS", "AngularJs", "Morris.js", "Bootstrap", "Loklak API"], "license": {"name": "LGPL 2.1", "url": "https://www.gnu.org/licenses/old-licenses/lgpl-2.1"}, "version": "1.0", "updated": "June 10,2017", The above code snippet shows the new fields included in app.json. The fields are as described below. appSource: Stores link to the source code of the app. Contributors: Stores a list containing objects. Each object stores name of the contributor and an url corresponding to that contributor. techStack: A list containing names of the technologies used. License: Name and link of the license. Version: The current version of the app. Updated: Date on which the app was last updated. These fields provide the source for the informations present in the bottom section of the app. Now we need to render these information on the store listing page. Let us take an example. Let us see how version is rendered. <div ng-if="appData.version !== undefined && appData.version !== ''" class="col-md-4 add-info"> <div class="info-type"> <h5 class="info-header"> <strong>Version</strong> </h5> </div> <div class="info-body"> {{appData.version}} </div> </div> We first check if version field is defined and version is not empty. Then we print a header (Version in this case) and then we print the value. This is how updated, appSource and license are also displayed. What about technology stack and contributors? Technology stack is basically an list and it may contain quite a number of strings(technology names). If we display all the values at once the bottom section will get crowded and it may degrade the UI of the page.To avoid this a popup dialog has been used. When user clicks on the technology stack label, a popup dialogue appears which shows the various technologies used in the app. <div class="info-body"> <div class="dropdown"> <div class="dropdown-toggle" type="button" data-toggle="dropdown"> View technology stack </div>…

Improving Loklak apps site

Post author:Deepjyoti Mondal
Post published:September 1, 2017
Post category:FOSSASIA loklak Open Event
Post comments:0 Comments

In this blog I will be describing some of the recent improvements made to the Loklak apps site. A new utility script has been added to automatically update the loklak app wall after a new app has been made. Invalid app query in app details page has been handled gracefully. A proper message is shown when a user enters an invalid app name in the url of the details page. Tests has been added for details page. Developing updatewall script This is a small utility script to update Loklak wall in order to expose a newly created app or update an existing app. Before moving into the working of this script let us discuss how Loklak apps site tracks all the apps and their details. In the root of the project there is a file names apps.json. This file contains an aggregation of all the app.json files present in the individual apps. Now when the site is loaded, index.html loads the Javascript code present in app_list.js. This app_list.js file makes an ajax call to root apps.json files, loads all the app details in a list and attaches this list to the AngularJS scope variable. After this the app wall consisting of various app details is rendered using html. So whenever a new app is created, in order to expose the app on the wall, the developer needs to copy the contents of the application’s app.json and paste it in the root apps.json file. This is quite tedious on the part of the developer as for making a new app he will first have to know how the site works which is not all directly related to his development work. Next, whenever he updates the app.json of his app, he needs to again update apps.json file with the new data. This newly added script (updatewall) automates this entire process. After creating a new app all that the developer needs to do is run this script from within his app directory and the app wall will be updated automatically. Now, let us move into the working of this script. The basic workflow of the updatewall script can be described as follows. The script loads the json data present in the app.json file of the app under consideration. Next it loads the json data present in the root apps.json file. if __name__ == '__main__': #open file containg json object json_list_file = open(PATH_TO_ROOT_JSON, 'r') #load json object json_list = json.load(json_list_file, object_pairs_hook=OrderedDict) json_list_file.close() app_json_file = open(PATH_TO_APP_JSON, 'r') app_json = json.load(app_json_file, object_pairs_hook=OrderedDict) app_json_file.close() #method to update Loklak app wall expose_app(json_list, app_json) When we are loading the json data we are using object_pairs_hook in order to load the data into an OrderedDict rather than a normal python dictionary. We are doing this so that the order of the dictionary items are maintained. Once the data is loaded we invoke the expose method. def expose_app(json_list, app_json): #if app is already present in list then fetch that app app = getAppIfPesent(json_list, app_json) #if app is not present then add…

Using Protractor for UI Tests in Angular JS for Loklak Apps Site

Post author:Deepjyoti Mondal
Post published:September 1, 2017
Post category:API FOSSASIA GSoC loklak
Post comments:0 Comments

Loklak apps site’s home page and app details page have sections where data is dynamically loaded from external javascript and json files. Data is fetched from json files using angular js, processed and then rendered to the corresponding views by controllers. Any erroneous modification to the controller functions might cause discrepancies in the frontend. Since Loklak apps is a frontend project, any bug in the home page or details page will lead to poor UI/UX. How do we deal with this? One way is to write unit tests for the various controller functions and check their behaviours. Now how do we test the behaviours of the site. Most of the controller functions render something on the view. One thing we can do is simulate the various browser actions and test site against known, accepted behaviours with Protractor. What is Protractor Protractor is end to end test framework for Angular and AngularJS apps. It runs tests against our app running in browser as if a real user is interacting with our browser. It uses browser specific drivers to interact with our web application as any user would. Using Protractor to write tests for Loklak apps site First we need to install Protractor and its dependencies. Let us begin by creating an empty json file in the project directory using the following command. echo {} > package.json Next we will have to install Protractor. The above command installs protractor and webdriver-manager. After this we need to get the necessary binaries to set up our selenium server. This can be done using the following. ./node_modules/protractor/bin/webdriver-manager update ./node_modules/protractor/bin/webdriver-manager start Let us tidy up things a bit. We will include these commands in package.json file under scripts section so that we can shorten our commands. Given below is the current state of package.json { "scripts": { "start": "./node_modules/http-server/bin/http-server", "update-driver": "./node_modules/protractor/bin/webdriver-manager update", "start-driver": "./node_modules/protractor/bin/webdriver-manager start", "test": "./node_modules/protractor/bin/protractor conf.js" }, "dependencies": { "http-server": "^0.10.0", "protractor": "^5.1.2" } } The package.json file currently holds our dependencies and scripts. It contains command for starting development server, updating webdriver and starting webdriver (mentioned just before this) and command to run test. Next we need to include a configuration file for protractor. The configuration file should contain the test framework to be used, the address at which selenium is running and path to specs file. // conf.js exports.config = { framework: "jasmine", seleniumAddress: "http://localhost:4444/wd/hub", specs: ["tests/home-spec.js"] }; We have set the framework as jasmine and selenium address as http://localhost:4444/wd/hub. Next we need to define our actual file. But before writing tests we need to find out what are the things that we need to test. We will mostly be testing dynamic content loaded by Javascript files. Let us define a spec. A spec is a collection of tests. We will start by testing the category name. Initially when the page loads it should be equal to All apps. Next we test the top right hand side menu which is loaded by javascript using topmenu.json file. it("should have a category name", function()…

Adding download feature to LoklakWordCloud app on Loklak apps site

Post author:Deepjyoti Mondal
Post published:August 21, 2017
Post category:FOSSASIA loklak Open Event
Post comments:0 Comments

One of the most important and useful feature that has recently been added to LoklakWordCloud app is enabling the user to download the generated word cloud as a png/jpeg image. This feature will allow the user to actually use this app as a tool to generate a word cloud using twitter data and save it on their disks for future use. All that the user needs to do is generate the word cloud, choose an image type (png or jpeg) and click on export as image, a preview of the image to be downloaded will be displayed. Just hit enter and the word cloud will be saved on your disk. Thus users will not have to use any alternative process like taking a screenshot of the word cloud generated, etc. Presently the complete app is hosted on Loklak apps site. How does it work? What we are doing is, we are exporting a part of the page (a div) as image and saving it. Apparently it might seem that we are taking a screenshot of a particular portion of a page and generating a download link. But actually it is not like that. The word cloud that is being generated by this app via Jqcloud is actually a collection of HTML nodes. Each node contains a word (part of the cloud) as a text content with some CSS styles to specify the size and color of that word. As user clicks on export to image option, the app traverses the div containing the cloud. It collects information about all the HTML nodes present under that div and creates a canvas representation of the entire div. So rather than taking a screenshot of the div, the app recreates the entire div and presents it to us. This entire process is accomplished by a lightweight JS library called html2canvas. Let us have a look into the code that implements the download feature. At first we need to create the UI for the export and download option. User should be able to choose between png and jpeg before exporting to image. For this we have provided a dropdown containing the two options. <div class="dropdown type" ng-if="download"> <div class="dropdown-toggle select-type" data-toggle="dropdown"> {{imageType}} <span class="caret"></span></div> <ul class="dropdown-menu"> <li ng-click="changeType('png', 'png')"><a href="">png</a></li> <li ng-click="changeType('jpeg', 'jpg')"><a href="">jpeg</a></li> </ul> </div> <a class="export" ng-click="export()" ng-if="download">Export as image</a> In the above code snippet, firstly we create a dropdown menu with two list items, png and jpeg. With each each list item we attach a ng-click event which calls changeType function and passes two parameters, image type and extension. The changeType function simply updates the current image type and extension with the selected ones. $scope.changeType = function(type, ext) { $scope.imageType = type; $scope.imageExt = ext; } The ‘export as image’ on clicking calls the export function. The export function uses html2canvas library’s interface to generate the canvas representation of the word cloud and also generates the download link and attaches it to the modal’s save button (described below). After everything is…

Enhancing LoklakWordCloud app present on Loklak apps site

Post author:Deepjyoti Mondal
Post published:August 14, 2017
Post category:API FOSSASIA GSoC loklak
Post comments:0 Comments

LoklakWordCloud app is presently hosted on loklak apps site. Before moving into the content of this blog, let us get a brief overview of the app. What does the app do? The app generates a word cloud using twitter data returned by loklak based on the query word provided by the user. The user enters a word in the input field and presses the search button. After that a word cloud is created using the content (text body, hashtags and mentioned) of the various tweets which contains the user provided query word. In my previous post I wrote about creating the basic functional app. In this post I will be describing the next steps that have been implemented in the app. Making the word cloud clickable This is one of the most important and interesting features added to the app. The words in the cloud are now clickable.Whenever an user clicks on a word present in the cloud, the cloud is replaced by the word cloud of that selected word. How do we achieve this behaviour? Well, for this we use Jqcloud’s handler feature. While creating the list of objects for each word and its frequency, we also specify a handler corresponding to each of the word. The handler is supposed to handle a click event. Whenever a click event occurs, we set the value of $scope.tweet to the selected word and invoke the search function, which calls the loklak API and regenerates the word cloud. for (var word in $scope.wordFreq) { $scope.wordCloudData.push({ text: word, weight: $scope.wordFreq[word], handlers: { click: function(e) { $scope.tweet = e.target.textContent; $scope.search(); } } }); } As it can be seen in the above snippet, handlers is simply an JavaScript object, which takes a function for the click event. In the function we pass the word selected as value of the tweet variable and call search method. Adding filters to the app Previously the app generated word cloud using the entire tweet content, that is, hashtags, mentions and tweet body. Thus the app was not flexible. User was not able to decide on which field he wants his word cloud to be generated. User might want to generate his word cloud using only the hashtags or the mentions or simply the tweet body. In order to make this possible, filters have been introduced. Now we have filters for hashtags, mentions, tweet body and date. <div class="col-md-6 tweet-filters"> <strong>Filters</strong> <hr> <div class="filters"> <label class="checkbox-inline"><input type="checkbox" value="" ng-model="hashtags">Hashtags</label> <label class="checkbox-inline"><input type="checkbox" value="" ng-model="mentions">Mentions</label> <label class="checkbox-inline"><input type="checkbox" value="" ng-model="tweetbody">Tweet body</label> </div> <div class="filter-all"> <span class="select-all" ng-click="selectAll()"> Select all </span> </div> </div> We have used checkboxes for the individual filters and have kept an option to select all the filters at once. Next we require to hook this HTML to AngularJS code to make the filters functional. if ($scope.hashtags) { tweet.hashtags.forEach(function (hashtag) { $scope.filteredWords.push("#" + hashtag); }); } if ($scope.mentions) { tweet.mentions.forEach(function (mention) { $scope.filteredWords.push("@" + mention); }); } In the above snippet, before adding the hashtags to the list…

Developing LoklakWordCloud app for Loklak apps site

Post author:Deepjyoti Mondal
Post published:August 12, 2017
Post category:API FOSSASIA GSoC loklak
Post comments:0 Comments

LoklakWordCloud app is an app to visualise data returned by loklak in form of a word cloud. The app is presently hosted on Loklak apps site. Word clouds provide a very simple, easy, yet interesting and effective way to analyse and visualise data. This app will allow users to create word cloud out of twitter data via Loklak API. Presently the app is at its very early stage of development and more work is left to be done. The app consists of a input field where user can enter a query word and on pressing search button a word cloud will be generated using the words related to the query word entered. Loklak API is used to fetch all the tweets which contain the query word entered by the user. These tweets are processed to generate the word cloud. Related issue: https://github.com/fossasia/apps.loklak.org/pull/279 Live app: http://apps.loklak.org/LoklakWordCloud/ Developing the app The main challenge in developing this app is implementing its prime feature, that is, generating the word cloud. How do we get a dynamic word cloud which can be easily generated by the user based on the word he has entered? Well, here comes in Jqcloud. An awesome lightweight Jquery plugin for generating word clouds. All we need to do is provide list of words along with their weights. Let us see step by step how this app (first version) works. First we require all the tweets which contain the entered word. For this we use Loklak search service. Once we get all the tweets, then we can parse the tweet body to create a list of words along with their frequency. var url = "http://35.184.151.104/api/search.json?callback=JSON_CALLBACK&count=100&q=" + query; $http.jsonp(url) .then(function (response) { $scope.createWordCloudData(response.data.statuses); $scope.tweet = null; }); Once we have all the tweets, we need to extract the tweet texts and create a list of valid words. What are valid words? Well words like ‘the’, ‘is’, ‘a’, ‘for’, ‘of’, ‘then’, does not provide us with any important information and will not help us in doing any kind of analysis. So there is no use of including them in our word cloud. Such words are called stop words and we need to get rid of them. For this we are using a list of commonly used stop words. Such lists can be very easily found over the internet. Here is the list which we are using. Once we are able to extract the text from the tweets, we need to filter stop words and insert the valid words into a list. tweet = data[i]; tweetWords = tweet.text.replace(", ", " ").split(" "); for (var j = 0; j < tweetWords.length; j++) { word = tweetWords[j]; word = word.trim(); if (word.startsWith("'") || word.startsWith('"') || word.startsWith("(") || word.startsWith("[")) { word = word.substring(1); } if (word.endsWith("'") || word.endsWith('"') || word.endsWith(")") || word.endsWith("]") || word.endsWith("?") || word.endsWith(".")) { word = word.substring(0, word.length - 1); } if (stopwords.indexOf(word.toLowerCase()) !== -1) { continue; } if (word.startsWith("#") || word.startsWith("@")) { continue; } if (word.startsWith("http") || word.startsWith("https")) { continue; } $scope.filteredWords.push(word); }…

Visualising Tweet Statistics in MultiLinePlotter App for Loklak Apps

Post author:Deepjyoti Mondal
Post published:July 24, 2017
Post category:FOSSASIA GSoC loklak
Post comments:0 Comments

MultiLinePlotter app is now a part of Loklak apps site. This app can be used to compare aggregations of tweets containing a particular query word and visualise the data for better comparison. Recently there has been a new addition to the app. A feature for showing tweet statistics like the maximum number of tweets (along with date) containing the given query word and the average number of tweets over a period of time. Such statistics is visualised for all the query words for better comparison. Related issue: https://github.com/fossasia/apps.loklak.org/issues/236 Obtaining Maximum number of tweets and average number of tweets Before visualising the statistics we need to obtain them. For this we simply need to process the aggregations returned by the Loklak API. Let us start with maximum number of tweets containing the given keyword. What we actually require is what is the maximum number of tweets that were posted and contained the user given keyword and on which date the number was maximum. For this we can use a function which will iterate over all the aggregations and return the largest along with date. $scope.getMaxTweetNumAndDate = function(aggregations) { var maxTweetDate = null; var maxTweetNum = -1; for (date in aggregations) { if (aggregations[date] > maxTweetNum) { maxTweetNum = aggregations[date]; maxTweetDate = date; } } return {date: maxTweetDate, count: maxTweetNum}; } The above function maintains two variables, one for maximum number of tweets and another for date. We iterate over all the aggregations and for each aggregation we compare the number of tweets with the value stored in the maxTweetNum variable. If the current value is more than the value stored in that variable then we simply update it and keep track of the date. Finally we return an object containing both maximum number of tweets and the corresponding date.Next we need to obtain average number of tweets. We can do this by summing up all the tweet frequencies and dividing it by number of aggregations. $scope.getAverageTweetNum = function(aggregations) { var avg = 0; var sum = 0; for (date in aggregations) { sum += aggregations[date]; } return parseInt(sum / Object.keys(aggregations).length); } The above function calculates average number of tweets in the way mentioned before the snippet. Next for every tweet we need to store these values in a format which can easily be understood by morris.js. For this we use a list and store the statistics values for individual query words as objects and later pass it as a parameter to morris. var maxStat = $scope.getMaxTweetNumAndDate(aggregations); var avg = $scope.getAverageTweetNum(aggregations); $scope.tweetStat.push({ tweet: $scope.tweet, maxTweetCount: maxStat.count, maxTweetOn: maxStat.date, averageTweetsPerDay: avg, aggregationsLength: Object.keys(aggregations).length }); We maintain a list called tweetStat and the list contains objects which stores the query word and the corresponding values. Apart from plotting these statistics, the app also displays the statistics when user clicks on an individual treat present in the search record section. For this we filter tweetStat list mentioned above and get the required object corresponding to the query word the user selected bind it to angular…

Developing MultiLinePlotter App for Loklak

Post author:Deepjyoti Mondal
Post published:July 24, 2017
Post category:FOSSASIA GSoC loklak Tutorial
Post comments:0 Comments

MultiLinePlotter is a web application which uses Loklak API under the hood to plot multiple tweet aggregations related to different user provided query words in the same graph. The user can give several query words and multiple lines for different queries will be plotted in the same graph. In this way, users will be able to compare tweet distribution for various keywords and visualise the comparison. All the searched queries are shown under the search record section. Clicking on a record causes a dialogue box to pop up where the individual tweets related to the query word is displayed. Users can also remove a series from the plot dynamically by just pressing the Remove button beside the query word in record section. The app is presently hosted on Loklak apps site. Related issue - https://github.com/fossasia/apps.loklak.org/issues/225 Getting started with the app Let us delve into the working of the app. The app uses Loklak aggregation API to get the data. A call to the API looks something like this: http://api.loklak.org/api/search.json?q=fossasia&source=cache&count=0&fields=created_at A small snippet of the aggregation returned by the above API request is shown below. "aggregations": {"created_at": { "2017-07-03": 3, "2017-07-04": 9, "2017-07-05": 12, "2017-07-06": 8, }} The API provides a nice date v/s number of tweets aggregation. Now we need to plot this. For plotting Morris.js has been used. It is a lightweight javascript library for visualising data. One of the main features of this app is addition and removal of multiple series from the graph dynamically. How do we achieve that? Well, this can be achieved by manipulating the morris.js data list whenever a new query is made. Let us understand this in steps. At first, the data is fetched using angular HTTP service. $http.jsonp('http://api.loklak.org/api/search.json?callback=JSON_CALLBACK', {params: {q: $scope.tweet, source: 'cache', count: '0', fields: 'created_at'}}) .then(function (response) { $scope.getData(response.data.aggregations.created_at); $scope.plotData(); $scope.queryRecords.push($scope.tweet); }); Once we get the data, getData function is called and the aggregation data is passed to it. The query word is also stored in queryRecords list for future use. In order to plot a line graph morris.js requires a data object which will contain the required values for a series. Given below is an example of such a data object. data: [ { x: '2006', a: 100, b: 90 }, { x: '2007', a: 75, b: 65 }, { x: '2008', a: 50, b: 40 }, { x: '2009', a: 75, b: 65 }, ], For every ‘x’, ‘a’ and ‘b’ will be plotted. Thus two lines will be drawn. Our app will also maintain a data list like the one shown above, however, in our case, the data objects will have a variable number of keys. One key will determine the ‘x’ value and other keys will determine the ordinates (number of tweets). All the data objects present in the data list needs to be updated whenever a new search is done. The getData function does this for us. var value = $scope.tweet; for (date in aggregations) { var present = false; for (var i = 0;…