Visualising Tweet Statistics in MultiLinePlotter App for Loklak Apps

MultiLinePlotter app is now a part of Loklak apps site. This app can be used to compare aggregations of tweets containing a particular query word and visualise the data for better comparison. Recently there has been a new addition to the app. A feature for showing tweet statistics like the maximum number of tweets (along with date) containing the given query word and the average number of tweets over a period of time. Such statistics is visualised for all the query words for better comparison.

Related issue: https://github.com/fossasia/apps.loklak.org/issues/236

Obtaining Maximum number of tweets and average number of tweets

Before visualising the statistics we need to obtain them. For this we simply need to process the aggregations returned by the Loklak API. Let us start with maximum number of tweets containing the given keyword. What we actually require is what is the maximum number of tweets that were posted and contained the user given keyword and on which date the number was maximum. For this we can use a function which will iterate over all the aggregations and return the largest along with date.

$scope.getMaxTweetNumAndDate = function(aggregations) {
        var maxTweetDate = null;
        var maxTweetNum = -1;

        for (date in aggregations) {
            if (aggregations[date] > maxTweetNum) {
                maxTweetNum = aggregations[date];
                maxTweetDate = date;
            }
        }

        return {date: maxTweetDate, count: maxTweetNum};
    }

The above function maintains two variables, one for maximum number of tweets and another for date. We iterate over all the aggregations and for each aggregation we compare the number of tweets with the value stored in the maxTweetNum variable. If the current value is more than the value stored in that variable then we simply update it and keep track of the date. Finally we return an object containing both maximum number of tweets and the corresponding date.Next we need to obtain average number of tweets. We can do this by summing up all the tweet frequencies and dividing it by number of aggregations.

$scope.getAverageTweetNum = function(aggregations) {
        var avg = 0;
        var sum = 0;

        for (date in aggregations) {
            sum += aggregations[date];
        }

        return parseInt(sum / Object.keys(aggregations).length);
    }

The above function calculates average number of tweets in the way mentioned before the snippet.

Next for every tweet we need to store these values in a format which can easily be understood by morris.js. For this we use a list and store the statistics values for individual query words as objects and later pass it as a parameter to morris.

var maxStat = $scope.getMaxTweetNumAndDate(aggregations);
        var avg = $scope.getAverageTweetNum(aggregations);

        $scope.tweetStat.push({
            tweet: $scope.tweet,
            maxTweetCount: maxStat.count,
            maxTweetOn: maxStat.date,
            averageTweetsPerDay: avg,
            aggregationsLength: Object.keys(aggregations).length
        });

We maintain a list called tweetStat and the list contains objects which stores the query word and the corresponding values.

Apart from plotting these statistics, the app also displays the statistics when user clicks on an individual treat present in the search record section. For this we filter tweetStat list mentioned above and get the required object corresponding to the query word the user selected bind it to angular scope. Next we display it using HTML.

<div class="tweet-stat max-tweet">
                  <div class="stat-label"> <h4>Maximum number of tweets containing '{{modalHeading}}' :</h4></div>
                  <div class="stat-value"> <strong>{{selectedTweetStat.maxTweetCount}}</strong> tweets on
                    <strong>{{selectedTweetStat.maxTweetOn}}</strong>
                  </div>
                </div>

Finally we need to plot the statistics. For this we use a function called plotStatGraph dedicated only for plotting statistics graph. We pass the tweetStat list as a parameter to morris and configure all the other parameters.

$scope.plotStatGraph = function() {
        $scope.plotStat = new Morris.Bar({
            element: 'graph',
            data: $scope.tweetStat,
            xkey: 'tweet',
            ykeys: ['maxTweetCount', 'averageTweetsPerDay'],
            labels: ['Maximum no. of tweets : ', 'Average no. of tweets/day'],
            parseTime: false,
            hideHover: 'auto',
            resize: true,
            stacked: true,
            barSizeRatio: 0.40
        });
        $scope.graphLoading = false;
    }

But now we have two graphs. One for showing variations in aggregation and the other for showing statistics. How do we manage them? Somehow we need to show them in the same page as this is a single page app. Also we need to avoid vertical scrolling as it would degrade both UI and UX. So we need to implement a switching mechanism. The user should be able to switch between the two graph views as per their wish. How to achieve that? Well, for this we maintain a global variable which will keep track of the current plot type. If the current graph type is aggregations then we call the function to plot aggregations otherwise we call the above mentioned function to plot statistics.

$scope.plotData = function() {
        $(".plot-data").html("");
        if ($scope.currentGraphType === "aggregations") {
            $scope.plotAggregationGraph();
        } else {
            $scope.plotStatGraph();
        }
    }

Lastly we integrate this state variable (currentGraphType) with the UI so that users can easily toggle between graph views with just a click.

<div class="switch" ng-click="toggle()">
                <span ng-if="queryRecords.length !== 0" class="glyphicon glyphicon-stats"></span>
              </div>

Important resources

Developing MultiLinePlotter App for Loklak

MultiLinePlotter is a web application which uses Loklak API under the hood to plot multiple tweet aggregations related to different user provided query words in the same graph. The user can give several query words and multiple lines for different queries will be plotted in the same graph. In this way, users will be able to compare tweet distribution for various keywords and visualise the comparison. All the searched queries are shown under the search record section. Clicking on a record causes a dialogue box to pop up where the individual tweets related to the query word is displayed. Users can also remove a series from the plot dynamically by just pressing the Remove button beside the query word in record section. The app is presently hosted on Loklak apps site.

Related issue – https://github.com/fossasia/apps.loklak.org/issues/225

Getting started with the app

Let us delve into the working of the app. The app uses Loklak aggregation API to get the data.

A call to the API looks something like this:

http://api.loklak.org/api/search.json?q=fossasia&source=cache&count=0&fields=created_at

A small snippet of the aggregation returned by the above API request is shown below.

"aggregations": {"created_at": {
    "2017-07-03": 3,
    "2017-07-04": 9,
    "2017-07-05": 12,
    "2017-07-06": 8,
}}

The API provides a nice date v/s number of tweets aggregation. Now we need to plot this. For plotting Morris.js has been used. It is a lightweight javascript library for visualising data.

One of the main features of this app is addition and removal of multiple series from the graph dynamically. How do we achieve that? Well, this can be achieved by manipulating the morris.js data list whenever a new query is made. Let us understand this in steps.

At first, the data is fetched using angular HTTP service.

$http.jsonp('http://api.loklak.org/api/search.json?callback=JSON_CALLBACK',
            {params: {q: $scope.tweet, source: 'cache', count: '0', fields: 'created_at'}})
                .then(function (response) {
                    $scope.getData(response.data.aggregations.created_at);
                    $scope.plotData();
                    $scope.queryRecords.push($scope.tweet);
                });

Once we get the data, getData function is called and the aggregation data is passed to it. The query word is also stored in queryRecords list for future use.

In order to plot a line graph morris.js requires a data object which will contain the required values for a series. Given below is an example of such a data object.

data: [
    { x: '2006', a: 100, b: 90 },
    { x: '2007', a: 75,  b: 65 },
    { x: '2008', a: 50,  b: 40 },
    { x: '2009', a: 75,  b: 65 },
],

For every ‘x’, ‘a’ and ‘b’ will be plotted. Thus two lines will be drawn. Our app will also maintain a data list like the one shown above, however, in our case, the data objects will have a variable number of keys. One key will determine the ‘x’ value and other keys will determine the ordinates (number of tweets).

All the data objects present in the data list needs to be updated whenever a new search is done.

The getData function does this for us.

var value = $scope.tweet;
        for (date in aggregations) {
            var present = false;
            for (var i = 0; i < $scope.data.length; i++) {
                var item = $scope.data[i];
                if (item['day'] === date) {
                    item[[value]] = aggregations[date];
                    $scope.data[i] = item
                    present = true;
                    break;
                }
            }
            if (!present) {
                $scope.data.push({day: date, [value]: aggregations[date]});
            }
        }


The for loop in the above code snippet updates the global data list used by morris.js. It simply iterates over the dates in the aggregation, extracts the object corresponding to a particular date, adds the new query word as a key and, the number of tweets on that date as the value.If a date is not already present in the list, then it inserts a new object corresponding to the date and query word. Once our data list is updated, we are ready to redraw the graph with the updated data. This is done using plotData function. The plotData function simply checks the user selected graph type. If the selected type is aggregations then it calls plotAggregationGraph() to redraw the aggregations plot.

$scope.remove = function(record) {
        $scope.queryRecords = $scope.queryRecords.filter(function(e) {
            return e !== record });

        $scope.data.forEach(function(item) {
            delete item[record];
        });

        $scope.data = $scope.data.filter(function(item) {
            return Object.keys(item).length !== 1;
        });

        $scope.ykeys = $scope.ykeys.filter(function(item) {
            return item !== record;
        });

        $scope.labels = $scope.labels.filter(function(item) {
            return item !== record;
        });

        $scope.plotData();
}

The above function simply scans the data list, filters the objects which contains selected record as a key and removes them using filter method of javascript arrays. It also removes the corresponding labels and entries from labels and ykeys arrays. Finally, it once again calls plotData function to redraw the plot.

Given below is a sample plot generated by this app with the query words – google, android, microsoft, samsung.

 

Conclusion

This blog post explained how multiple series are plotted dynamically in the MultiLinePlotter app. Apart from aggregations plot it also plots tweet statistics like maximum tweets and average tweets containing a query word and visualises them using stacked bar chart. I will be discussing about them in my subsequent blogs.

Important resources

Implementing a suggestion box in Susper Angular JS Front-end

In Susper, we have implemented a suggestion box for our input box. This was done using Yacy suggest API. This is how the box looks:

In this blog, let us see how to implement such a box using html, css, twitter-bootstrap and typescript. You can also check the code at the Susper repository.

The html code is simple and straightforward:

 

class=“suggestion-box” *ngIf=“results”>

</div>

A few points to notice :

  • *ngIf=”results” ensures that the box is displayed only when it has suggestions to display and not otherwise
  • The [routerLink] and [queryParams] attributes together link every result to the search page, with the correct query.

This is the css code :

a {
text-decoration: none;
}.suggestions {
font-family: Arial, sans-serif;
font-size: 17px;
margin-left: 2.4%;
}.query-suggestions:hover {
background: #E3E3E3;
}.suggestion-box{
width: 635.2px;
max-width: 100%;
border: 1px solid rgba(150,150,150,0.3);
background-color: white;
margin-left: -25.7px;
position: absolute;
boxshadow: 0px 0.2px 0px;
}

A few points to notice again:

    • Box-shadow: This gives the drop up a shadow effect, which looks really nice, the first 3 parameters are for dimensions (X-offset, Y-offset, Blur). The rgba specifies color, with parameters as (red-component, green-component, blue-component, opacity).
    • Text-decoration: This attribute is used to add/remove decoration like underline for links.
    • Font-family: The font-family mentioned here is Arial, if Arial is unavailable sans-serif is used.

In the typescript file, there are two major tasks:

  1. Splice the results if greater than five. A neat suggestion-box should have a maximum of five results, hence we splice the results:

this.results.concat(res[0]);
if ( this.results.length > 5) {
this.results = this.results.splice (0, 5);
  1. Hide the suggestion-box if there are no suggestions from the API:
@Output() hidecomponent: EventEmitter<any> = new EventEmitter<any>();

this.query$.subscribe( query => {
if (query) {
this.autocompleteservice.getsearchresults(query).subscribe(res => {
if (res) {
this.results = res[1];
if (this.results.length === 0) {
this.hidecomponent.emit(1);
} else {
this.hidecomponent.emit(0);
}

If you want a more elaborate picture, you can view the entire html, css and typescript files of the auto-complete component.

This tutorial, http://4dev.tech/2016/03/tutorial-creating-an-angular2-autocomplete/ is very useful in case you want to implement auto complete suggestion feature from scratch.

In addition, this stack overflow thread has some interesting insights too: https://stackoverflow.com/questions/35881815/implementing-autocomplete-for-angular2

AngularJS structure and directives

Today I am going to tell you how to build a well worked Angular JS structure, as well as to be able to maintain your app’s elements correctly. It’s a basic and most important thing to learn and remember before you start building your own website or app because in the early phases of a project the structure doesn’t matter too much, and many people tend to ignore it, but in the long term it will affect code maintainability. It also helps you to develop it quicker and solve bugs without any additional troubles.

I must admit that I was one of them who  had made this mistake and ignore AngularJS correct structure. The thing is that I had not known how to build Angular JS app correctly before I started coding CommonsNet, and I wanted to start and provide a real outcome as quickly as possible. But unfortunately my mistake has stoped me later. I have realised it this week while implementing a quite simple feature – adding HTML elements dynamically. It turned out that I had a trouble while adding them. I had’t known that it’s better not to manipulate DOM elements in Controllers, but instead do it in directives.

Let me to explain it step by step from lessons I’ve learned while refactoring CommonsNet code structure. Please note that I’m just sharing my experience in building rather a small project and I’m covering only a small part of that subject, so some of these tips may be first – useless if you want to build an extended one, and then – not enough if you want to learn more. I recommend you to find further resources.

AngularJS Structure

First of all please see at AngularJS structure provided below. It’s a very basic structure, but it’ is important to have in mind that you should follow that pattern while developing your own app. If you want to have an order in your files you should seperate them by directories and create different ones like: controllers, directives, services, js and views. This structure makes it very easy for the reader to visualize and conceptualize the concepts you are covering.

AngularJS structure.png

My CommonsNet structure differs a bit because I have one main js directory, and all subdirectories like directives, controllers and services  inside it. I think it’s also an acceptable solution. Imagine that I hadn’t had any directives or services directives’ files before last Wednesday…

Then,  it’s also recommended to put index.html at the root of front-end structure. The index.html file will primarily handle loading in all the libraries and Angular elements.

Next, controllers are main part of AngularJS apps. It always happens that developers when starting out tend to put too much logic in the controllers. It’s a bug. Controllers should never do DOM manipulation or hold DOM selectors, that’s where we use directives and ng-model. Likewise business logic should live in services, not controllers.

Data should also be stored in services, except where it is being bound to the $scope. Services are singletons that persist throughout the lifetime of the application, while controllers are transient between application states. So, if the controller is a coordinator between the view and the model, then the amount of the logic it has should be minimal.

Directives

Take a look at a very basic AngularJS directive. Let’s name this file appInfo.js as well as create an .html file appInfo.html as in example below.

Learn AngularJS 1.X   Codecademy.png

In this example we create a new directive called appInfo. It returns an object with three options – restrict, scope and templateURL

  1. restrict specifies how the directive will be used in the view. The 'E' means it will be used as a new HTML element.
  2. scope specifies that we will pass information into this directive through an attribute named info. The = tells the directive to look for an attribute named info in the <app-info> element, like this: The data in info becomes available to use in the template given by templateURL.
  3. templateUrl specifies the HTML to use in order to display the data in scope.info. Here we use the HTML in js/directives/appInfo.html.

Then we need to build a structure to directive in our appInfo.html

<h2 class="title">{{ info.title }}</h2> 
<p class="developer">{{ info.developer }}</p> 
<p class="price">{{ info.price | currency }}</p>

Of course <h2> and <p> are just examples. You can use here what you only want. The title and developer are controller $scope.elements. Let’s take a look at controller’s content.

$scope.apps = [ 
 { 
 title: 'MOVE', 
 developer: 'MOVE, Inc.', 
 price: 0.99 
 }, 
 { 
 title: 'Shutterbugg', 
 developer: 'Chico Dusty', 
 price: 2.99 
 }, 
]

In index.html we use only  ng-repeat to loop through app element in $scope.apps, and then app-info info=”app”

<div ng-repeat="app in apps">
     <app-info info="app"></app-info>
</div>

That’s it. It’s quite a simple example, but it helps me to refactor my code in CommonsNet project and avoid a long, boring, repetitive work while adding new elements to website. It also helps me to implement a feature I have mentioned above. I’am going to tell more about it in next blog post. Stay tuned!

Resources:

https://scotch.io/tutorials/angularjs-best-practices-directory-structure

https://www.codecademy.com/learn/learn-angularjs.

https://commonsnetblog.wordpress.com/

 

sTeam REST API Unit Testing

(ˢᵒᶜⁱᵉᵗʸserver) aims to be a platform for developing collaborative applications.
sTeam server project repository: sTeam.
sTeam-REST API repository: sTeam-REST

Unit Testing the sTeam REST API

The unit testing of the sTeam REST API is done using the karma and the jasmine test runner. The karma and the jasmine test runner are set up in the project repository.

The karma test runner : The main goal for Karma is to bring a productive testing environment to developers. The environment being one where they don’t have to set up loads of configurations, but rather a place where developers can just write the code and get instant feedback from their tests. Because getting quick feedback is what makes you productive and creative.

The jasmine test runner: Jasmine is a behavior-driven development framework for testing JavaScript code. It does not depend on any other JavaScript frameworks. It does not require a DOM. And it has a clean, obvious syntax so that you can easily write tests.

The karma and jasmine test runner were configured for the project and basic tests were ran. The angular js and angular mocks version in the local development repository was different. This had resulted into a new error been incorporated into the project repo. The ‘angular.element.cleanData is not a function’ error is thrown in the local development repository. This error happens when the local version of the angular.js and angular-mocks.js doesn’t match. The testing framework would test you if the versions f the two libraries is not the same.

The jasmine test runner can be accessed from the browser. The karma tests can be performed from the command line.

To access the jasmine test runner from the web browser, go to the url

http://localhost:7000/test/unit/runner.html

To run the karma test suite, run the following command

$ karma start

The unit tests of the sTeam REST service were done using jasmine. The unit tests were written in coffee script. The preprocessor to compile the files from coffee script to javascript is defined in the karma configuration file.

Jasmine Test RunnerJasmineRunner
Jasmine Test Failure

JasmineRunnerFailure

First a dummy pass case and a fail case is tested to check there are no errors in the test suite during the test execution.

The localstoragemodule.js which is used in the steam service is injected in the test module. Then the steam service version is tested.

describe 'Check version of sTeam-service', -> 
 		it 'should return current version', inject (version) -> 
 			expect(version).toEqual('0.1') 

steam service should be injected in a global variable as the same service functions shall be tested while performing the remaining tests.
Then the steam service is injected and checked whether it exists or not.

beforeEach inject (_steam_) -> 
 		steam= _steam_ 
 	describe 'Check sTeam service injection', ->  
 		it 'steam service should exist', -> 
 			expect(steam).toBeDefined() 

The sTeam service has both private and public functions. The private functions cannot be accessed from outside. The private functions defined in the sTeam service arehandle_request and headers.

describe 'Check sTeam service functions are defined.', ->  
 		describe ' Check the sTeam REST API private functions.', -> 
 			it 'steam service handle request function should exist', -> 
 				expect(steam.handle_request).toBeDefined() 
 			it 'steam service headers function should exist', -> 
 				expect(steam.headers).toBeDefined() 

The public functions of the sTeam service are then tested.

describe 'Check sTeam service functions are defined.', ->  
 		describe ' Check the sTeam REST API public functions.', -> 
 			it 'steam service login function should exist', -> 
 				expect(steam.login).toBeDefined() 
 			it 'steam service loginp function should exist', -> 
 				expect(steam.loginp).toBeDefined() 
 			it 'steam service logout function should exist', -> 
 				expect(steam.logout).toBeDefined() 
 			it 'steam service user function should exist', -> 
 				expect(steam.user).toBeDefined() 
 			it 'steam service get function should exist', -> 
 				expect(steam.get).toBeDefined() 
 			it 'steam service put function should exist', -> 
 				expect(steam.put).toBeDefined() 
 			it 'steam service post function should exist', -> 
 				expect(steam.post).toBeDefined() 
 			it 'steam service delete function should exist', -> 
 				expect(steam.delete).toBeDefined() 

The test suite written by Siddhant for the sTeam server was tested. The merging of the code from different branches which includes the work done during the course of GSoC 2016 will be merged subsequently.

Karma test runner

KarmaTestCase

Feel free to explore the repository. Suggestions for improvements are welcomed.

Checkout the FOSSASIA Idea’s page for more information on projects supported by FOSSASIA.

 

Angular API testing with frisby and jasmine-node

week9gsoc1

sTeam-web-UI is an application written in angular. So the application relies on sTeam's existing REST api which is written in pike. Since there are a lot of API calls and actions involved ensuring that the application is making proper API calls and getting proper responses is crucial. In order to go ahead with the API testing there are two important things to be know about first.

  • frisby
  • jasmine

What is frisby ?

“Frisby is a REST API testing framework built on node.js and Jasmine that makes testing API endpoints easy, fast, and fun. Read below for a quick overview, or check out the API documentation.”

What is Jasmine ?

“Jasmine is a Behavior Driven Development testing framework for JavaScript. It does not rely on browsers, DOM, or any JavaScript framework. Thus it’s suited for websites, Node.js projects, or anywhere that JavaScript can run.”

The api methods present in frisby can be categorized into :

  • Expectations
  • Helpers
  • Headers
  • Inspectors

So let me show couple of tests that are written with the help of frisby for the sTeam's web interface.

Checking the Home directory of the user :


frisby.create('Request `/home` returns proper JSON')
  .get(config.baseurl + 'rest.pike?request=/home')
  .expectStatus(200)
  .expectJSON({
    'request': '/home',
    'request-method': 'GET',
    'me': objUnit.testMeObj,
    'inventory': function (val) {
      val.forEach(function (e) {
        objUnit.testInventoryObj(e)
      })
    }
  })
.toss()

Checking the container of the user :


frisby.create('Request `/home/:user` returns proper JSON')
  .get(config.baseurl + 'rest.pike?request=/home/akhilhector/container')
  .expectStatus(200)
  .expectJSON({
    'request': '/akhilhector',
    'request-method': 'GET',
    'me': objUnit.testMeObj,
    'inventory': function (val) {
      val.forEach(function (e) {
        objUnit.testInventoryObj(e)
      })
    }
  })
.toss()

Accessing the file in the user’s workarea :


frisby.create('Request `/home/:user/:container/:filename` returns proper JSON')
  .get(config.baseurl + 'rest.pike?request=//home/akhilhector/playground-hector/icon1.png')
  .expectStatus(200)
  .expectJSON({
    'request': '/akhilhector/playground-hector/icon1.png',
    'request-method': 'GET',
    'me': objUnit.testMeObj,
    'inventory': function (val) {
      val.forEach(function (e) {
        objUnit.testInventoryObj(e)
      })
    }
  })
.toss()

Observing the above three modules, each of them are different from one another. We have to understand that depending on the API the request payload must be sent. So for suppose the API needs authentication headers to be sent then we need to use the proper helper methods of frisby in order to send the same.
The .get() can be substituted with put, post, delete. Basically all the frisby tests start with frisby.create with a description of the test followed by one of get, post, put, delete, or head, and ending with toss to generate the resulting response.
Apart from the request payload there is an interesting thing with the helper method expectJSON, it is used for testing the given path or full JSON response of the specified length. In order to use expectJSON we must pass the path as the primary argument and the JSON as the second argument.
The expectStatus helper method is pretty straightforward. It helps us in determining the HTTP Status code equivalent of the request so made.

NOTE While writing the tests it is better to put all the config in one place and use that config variable for all the operations.

Thats it folks,
Happy Hacking !!

sTeam API Endpoint Testing

(ˢᵒᶜⁱᵉᵗʸserver) aims to be a platform for developing collaborative applications.
sTeam server project repository: sTeam.
sTeam-REST API repository: sTeam-REST

sTeam API Endpoint Testing using Frisby

sTeam API endpoint testing is done using Frisby.  Frisby is a REST API testing framework built on node.js and Jasmine that makes testing API endpoints very easy, speedy and joyous.

Issue. Github Issue Github PR
sTeam-REST Frisby Test for login Issue-38 PR-40
sTeam-REST Frisby Tests Issue-41 PR-42

Write Tests

Frisby tests start with frisby.create with a description of the test followed by one of get, post, put, delete, or head, and ending with toss to generate the resulting jasmine spec test. Frisby has many built-in test helpers like expectStatus to easily test HTTP status codes, expectJSON to test expected JSON keys/values, and expectJSONTypes to test JSON value types, among many others.

// Registration Tests
frisby.create('Testing Registration API calls')
.post('http://steam.realss.com/scripts/rest.pike?request=register', {
email: "[email protected]",
fullname: "Ajinkya Wavare",
group: "realss",
password: "ajinkya",
userid: "aj007"
}, {json: true})
.expectStatus(200)
.expectJSON({
"request-method": "POST",
"request": "register",
"me": restTest.testMe,
"__version": testRegistrationVersion,
"__date": testRegistrationDate
})
.toss();
The testMe, testRegistrationVersion and testRegistrationDate are the functions written in the rest_spec.js.

The frisby API endpoint tests have been written for testing the user login, accessing the user home directory, user workarea, user container, user document, user created image,  groups and subgroups.

The REST API url’s used for testing are described below. A payload consists of the user id and password.

Check if the user can login.

http://steam.realss.com/scripts/rest.pike?request=aj007

Test whether a user workarea exists or not. Here aj workarea has been created by the user.

http://steam.realss.com/scripts/rest.pike?request=aj007/aj

Test whether a user created container exists or not.

http://steam.realss.com/scripts/rest.pike?request=aj007/container

Test whether a user created document exists or not.

http://steam.realss.com/scripts/rest.pike?request=aj007/abc.pike

Test whether a user created image(object of any mime-type) inside a container exists or not.

http://steam.realss.com/scripts/rest.pike?request=aj007/container/Image.jpeg

Test whether a user created document exists or not. The group name and the subgroups can be queried.
eg. GroupName: groups, Subgroup: test.
The subgroup should be appended using “.” to the groupname.

http://steam.realss.com/scripts/rest.pike?request=groups.test

Here “groups” is a Groupname and “gsoc” is a subgroup of it.

http://ngtg.techgrind.asia/scripts/rest.pike?request=groups.gsoc

FrisbyTests

FrisbyTestCount

Unit Testing the sTeam REST API

The unit testing of the sTeam REST API is done using the karma and the jasmine test runner. The karma and the jasmine test runner are set up in the project repository.

The karma test runner : The main goal for Karma is to bring a productive testing environment to developers. The environment being one where they don’t have to set up loads of configurations, but rather a place where developers can just write the code and get instant feedback from their tests. Because getting quick feedback is what makes you productive and creative.

The jasmine test runner: Jasmine is a behavior-driven development framework for testing JavaScript code. It does not depend on any other JavaScript frameworks. It does not require a DOM. And it has a clean, obvious syntax so that you can easily write tests.

The karma and jasmine test runner were configured for the project and basic tests were ran. The angular js and angular mocks version in the local development repository was different. This had resulted into a new error been incorporated into the project repo.

Angular Unit-Testing: TypeError ‘angular.element.cleanData is not a function’

When angular and angular-mocks are not of the same version, these error occurs while running the tests. If the versions of the two javascript libraries don’t match your tests will be testing to you.

The jasmine test runner can be accessed from the browser. The karma tests can be performed from the command line. These shall be enhanced further during the course of work.

Feel free to explore the repository. Suggestions for improvements are welcomed.

Checkout the FOSSASIA Idea’s page for more information on projects supported by FOSSASIA.

Developing Image Viewer in Angular JS

week8gsoc0

In the previous article i have addressed about developing audio and video players for sTeam. To know more about developing audio and video players, click here. This article is an extension to the previous one, as to how one could develop image viewer in angular.
Since sTeam is a collaboration platform there was the necessity of having an image viewer. So in angular bootstrapLightbox is very useful in building image viewers, galleries, sliders etc.

Implementation Strategy :
There are basically two things involved in developing this;

The Controller

angular.module('steam', [ 'bootstrapLightbox', 'ui.bootstrap' ])
  .controller('steamImageViewer', ['$scope', 'Lightbox', function($scope, Lightbox) {
    $scope.images = [
    {
      'url': 'https://societyserver.org/home/akhilhector/playground%20-%20hector/icon1.png',
      'title': 'icon1.png'
    },
    {
      'url': 'https://societyserver.org/home/akhilhector/playground%20-%20hector/icon3.jpg',
      'title': 'icon3.jpg'
    },
    {
      'url': 'https://societyserver.org/home/akhilhector/playground%20-%20hector/icon4.png',
      'title': 'icon4.png'
    },
    {
      'url': 'https://societyserver.org/home/akhilhector/playground%20-%20hector/icon4.jpg',
      'title': 'icon4.jpg'
    }
    ];
    $scope.imageViewer = function (index) {
      Lightbox.openModal($scope.images, index);
    };
  }]);

Observe the controller over here, we pass an array of images or a single image to an array object and then we use the same for our DOM. Options like URL, Title, and Thumbnail can be passed in the same object. In the same controller we need to write another function that uses the LightBox service in order to open a modal and serve the image which was passed as an URL. So the modal displays whatever source is being passed as an argument. Also if we are using a REST api for retrieving the images then we can have a we can have similar array object that stores all the information recieved from the api response. In order to better this, there can be a mime-handler written so that if an unknown/unsupported mime type is being requested for, then an error can be popped.

The HTML

<div class="row" ng-controller="steamImageViewer">
    <div class="col-md-12" ng-repeat="image in images">
        <div class="roomItem">
            <div class="roomItemIcon roomItemBlue"><i class="fa fa-image"></i>
            </div>
            <div class="itemText"> {{image.title}} <a href="#" ng-click="imageViewer($index)">view</a>
            </div>
        </div>
    </div>
</div>

The above DOM seems pretty straight forward, we use the help of the function which we previously wrote in the controller in order to get the data. So as soon as we click the Lightbox modal fires and displays the image. It is to be observed that this implementation goes with consideration that there are an array of images that are ought to be retrieved, but the same logic wouldn’t be fruitful for displaying a single image. In order to acheive that we can pass the image object to the $scope and use the same for displaying the image. And minor changes in the HTML will help displaying a single image instead of an array of images.

So here is how it looks :

week8gsoc3

week8gsoc2

week8gsoc1

Thats it folks,
Happy Hacking !!

Generating Documentation and Modifying the sTeam-REST API

(ˢᵒᶜⁱᵉᵗʸserver) aims to be a platform for developing collaborative applications.
sTeam server project repository: sTeam.
sTeam-REST API repository: sTeam-REST

Documentation

Documentation is an important part of software engineering. Types of documentation include:

  1. Requirements – Statements that identify attributes, capabilities, characteristics, or qualities of a system. This is the foundation for what will be or has been implemented.
  2. Architecture/Design – Overview of software. Includes relations to an environment and construction principles to be used in design of software components.
  3. Technical – Documentation of code, algorithms, interfaces, and APIs.
  4. End user – Manuals for the end-user, system administrators and support staff.
  5. Marketing – How to market the product and analysis of the market demand.

Doxygen

Doxygen is the de facto standard tool for generating documentation from annotated C++ sources, but it also supports other popular programming languages such as C, Objective-C, C#, PHP, Java, Python, IDL (Corba, Microsoft, and UNO/OpenOffice flavors), Fortran, VHDL, Tcl, and to some extent D.
The Doxygen treats files of other languages as C/C++ and creates documentation for it accordingly.
sTeam documentation was tried to be created with the doxygen. But empty documentation was created due to the lack of the doxygen annotations used in the project.
 Doxygen doc generation.
DoxyGenTerminal
Doxygen Docs
DoxyDoc
The next way to create documentation was to make use of the autodoc utility provided by the Pike. The utility to generate docs was provided in the later versions of the Pike(>=8.0.155).
The autodoc files are generated and  later these are converted into  html pages. The commands used for generating the autodoc include:-
pike -x extract_autodoc /source
pike -x autodoc_to_html /src /opfile
The autodoc_to_html utility converts a single autodoc file to an html page. As a result a shell script was written to convert all the generated autodoc files to the html file.
docGenerator.sh
#!/bin/bash  

shopt -s globstar  
for filename in ./**/*.pike.xml; do  
    outputFile=doc/${filename#./}  
    outputFile=${outputFile%.xml}."html"  
    if [ -d $(dirname "./"$outputFile) ]; then  
      touch "./"$outputFile  
    else  
      mkdir -p $(dirname "./"$outputFile) && touch "./"$outputFile  
    fi  
    pike -x autodoc_to_html $filename "./"$outputFile  
done  

Autodoc Documentation
AutoDocThe documentation generated by this was less informative and lacked the referrals to other classes and headers. The societyserver project was developed long back but the autodoc utility was introduced in the later versions of pike. As a result the source files lacked the autodoc tags which are required to generate a well informative documentation with bindings to other files.

Restructuring the sTeam-REST API

The sTeam-REST API project made use of the angular-seed to initiate the development during the early phases. However these files still existed in the project. This had lead to a pandemonium and created difficulty in understanding the project. The files had to be removed and the app was in dire need of a restructuring. The following issues have been reported and resolved.

Issue. Github Issue Github PR
sTeam-REST Issues Issues PR

The new UI can be seen below.

Home

Home

Register

Register

About

About

Testing the REST API

The functionality to run the tests using the npm test command was added to the project. Now the user can run the tests using these commands instead of the traditional approach of running the tests using jasmine-node and specifying the test directory. The domain name of the urls to which the request was sent was changed. The e2e tests and frisby tests were conducted successfully.

e2e Tests.

e2eTests

Frisby Tests

NPM tests

The next step would be to do add more tests for the REST API.

Feel free to explore the repository. Suggestions for improvements are welcomed.

Checkout the FOSSASIA Idea’s page for more information on projects supported by FOSSASIA.

sTeam REST API

(ˢᵒᶜⁱᵉᵗʸserver) aims to be a platform for developing collaborative applications.
sTeam server project repository: sTeam.
sTeam-REST API repository: sTeam-REST

REST Services

REST is the software architectural style of the World Wide Web. REST (Representational State Transfer) was introduced by Roy Fielding in his doctoral dissertation in 2000. Its purpose is to induce performance, scalability, simplicity, modifiability, visibility, portability, and reliability.It has client/server relationship with a uniform interface and is stateless. REST is most commonly associated with HTTP but it is not strictly related to it.

REST Principles

  • Resources : Each and every component is a resource.A resource is accessed by a common interface using HTTP standard methods.
  • Messages use HTTP methods like GET, POST, PUT, and DELETE.
  • Resource identification through URI: Resources are identified using URI. Resources are represented using JSON or XML.
  • Stateless interactions take place between the server and the client. No context is saved for the requests at the server.The client maintains the state of the session.

HTTP methods

The CRUD(create, retrieve, update and delete ) operations are performed using the HTTP methods.

GET

It is used to retrieve information. GET requests executed any number of times with the same parameters, the results would not change. This makes it idempotent. Partial or conditional requests can be sent. It is a read only type of operation.

Retrieve a list of users:

GET /api.example.com/UserService/users

POST

POST is usually used to create a new entity. It can also be used to update an existing entity. The request will have to do something with the entity provided in the URI.

Create a new user with an ID 2:

POST /api.example.com/UserService/users/2

PUT

PUT request is always idempotent. Executing the same request any number of times will not change the output. PUT can be used to create or update an existing entity.

Modify the user with an ID of 1:

PUT /api.example.com/UserService/users/1

PATCH

It is idempotent. PATCH requests only updates the specified fields of an entity.

Modify a user with an id of 1:

PATCH /api.example.com/UserService/users/1

DELETE

It can be asynchronous or a long-running request. It removes the resource. It can be removed immediately or at a later instance of time.

Delete a user with an ID of 1:

DELETE /api.example.com/UserService/users/1

 sTeam-REST API

Installing and activating the REST API

The REST API is developed as an application inside the sTeam server. This simplifies development quite a lot, as we don’t need to restart the server for every change. Instead just the API code gets updated and reloaded. It may eventually be integrated into the core, however the longterm plan is actually to move functionality out of the core, to make development easier.

To get the current version of the API clone the steam-rest repo into your home or to any place where you keep your development repos. Then change to the tools directory of your installation and run import-from-git.

git clone https://github.com/societyserver/steam-rest
cd steam-rest
git checkout origin/rest.pike
export steamrest=`pwd`
cd /usr/local/lib/steam/tools
./import-from-git.pike -u root $steamrest /

Note: The new import-from-git.pike script supports importing documents of all mime types.

It is important that the first import is done as root because the API code needs to run with root privileges and it will only do that if the object that holds the source is created as root.

Once the api code is loaded there are just a few tweaks needed to make it work.

We need to fix the mime-type, as the import script is not doing that yet.

OBJ("/sources/rest.pike")->set_attribute("DOC_MIME_TYPE", "source/pike");

Changing the mime type will change the class of the rest api script from Document to DocLpc.

> OBJ("/sources/rest.pike");                                               
(1) Result: 127.0.0.1:1900/rest.pike(#840,/classes/Document,17,source/pike)
> OBJ("/sources/rest.pike");                                               
(2) Result: 127.0.0.1:1900/rest.pike+(#840,/classes/DocLpc,529,source/pike,0 Instances, ({  }))

This takes a moment, check the type a few times until it’s done. Then instantiate an object from the source, give it a proper name, and move it to the /scripts/ container”

object rest = OBJ("/sources/rest.pike")->provide_instance();
rest->set_attribute("OBJ_NAME", "rest.pike");
rest->move(OBJ("/scripts/"));

Instantiating the object needs to be done as sTeam-root, in order for it to have permissions to run on behalf of other users.

Once this is done you are ready to start using the API.

sTeam-REST API tests

The project contains a set of examples and tests for the RESTful API for the sTeam server.

The code is written in coffee script and needs node.js only for coffeescript translation. Deployment can be done as static javascript files, and does not need any kind of dynamic server for the front-end. The back-end is a RESTful API written for the sTeam server as used by steam.realss.com

Development instructions

step 1: install node.js

http://nodejs.org/download/

step 2: clone the repository

git clone https://github.com/societyserver/steam-rest

step 3: install node packages:

npm install

This installs all dependencies (including coffee) for our project into the project’s node_modules directory based on the ‘package.json’ file

step 4: start the server

node_modules/.bin/coffee scripts/server.coffee

but for convenience we can install coffee in the global node environment:

npm install -g coffee-script

so we can just say

coffee scripts/server.coffee

if the server is working you’ll see:

Listening on port 7000

Testing

FrisbyJS is used to test the API. It is run through Jasmine and is based on nodejs.

Once you have nodejs installed, run the following statement to install Frisby and Jasmine:

npm install -g jasmine-node frisby

Then execute the test by:

cd project/directory
jasmine-node test/

The karma testing framework is also used for testing the sTeam REST API.

There were some inherent issues with the test framework which were addressed.

Issue. Github Issue Github PR
Update Readme.md Update Readme PR-2
Add javascript dependencies Issue-4 PR-6
Add node dependencies Issue-5 PR-7
Add angular-mocks.js script for testing the REST services. Issue-8 PR-9

The project dependencies were not met and this resulted into the error when the project was run on the localhost. The angular-ui-router, angular-bootstrap and bootstrap js frameworks were not installed in the node modules of the project. As a result the bower.json script was modified to include these dependencies.

bower.json

{
  "name": "bower",
  "version": "0.1",
  "private": true,
  "ignore": [
    "**/.*",
    "node_modules",
    "bower_components",
    "test",
    "tests"
  ],
  "dependencies": {
    "angular": "",
    "angular-route": "~1.4.8",
    "angular-ui-router": "",
    "angular-bootstrap": "",
    "bootstrap": ""
  }
}

The node dependencies of karma, frisby and jasmine-node were included in the package.json. These would be installed when the npm install is executed.

package.json

{
"name": "TechGrind",
"version": "0.1.1",
"private": true,
"dependencies": {
"express": "",
"coffee-script": "",
"morgan": "",
"compression": "",
"method-override": "",
"body-parser": "",
"serve-static": "",
"errorhandler": "",
"bower": "",
"jasmine-node": "",
"frisby": "",
"karma": ""
},
"production_dirs": {
"coffee_src": "src/",
"src": "app/",
"dest": "app_production/"
},
"devDependencies": {
},
"scripts": {
"postinstall": "bower install"
}
}

Feel free to explore the repository. Suggestions for improvements are welcomed.

Checkout the FOSSASIA Idea’s page for more information on projects supported by FOSSASIA.