Writing Simple Unit-Tests with JUnit

In the Loklak Server project, we use a number of automation tools like the build testing tool ‘TravisCI’, automated code reviewing tool ‘Codacy’, and ‘Gemnasium’. We are also using JUnit, a java-based unit-testing framework for writing automated Unit-Tests for java projects. It can be used to test methods to check their behaviour whenever there is any change in implementation. These unit-tests are handy and are coded specifically for the project. In the Loklak Server project it is used to test the web-scrapers. Generally JUnit is used to check if there is no change in behaviour of the methods, but in this project, it also helps in keeping check if the website code has been modified, affecting the data that is scraped.

Let’s start with basics, first by setting up, writing a simple Unit-Tests and then Test-Runners. Here we will refer how unit tests have been implemented in Loklak Server to familiarize with the JUnit Framework.

Setting-UP

Setting up JUnit with gradle is easy, You have to do just 2 things:-

1) Add JUnit dependency in build.gradle

Dependencies {

. . .

. . .<other compile groups>. . .

compile group: 'com.twitter', name: 'jsr166e', version: '1.1.0'

compile group: 'com.vividsolutions', name: 'jts', version: '1.13'

compile group: 'junit', name: 'junit', version: '4.12'

compile group: 'org.apache.logging.log4j', name: 'log4j-1.2-api', version: '2.6.2'

compile group: 'org.apache.logging.log4j', name: 'log4j-api', version: '2.6.2'

. . .

. . .

}

 

2) Add source for ‘test’ task from where tests are built (like here).

Save all tests in test directory and keep its internal directory structure identical to src directory structure. Now set the path in build.gradle so that they can be compiled.

sourceSets.test.java.srcDirs = ['test']

 

Writing Unit-Tests

In JUnit FrameWork a Unit-Test is a method that tests a particular behaviour of a section of code. Test methods are identified by annotation @Test.

Unit-Test implements methods of source files to test their behaviour. This can be done by fetching the output and comparing it with expected outputs.

The following test tests if twitter url that is created is valid or not that is to be scraped.

/**

 * This unit-test tests twitter url creation

 */

@Test

public void testPrepareSearchURL() {

String url;

String[] query = {"fossasia", "from:loklak_test",

"spacex since:2017-04-03 until:2017-04-05"};

String[] filter = {"video", "image", "video,image", "abc,video"};

String[] out_url = {

"https://twitter.com/search?f=tweets&vertical=default&q=fossasia&src=typd",

"https://twitter.com/search?f=tweets&vertical=default&q=from%3Aloklak_test&src=typd",

"and other output url strings to be matched…..."

};


// checking simple urls

for (int i = 0; i < query.length; i++) {

url = TwitterScraper.prepareSearchURL(query[i], "");


//compare urls with urls created

assertThat(out_url[i], is(url));

}


// checking urls having filters

for (int i = 0; i < filter.length; i++) {

url = TwitterScraper.prepareSearchURL(query[0], filter[i]);


//compare urls with urls created

assertThat(out_url[i+3], is(url));

}

}

 

Testing the implementation of code is useless as it will either make code more difficult to change or tests useless  . So be cautious while writing tests and keep difference between Implementation and Behaviour in mind.

This is the perfect example for a simple Unit-Test. As we see there are some points, which needs to be observed like:-

1) There is a annotation @Test .

2) Input array of query which is fed to the method TwitterScraper.prepareSearchURL() .

3) Array of urls out_url[], which are the expected urls to output.

4) asserThat() to compare the expected url (in array out_url[]) and the output url (in variable ‘url’).

NOTE: assertEquals() could also be used here, but we prefer to use assert methods to get error message that is readable (We will discuss about this some time later)

And the TestRunner

When we are working on a project, It is not feasible to run tests using gradle as they are first built  (else verified whether tests are build-ready) and then executed. gradle test shall be used only for building and testing the tests. For testing the project, one shall set-up TestRunner. It allows to run specific set of tests, one wants to run.

TestRunners are built once using gradle (with other tests) and can be run whenever you want. Also it is easy to stack up the test classes you want to run in SuiteClasses and @RunWith to run SuiteClasses with the TestRunner.

In loklak server, TestRunner runs the web-scraper tests. They are used by developers to test the changes they have made.

This is a sample TestRunner, code link here .

package org.loklak;


// Library classes imported

import org.junit.runner.RunWith;

import org.junit.runners.Suite;

// Source files to be tested

import org.loklak.harvester.TwitterScraperTest;

import org.loklak.harvester.YoutubeScraperTest;


/*

* TestRunner for harvesters

*/

@RunWith(Suite.class)

@Suite.SuiteClasses({

TwitterScraperTest.class,

YoutubeScraperTest.class

})

public class TestRunner {

}

 

You can also add TestRunners for different sections of the project. Like here it is initialized only to test harvesters.

To run the TestRunner

Add classpath of the jar file of the project and run ‘JUnitCore’ with TestRunner to get output on terminal.

java -classpath .:build/libs/<yourProject>.jar:build/classes/test org.junit.runner.JUnitCore org.loklak.TestRunner

In the project we have set up a shell script to run the tests.

Few points

1) Build the project and tests separately. Build tests only when changed as they take time to be built and executed.

2) Whenever you are done with the coding part, run the tests using TestRunner.

3) Write unit-tests whenever you add a new feature to the project to keep it up-to-date.

Now lets end up here.

So for now, Code it, Test it and Repeat.

Resources:

Continue ReadingWriting Simple Unit-Tests with JUnit

How to teach SUSI skills calling an External API

SUSI is an intelligent  personal assistant. SUSI can learn skills to understand and respond to user queries better. A skill is taught using rules. Writing rules is an easy task and one doesn’t need any programming background too. Anyone can start contributing. Check out these tutorials and do watch this video to get started and start teaching susi.

SUSI can be taught to call external API’s to answer user queries.

While writing skills we first mention string patterns to match the user’s query and then tell SUSI what to do with the matched pattern. The pattern matching is similar to regular expressions and we can also retrieve the matched parameters using $<parameter number>$ notation.

Example :

My name is *
Hi $1$!

When the user inputs “My name is Uday” , it is matched with “My name is *” and “Uday” is stored in $1$. So the output given is “Hi Uday!”.

SUSI can call an external API to reply to user query. An API endpoint or url when called must return a JSON or JSONP response for SUSI to be able to parse the response and retrieve the answer.

Rule Format for a skill calling an external API

The rule format for calling an external API is :

<regular expression for pattern matching>
!console: <return answer using $object$ or $required_key$>
{
  “url”:<API endpoint or url>,
  “path”:$.<key in the API response to find the answer>,
}
eol
  • Url is the API endpoint to be called which returns a JSON or JSONP response.
    The parameters to the url if any can be added using $$ notation.
  • Path is used to help susi know where to look for the answer in the returned response.
    If the path points to a root element, then the answer is stored in $object$, otherwise we can query $key$ to get the answer which is a value to the key under the path.
  • eol or end of line indicates the end of the rule.

Understanding the Path Attribute

Let us understand the Path attribute better through some test cases.

In each of the test cases we discuss what the path should be and how to retrieve the answer for a given required answer from the json response of an API.

  1. API response in json :

    {
      “Key1” : “Value1”
    }
    

Required answer : Value1
Path : “$.Key1    =>   Retrieve Answer:  $object$

 

  1. API response in json :
{
  “Key1” : [{“Key11” : “Value11”}]
}

Required answer : Value11
Path : $.Key1[0]   =>  Retrieve Answer: $Key11$
Path : $.Key1[0].Key11   => Retrieve Answer: $object$

 

  1. API response in json :
{
  “Key1” : {“Key11” : “Value11”}
}


Required answer : Value11
Path : $.Key1  => Retrieve Answer:  $Key11$
Path : $.Key1.Key11  => Retrieve Answer: $object$

 

  1. API response in json :
{
  “Key1” : {
           “Key11” : “Value11”,
           “Key12” : “Value12”
         }
}

Required answer : Value11 , Value12
Path : $.Key1  => Retrieve Answer:  $Key11$ , $Key12$

Where to write these rules?

Now, since we know how to write rules let’s see where to write them.

We use etherpads to write and test rules and once we finish testing our rule we can push those rules to the repo.

Steps to open, write and test rules:

  1. Open a new etherpad with a desired name <etherpad name> at http://dream.susi.ai/
  2. Write your skills code in the etherpad following the code format explained above.
  3. Now, to test your skill let’s chat with susi. Start a conversation with susi at http://susi.ai/chat to test your skills.
  4. Load your skills by typing dream <etherpad name> and wait for a response saying dreaming enabled for <etherpad name>
  5. Test your skill and follow step 4 every time you make changes to the code in your etherpad.
  6. After you are done testing, type stop dreaming and if you are satisfied with your skill do send a PR to help susi learn.

Examples

Let us try an example to understand this better.

1. Plot of a TV Show

Tvmaze is an open  TV API that provides information about tv shows. Let us write a rule to know the plot of a tv show. We can find many such APIs. Check out this link listing few of them.

  1.  Open an etherpad at http://dream.susi.ai/ named tvshowplot.
  2.   Enter the code to query plot of a TV show in the etherpad at                           http://dream.susi.ai/p/tvshowplot
* plot of *|* summary of *
!console:$object$
{
  "url":"http://api.tvmaze.com/singlesearch/shows?q=$2$",
  "path":"$.summary"
}
eol
  1. Now lets test our skill by starting a conversation with susi at http://susi.ai/chat.
  • User Query: dream tvshowplot
    Response:  dreaming enabled  for tvshowplot
  • User Query: what is the plot of legion
    Response: Legion introduces the story of David Haller: Since he was a teenager, David has struggled with mental illness. Diagnosed as schizophrenic, David has been in and out of psychiatric hospitals for years. But after a strange encounter with a fellow patient, he’s confronted with the possibility that the voices he hears and the visions he sees might be real. He’s based on the Marvel comics character Legion, the son of X-Men founder Charles Xavier (played by Patrick Stewart and James McAvoy in the films), first introduced in 1985.

Intermediate Processing:

Pattern Matching : $1$ = “what is the” ; $2$ = “legion”

Url : http://api.tvmaze.com/singlesearch/shows?q=legion

API response:

{
  "id": 6393,
  "url": "http:\/\/www.tvmaze.com\/shows\/6393\/legion",
  "name": "Legion",
  "type": "Scripted",
  "language": "English",
  "genres": [
             "Drama",
             "Action",
             "Science-Fiction"
            ],
  "summary": "<p><strong>Legion<\/strong> introduces the story of David Haller: Since he was a teenager, David has struggled with mental illness. Diagnosed as schizophrenic, David has been in and out of psychiatric hospitals for years. But after a strange encounter with a fellow patient, he's confronted with the possibility that the voices he hears and the visions he sees might be real. He's based on the Marvel comics character Legion, the son of X-Men founder Charles Xavier (played by Patrick Stewart and James McAvoy in the films), first introduced in 1985.<\/p>",
  "updated": 1491955072,
}

Note: The API response has been trimmed to show only the relevant content.

Path : $.summary

Retrieving Answer: so our required answer in the api response is under the key summary and is retrieved using $object$ since it is a root element.

 

Screenshots:

2. Cooking Recipes

Let us try it out with another API.
Recipepuppy is an cooking recipe API where users can query various recipes.

  1.  Open a etherpad at http://dream.susi.ai/ named recipe.
  2.   Enter the code to query a recipe in the etherpad at  http://dream.susi.ai/p/recipe
#Gives recipes and links to cook a dish
* cook *
!console:<p>To cook  <strong>$title$</strong> : <br>The ingridients required are: $ingredients$. <br> For instruction to prepare the dish $href$ </p>
{
  "url":"http://www.recipepuppy.com/api/?q=$2$",
  "path":"$.results"
}
eol
  1. Now lets test our skill by starting a conversation with susi at http://susi.ai/chat.
  • User Query: dream recipe
    Response:  dreaming enabled  for recipe
  • User Query: how to cook chicken biryani
    Response: To cook Chicken Biryani Recipe :
    The ingridients required are: chicken, seeds, chicken broth, rice, butter, peas, garlic, red onions, cardamom, curry paste, olive oil, tomato, coriander, cumin, brown sugar, tumeric.
    For instruction to prepare the dish Click Here!

Intermediate Processing:

Pattern Matching : $1$ = “how to” ; $2$ = “chicken biryani”

Url : http://www.recipepuppy.com/api/?q=chicken biryani

API response:

{
  "title": "Recipe Puppy",
  "version": 0.1,
  "href": "http:\/\/www.recipepuppy.com\/",
  "results": [
              {
                "title": "Chicken Biryani Recipe",
                "href": "http:\/\/www.grouprecipes.com\/53040\/chicken-biryani.html",
                "ingredients": "chicken, seeds, chicken broth, rice, butter, peas, garlic, red onions, cardamom, curry paste, olive oil, tomato, coriander, cumin, brown sugar, tumeric",
                "thumbnail": "http:\/\/img.recipepuppy.com\/413822.jpg"
             },
           ]
}

Note: The API response has been trimmed to show only the relevant content.

Path : $.results[0]

Retrieving Answer: so our required answer in the api response is under the key results and since it’s an array we are using the first element of the array and since the element is a dictionary too we use its keys correspondingly to answer. The $href$ is rendered as “Click Here” hyperlinked to the actual url.

 

Screenshots:

 

We have successfully taught susi a skill which tells users about the plot of a tv show and a skill to query recipes.
Cheers!
Following similar procedure, we can make use of other APIs and teach susi several new skills.

Resources
Continue ReadingHow to teach SUSI skills calling an External API

Deploying Susi Server on Google Cloud with Kubernetes

Susi (acronym for Scientific User Support Intelligence) is an advanced AI made by people at FOSSASIA. It is an AI made by the people and for the people.
Susi is an Open Source Project under LGPL Licence.

SUSI.AI already has many Skills and anyone can add new skills through simple console rules.

If you want to participate in the development of the SUSI server you can start by learning to deploy it on a cloud system like Google Cloud.

This way whenever you make a change to Susi Server, you can test it out on various Susi Apps instantly.

Google Cloud with Kubernetes provide this ability. Let’s dig deep into what is Google Cloud Platform and Kubernetes.

What is Google Cloud Platform ?

Google Cloud Platform lets you build and host applications and websites, store data, and analyze data on Google’s scalable infrastructure.
Google Cloud Platform (at the time of writing this article) also provides free credits worth $300 for 1 year for testing out the Platform and test your applications.

What is Kubernetes ?

Kubernetes is an open-source system for automatic deployment, management and scaling of containerized applications. It makes it easy to roll out updates to your application with simple commands from your development machine and scale horizontally easily by adding more clusters as demand increase.

Deploying Susi Server on Kubernetes

Deploying Susi Server on Kubernetes is a fairly easy task. Follow up the steps to get it running.

Create a Google Cloud Account

Sign up for a Google Cloud Account (https://cloud.google.com/free-trial/) and get 300$ credits for initial use.

Create a New Project

After successful sign up, create a new project on Google Cloud Console.
Let’s name it Susi-Kubernetes . 

You will be provided a ProjectID. Remember it for further reference.

Install Google Cloud SDK and kubectl

Go to https://cloud.google.com/sdk/ and see instructions to setup Google Cloud SDK on your respective OS.

After Google Cloud SDK install, run

gcloud components install kubectl

This will install kubectl for interacting with Kubernetes.

Login and setup project

  1. Login to your Google Cloud Account using
$ gcloud auth login

2. List all the projects using

$ gcloud config list project
[core]
project = <PROJECT_ID>

3. Select your project

$ gcloud config set project <PROJECT_ID>

4. Install JDK8 for susi_server setup and set it as default.

5. Clone your fork of the Susi Server Repository

$ git clone https://github.com/<your_username>/susi_server.git
$ cd susi_server/

6. Build project and run Susi Server locally

$ ./gradlew build
$ bin/start.sh

Susi server must have been started started and web interface is accessible on http://localhost:4000

Install Docker and build Docker image for Susi

  1. Install Docker.
    Debian and derivatives:  sudo apt install docker
    Arch Linux:   sudo pacman -S docker 
  2. Build Docker Image for Susi
    $ docker build -t gcr.io/<Project_id>/susi:v1 .
  3. Push Image to Google Container Registry private to your project.
$ gcloud docker -- push gcr.io/<Project_id>/susi:v1

Create Cluster and Deploy your Susi Server there

  1. Create Cluster. You may specify different zone, number of nodes and machine type depending upon requirement.
    $ gcloud container clusters create <Cluster-Name> --num-nodes 2 --machine-type n1-standard-1 --zone us-central1-c
  2. Run your deployment. You may specify any name for deployment.
    $ kubectl run <deployment_name> --image=gcr.io/<Project_id>/susi:v1 --port=80
    $ kubectl get deployments
    $ kubectl expose deployment susi --type=LoadBalancer
  3. Check your deployment and get Public IP for Access.
    $ kubectl get services
    NAME         CLUSTER-IP     EXTERNAL-IP     PORT(S)       AGE
    kubernetes   10.3.240.1     <none>          443/TCP        1d
    susi         10.3.241.145   <PUBLIC_IP>     80:31155/TCP   1d
  4. Go to provided public IP to check, if Susi Server is running.

Congratulations, you successfully setup Susi Server on Google Cloud with Kubernetes.

Updating the deployment

Next step is to update deployment when you wish to roll out changes. To do so.

Build Docker Image and Push it to Google Container Registry

$ docker build -t gcr.io/<Project_Id>/susi:v2 .
$ gcloud docker -- push gcr.io/<Project_Id>/susi:v2

Update Deployment Image with Kubernetes

$ kubectl set image deployment/<Deployment_Name> \
  <Deployment_Name>=gcr.io/<Project_id>/susi:v2
deployment "<Deployment_Name>" image updated

Go to public ip to see the changes.

That’s it. Now, you have fully running Susi Server on your own Google Cloud Cluster using Kubernetes.

Continue ReadingDeploying Susi Server on Google Cloud with Kubernetes

Susi AI Skill Development

What is Susi?

Susi is an open source intelligent personal assistant which has the capability to learn and respond better to queries. It is also capable of making to-do lists, setting alarms, providing weather and traffic info all in real time. Susi responds based on skills.

What is a skill? How do we teach a skill?

A skill is a piece of code which performs a set of actions in order to respond to the user’s query. These skills are based on pattern matching which help them mapping the user’s query to a specific skill and responding accordingly. Teaching a skill to Susi is surprisingly very easy to implement. One can take a look at the Susi Skill Development Tutorial and a video workshop by Michael Christen.

I will try to give a basic idea on how to create a skill, it’s basic structure and some of the skills I developed in the first week.

Prepare to create a skill:

  • Head over to http://dream.susi.ai
  • Create a etherpad with some relevant name
  • Delete all text currently present in there
  • Start writing your skill

Adding to this, for testing a skill one can head over to Susi Web Chat Interface.

Basic Structure for calling an API:

<Regular expression to be matched here>

!console:<response given to the user>
 {
 "url":"<API endpoint>",
 "path":"<Json path here>"
 }
 eol

So, let me explain this line by line.

  1. The regular expression is the one to which the user’s query is matched first.
  2. The console is meant to output the actual response the user sees as response.
  3. In place of the “url”, the API endpoint is passed in.
  4. “path” here specifies how we traverse through the response Json or Jsonp to get the object, starts with “$.”.
  5. At last, “eol” which is the end-of-line marks the end of a skill.

Let’s take an example for better understanding of this:

random gif
!console: $url$
{
    "url" : "http://api.giphy.com/v1/gifs/trending?api_key=dc6zaTOxFJmzC",
    "path" : "$.data[0].images.fixed_height"
}
eol 

 

This skill responds with a link to a random gif.

Steps involved:

  1. Match the string “random gif” with the user’s query.
  2. On successful match, make an API call to the API endpoint specified in “url”
  3. On response, extract the object at the specified path in the json under “path”
  4. Respond to the user with the “url” key’s value which would here be an URL of a GIF.

Let’s try it out on Susi Web Chat. For this, you will first have to load your skill using the dream command followed by etherpad name: dream <etherpad name>. And then you can start testing your skill.

So, we queried “random gif” and we got a response “Click Here!”. The complete URL didn’t show up because all the URLs are currently parsed and a hyperlink for each is created. So try clicking on it to find a GIF.

 

Now, let’s look at one more skill I developed during this period.

# Returns the name of the president of a country

 president of *|who is the president of *| president *
 !console:$plaintext$
 {      "url":"https://api.wolframalpha.com/v2/query?input=president+$1$&output=JSON&appid=9WA6XR-26EWTGEVTE&includepodid=Result",
   "path" : "$.queryresult.pods[0].subpods[0]"
 }
 eol

 

Let’s understand this step by step:

  1. We have here “president of *|who is the president of *| president *”, which means the user’s query matches with anyone of the following because of the use of pipe symbol “|”. The “*” here replaces a word or a list of words, which can be accessed like: “${index}$”  where index is replaced by the position of the “*” in the expression starting from 1.
  2. Now we have something new in the URL. See that  $1$  inside the URL? On runtime, that is replaced with the content of the “*” variable. So if a user puts in query like: “president of usa”, “usa” is mapped to $1$ and is replaced in the URL and appropriate API request is made.
  3. Then the path is traversed in the json response and the value of the “plaintext” key is used to respond to the user.

 

It’s now time to try it out on Susi Web Chat.

So, we got our desired response here, i.e., the name of the president of usa.

Continue ReadingSusi AI Skill Development

Using NodeBuilder to instantiate node based Elasticsearch client and Visualising data

As elastic.io mentions, Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. But in many setups, it is not possible to manually install an Elasticsearch node on a machine. To handle these type of scenarios, Elasticsearch provides the NodeBuilder module, which can be used to spawn Elasticsearch node programmatically. Let’s see how.

Getting Dependencies

In order to get the ES Java API, we need to add the following line to dependencies.

compile group: 'org.elasticsearch', name: 'securesm', version: '1.0'

The required packages will be fetched the next time we gradle build.

Configuring Settings

In the Elasticsearch Java API, Settings are used to configure the node(s). To create a node, we first need to define its properties.

Settings.Builder settings = new Settings.Builder();

settings.put("cluster.name", "cluster_name");  // The name of the cluster

// Configuring HTTP details
settings.put("http.enabled", "true");
settings.put("http.cors.enabled", "true");
settings.put("http.cors.allow-origin", "https?:\/\/localhost(:[0-9]+)?/");  // Allow requests from localhost
settings.put("http.port", "9200");

// Configuring TCP and host
settings.put("transport.tcp.port", "9300");
settings.put("network.host", "localhost");

// Configuring node details
settings.put("node.data", "true");
settings.put("node.master", "true");

// Configuring index
settings.put("index.number_of_shards", "8");
settings.put("index.number_of_replicas", "2");
settings.put("index.refresh_interval", "10s");
settings.put("index.max_result_window", "10000");

// Defining paths
settings.put("path.conf", "/path/to/conf/");
settings.put("path.data", "/path/to/data/");
settings.put("path.home", "/path/to/data/");

settings.build();  // Buid with the assigned configurations

There are many more settings that can be tuned in order to get desired node configuration.

Building the Node and Getting Clients

The Java API makes it very simple to launch an Elasticsearch node. This example will make use of settings that we just built.

Node elasticsearchNode = NodeBuilder.nodeBuilder().local(false).settings(settings).node();

A piece of cake. Isn’t it? Let’s get a client now, on which we can execute our queries.

Client elasticsearhClient = elasticsearchNode.client();

Shutting Down the Node

elasticsearchNode.close();

A nice implementation of using the module can be seen at ElasticsearchClient.java in the loklak project. It uses the settings from a configuration file and builds the node using it.


Visualisation using elasticsearch-head

So by now, we have an Elasticsearch client which is capable of doing all sorts of operations on the node. But how do we visualise the data that is being stored? Writing code and running it every time to check results is a lengthy thing to do and significantly slows down development/debugging cycle.

To overcome this, we have a web frontend called elasticsearch-head which lets us execute Elasticsearch queries and monitor the cluster.

To run elasticsearch-head, we first need to have grunt-cli installed –

$ sudo npm install -g grunt-cli

Next, we will clone the repository using git and install dependencies –

$ git clone git://github.com/mobz/elasticsearch-head.git
$ cd elasticsearch-head
$ npm install

Next, we simply need to run the server and go to indicated address on a web browser –

$ grunt server

At the top, enter the location at which elasticsearch-head can interact with the cluster and Connect.

Upon connecting, the dashboard appears telling about the status of cluster –

The dashboard shown above is from the loklak project (will talk more about it).

There are 5 major sections in the UI –
1. Overview: The above screenshot, gives details about the indices and shards of the cluster.
2. Index: Gives an overview of all the indices. Also allows to add new from the UI.
3. Browser: Gives a browser window for all the documents in the cluster. It looks something like this –

The left pane allows us to set the filter (index, type and field). The table listed is sortable. But we don’t always get what we are looking for manually. So, we have the following two sections.
4. Structured Query: Gives a dead simple UI that can be used to make a well structured request to Elasticsearch. This is what we need to search for to get Tweets from @gsoc that are indexed –

5. Any Request: Gives an advance console that allows executing any query allowable by Elasticsearch API.

A little about the loklak project and Elasticsearch

loklak is a server application which is able to collect messages from various sources, including twitter. The server contains a search index and a peer-to-peer index sharing interface. All messages are stored in an elasticsearch index.

Source: github/loklak/loklak_server

The project uses Elasticsearch to index all the data that it collects. It uses NodeBuilder to create Elasticsearch node and process the index. It is flexible enough to join an existing cluster instead of creating a new one, just by changing the configuration file.

Conclusion

This blog post tries to explain how NodeBuilder can be used to create Elasticsearch nodes and how they can be configured using Elasticsearch Settings.

It also demonstrates the installation and basic usage of elasticsearch-head, which is a great library to visualise and check queries against an Elasticsearch cluster.

The official Elasticsearch documentation is a good source of reference for its Java API and all other aspects.

Continue ReadingUsing NodeBuilder to instantiate node based Elasticsearch client and Visualising data

Software Testing

In this post I would like to explain what is software testing and different methods, levels of software testing.

What is software testing?

Software Testing is the process of evaluating a system or its components with the intent to find whether it satisfies the specified requirements or not.

Testing is executing a system in order to identify any gaps, errors, or missing requirements in contrary to the actual requirements.

What are the Levels of software testing?

Software products are tested at four levels:
  • Unit testing
  • Integration testing
  • System testing
  • Acceptance testing

Unit testing

  • During unit testing, modules are tested in isolation. If all modules were to be tested together, it may not be easy to determine which module has the error.
  • Unit testing reduces debugging effort several folds. Programmers carry out unit testing immediately after they complete the coding of a module.

Role of Unit Testing

  • Assure minimum quality of units before integration into system
  • Focus attention on relatively small units
  • Testing forces us to read our own code – spend more time reading than writing
  • Automated tests support maintainability and extendibility

Integration testing

After different modules of a system have been coded and unit tested:
  • modules are integrated in steps according to an integration plan
  • partially integrated system is tested at each integration step.

Role of Integration Testing

  • Gain confidence in the integrity of overall system design
  • Ensure proper interaction of components

Integration Testing Strategies

  • Big-bang
  • Top-down
  • Bottom-up
  • Critical-first
  • Function-at-a-time
  • As-delivered
  • Sandwich

System Testing

  • Gain confidence in the integrity of the system as a whole
  • Ensure compliance with functional requirements
  • Ensure compliance with performance requirements

Acceptance Testing

  • Testing performed by the customer or end-user himself
  • to determine whether the system should be accepted or reject

Testing is very important step of software development. We have used unit testing for our project Engelsystem. We are developing new features. Interested Developers can work with us.

Development: https://github.com/fossasia/engelsystem

Issues/Bugs:https://github.com/fossasia/engelsystem/issues

Continue ReadingSoftware Testing

AngularJS structure and directives

Today I am going to tell you how to build a well worked Angular JS structure, as well as to be able to maintain your app’s elements correctly. It’s a basic and most important thing to learn and remember before you start building your own website or app because in the early phases of a project the structure doesn’t matter too much, and many people tend to ignore it, but in the long term it will affect code maintainability. It also helps you to develop it quicker and solve bugs without any additional troubles.

I must admit that I was one of them who  had made this mistake and ignore AngularJS correct structure. The thing is that I had not known how to build Angular JS app correctly before I started coding CommonsNet, and I wanted to start and provide a real outcome as quickly as possible. But unfortunately my mistake has stoped me later. I have realised it this week while implementing a quite simple feature – adding HTML elements dynamically. It turned out that I had a trouble while adding them. I had’t known that it’s better not to manipulate DOM elements in Controllers, but instead do it in directives.

Let me to explain it step by step from lessons I’ve learned while refactoring CommonsNet code structure. Please note that I’m just sharing my experience in building rather a small project and I’m covering only a small part of that subject, so some of these tips may be first – useless if you want to build an extended one, and then – not enough if you want to learn more. I recommend you to find further resources.

AngularJS Structure

First of all please see at AngularJS structure provided below. It’s a very basic structure, but it’ is important to have in mind that you should follow that pattern while developing your own app. If you want to have an order in your files you should seperate them by directories and create different ones like: controllers, directives, services, js and views. This structure makes it very easy for the reader to visualize and conceptualize the concepts you are covering.

AngularJS structure.png

My CommonsNet structure differs a bit because I have one main js directory, and all subdirectories like directives, controllers and services  inside it. I think it’s also an acceptable solution. Imagine that I hadn’t had any directives or services directives’ files before last Wednesday…

Then,  it’s also recommended to put index.html at the root of front-end structure. The index.html file will primarily handle loading in all the libraries and Angular elements.

Next, controllers are main part of AngularJS apps. It always happens that developers when starting out tend to put too much logic in the controllers. It’s a bug. Controllers should never do DOM manipulation or hold DOM selectors, that’s where we use directives and ng-model. Likewise business logic should live in services, not controllers.

Data should also be stored in services, except where it is being bound to the $scope. Services are singletons that persist throughout the lifetime of the application, while controllers are transient between application states. So, if the controller is a coordinator between the view and the model, then the amount of the logic it has should be minimal.

Directives

Take a look at a very basic AngularJS directive. Let’s name this file appInfo.js as well as create an .html file appInfo.html as in example below.

Learn AngularJS 1.X   Codecademy.png

In this example we create a new directive called appInfo. It returns an object with three options – restrict, scope and templateURL

  1. restrict specifies how the directive will be used in the view. The 'E' means it will be used as a new HTML element.
  2. scope specifies that we will pass information into this directive through an attribute named info. The = tells the directive to look for an attribute named info in the <app-info> element, like this: The data in info becomes available to use in the template given by templateURL.
  3. templateUrl specifies the HTML to use in order to display the data in scope.info. Here we use the HTML in js/directives/appInfo.html.

Then we need to build a structure to directive in our appInfo.html

<h2 class="title">{{ info.title }}</h2> 
<p class="developer">{{ info.developer }}</p> 
<p class="price">{{ info.price | currency }}</p>

Of course <h2> and <p> are just examples. You can use here what you only want. The title and developer are controller $scope.elements. Let’s take a look at controller’s content.

$scope.apps = [ 
 { 
 title: 'MOVE', 
 developer: 'MOVE, Inc.', 
 price: 0.99 
 }, 
 { 
 title: 'Shutterbugg', 
 developer: 'Chico Dusty', 
 price: 2.99 
 }, 
]

In index.html we use only  ng-repeat to loop through app element in $scope.apps, and then app-info info=”app”

<div ng-repeat="app in apps">
     <app-info info="app"></app-info>
</div>

That’s it. It’s quite a simple example, but it helps me to refactor my code in CommonsNet project and avoid a long, boring, repetitive work while adding new elements to website. It also helps me to implement a feature I have mentioned above. I’am going to tell more about it in next blog post. Stay tuned!

Resources:

https://scotch.io/tutorials/angularjs-best-practices-directory-structure

https://www.codecademy.com/learn/learn-angularjs.

https://commonsnetblog.wordpress.com/

 

Continue ReadingAngularJS structure and directives

Working with Apache Web Server

In this article, I will examine what apache is how to setup apache for ubuntu or debian and how to view error logs for apache and how to understand what the logs contain which will be useful for web development.

What is Apache?

Apache is the most popular web server on the internet. It is used to serve more than half of all active websites.

It is fast, reliable, and secure. It can be highly customized to meet the needs of many different environments by using extensions and modules.

Installing Apache on Ubuntu and Debian

If you do not already have Apache installed, you can do so now by issuing the following commands:

sudo apt-get update
sudo apt-get install apache2

If you visit your VPS’s IP address in a web browser, you will get the default Apache index page. We get the following output.

your_domain_name_or_ip_address
It works!
This is the default web page for this server.
The web server software is running but no content has been added, yet.

Looking into apache error.log files

In order to effectively manage a web server, it is necessary to get feedback about the activity and performance of the server as well as any problems that may be occuring. The Apache HTTP Server provides very comprehensive and flexible logging capabilities.

Error.log file

The server error log, whose name and location is set by the ErrorLog directive, is the most important log file. This is the place where Apache httpd will send diagnostic information and record any errors that it encounters in processing requests.

It is the first place to look when a problem occurs with starting the server or with the operation of the server, since it will often contain details of what went wrong and how to fix it.

Explanation of log message

There is certain information that is contained in most error log entries. For example, here is a typical message.

[Sat Aug 20 08:09:38.958885 2016] [:error] [pid 1489] [client 127.0.0.1:40810] PHP Notice:  Constant MIN_PASSWORD_LENGTH already defined in /var/www/html/test/engelsystem/config/config.php on line 30, referer: http://localhost/

[Sat Aug 20 08:09:38.958885 2016] The first item in the log entry is the date and time of the message.

[:error] The second entry lists the severity of the error being reported.

[pid 1489] Indicates process Id

[client 127.0.0.1:40810] The fourth entry gives the IP address of the client that generated the error.

Fifth one is error message explaining the error.

Constant MIN_PASSWORD_LENGTH already defined.

File and line number

 /var/www/html/test/engelsystem/config/config.php on line 30

Error logs are easy to understand. Using error logs we can easily figure out the problem in your server or your code and fix it.

For more information about the log files and configuration please visit Apache website Apache provides excellent documentation.

Apache log files were useful for me during web development project Engelsystem. We are developing new features. Interested developers can work with us.

Development: https://github.com/fossasia/engelsystem

Issues/Bugs:https://github.com/fossasia/engelsystem/issues

Continue ReadingWorking with Apache Web Server

Importing database with PHP script

In this post I would like to discuss how to import tables to database directly through PHP script.

For our Project Engelsystem we need to import the tables manually by the command line or PHPMYADMIN. Now the user need not worry about the importing table. They are directly imported through the script.

Initially we used to import the tables in a sql file by source command or mysql command.

$ mysql -u root -p

CREATE DATABASE engelsystem;

use engelsystem;

source db/install.sql;

source db/update.sql;

Script to import tables directly through PHP.

function import_tables() {
// get the database variables. 
global $DB_HOST, $DB_PASSWORD, $DB_NAME, $DB_USER;
// file names to import
 $import_install = '../db/install.sql';
 $import_update = '../db/update.sql';
 // command to import both the files.
 $command_install = 'mysql -h' .$DB_HOST .' -u' .$DB_USER .' -p' .$DB_PASSWORD .' ' .$DB_NAME .' < ' .$import_install;
 $command_update = 'mysql -h' .$DB_HOST .' -u' .$DB_USER .' -p' .$DB_PASSWORD .' ' .$DB_NAME .' < ' .$import_update;
 $output = array();
 // execute the command
 exec($command_install, $output, $worked_install);
 exec($command_update, $output, $worked_update);

// test whether they are imported successfully or not
switch ($worked_install && $worked_update) {
 case 0:
 return true;
 case 1:
 return false;
 }
}

Once we execute the above script the tables will be imported automatically. User need not import the tables manually.

In this way we can import tables using script. For more information visit about project please visit here.

Development: https://github.com/fossasia/engelsystem

Issues/Bugs:https://github.com/fossasia/engelsystem/issues

Continue ReadingImporting database with PHP script

CommonsNet’s main features are ready

Main CommonsNet features are generating a file and  generating a code script. Today I would like to provide you with a simple and short guideline how I have managed to implement them.

Generate a file

Generating a file is the most important feature of our website, and it is not only a condition of its existance but also a basis to find stakeholders – in general people who want to use CommonsNet webiste.  It is the most complicated CommonsNet feature as well even if it is not so difficult as you may think.

To implement this feature a pure JavaScript is used and for now we have chosen the simplest solution to build it, mainly because we need to have a working webiste now to start a real promotion and find many customers.

Important steps here are:

  1. Manually prepare a .fodt file
  2. Get .fodt file and modify it
  3. Generate a pdf file from a ready .fodt file
  4. Write function to download a pdf file.

 

Preparing .fodt file

My mentor Andre Rebentisch has recommended me to use .fodt file because it’s very user-friendly and let you to prepare a file’s structure very easy. Actually that’s it! I have prepared manually in Libre Office a default doc file, and then save it in .fodt extension. .Fodt is a great option because it behaves as .xml file and let you modify this file using any programming language.

So, as I have already mentioned above I have created a simple doc file and prepare the structure. It is very clear and looks like a default Libre Office file. Only thing is that it has .fodt extension which means that you can open it in a code editior like for e.g. Sublime and start modifying easily.

If you open your .fodt file in a Sublime, it looks like this. Isn’t it simply? And above all it helps you to easily manipulate your data using for e.g a JavaScript.

Screen Shot 2016-08-13 at 17.19.06

Screen Shot 2016-08-13 at 17.18.00Get .fodt file to manipulate it

In my AngularJS Wizard Controller I write a function to get data from my .fodt file

$http({
 method: 'GET',
 url: 'http://commonsnet.herokuapp.com/generatefile.fodt',
 }).success(function(data){}) .error(function(){
 alert("error");
 });

It’s very easy code sample and you can find it in many Internet resources. I hope it is understandable to you as well. I have written a function which get a .fodt file and return data. Inside .success I have written my whole code to manipulate a .fodt file.  To use it I have decided to use a simple JavaScript replace method and if-else conditions, and in dependence on user’s inputs in a wizard form create different file’s structures. Let me show you a small section :

result = result.replace("NETWORK_NAME", "The owner provides" + " " + vm.ssid + "network connection")

if ((vm.password !== "") && (typeof vm.password !== "undefined")) {
 result = result.replace("INPUT_PASSWORD", "The owner informs that password is" + " " + vm.password);

}

Generate a pdf file from a ready .fodt file

It is still in progress! I will update it as soon as I manage to successfully implement it!

Write function to save it

To save my ready file I have used save function() and put in my AngularJS Wizard Controller.

$scope.save = function() {}

Generate a code script

We have decided to implement a generating code feature, because we want our users to be able to simply fill a wizard form with all Wifi details, then copy  a generated code  and be able to simply  paste it to their website.

We believe it is a very convenient solution to them because all they need to do it is to go through a form and it is ready. Besides, we have already decided to get rid of a database and use only a JSON file to not complicate our website’s working. So, that’s also the reason why we want to make it as simple as possible.

That’s why I have decided to create a link snd store there all wizard form’s values Undoubtedly it causes to build a very long link, but in the same time I believe that it is a good solution if we don’t use a database. I have also already found some ways to shorten and encrypt the generated link.

In my WizardController AngularJS I have write a function with a condtion

$scope.gotoStep = function(newStep) {
 currentStep = newStep;
 if ( currentStep === 3) {
 var link = "commonsnet.herokuapp.com/#/file?ssid=" + vm.ssid + "&password=" + vm.password + "&security=" + vm.securitytypes + "&standard=" + vm.wifistandards + "&payment=" + vm.paymentfieldyes + "&fee=" + vm.paymentfield + "&timelimit=" + vm.timelimityes + "&limit=" + vm.timelimitfield + "&service=" + vm.serviceyes + "&specialdevices=" + vm.specialdevices + "&devices=" + vm.specialdevicesfield + "&specialsettings=" + vm.specialsettings + "&settings=" + vm.specialsettingsfield + "&acceptterms=" + vm.acceptterms + "&liking=" + vm.socialprofile + "&downloading=" + vm.downloading + "&restrictions=" + vm.country + "&country=" + vm.countries + "&law=" + vm.legalrestrictions
 vm.code = '<a href="' + link + '">CommonsNet</a>'
 }
 }

If my currentStep === 3 (actually it’s a  confirmation.html templateUrl) I create a link (a code) which is available to users to copy and paste to their website.

Then I have also created a new FileController (FileCtrl) which is called when users enter a commonsnet.herokuapp.com/#/file (if they paste in an url a copied generated link)

app.controller('FileCtrl',['$scope', '$routeParams', function ($scope, $routeParams) { $scope.ssid = $routeParams.ssid; $scope.password = $routeParams.password; }])

Finally I have created a .html structure to post data from a wizard from which are all stored in our link. It’s a normal html structure but in order to adjust the html’s content to user’s wizard inputs I have used an Angular ng-if . A little example looks like this:

ng-if="ssid && ssid != 'undefined'

And that’s it. It still needs some improvements and fixing bugs but main features work according to my plan!!!

It’ time to intensively extend the idea of CommonsNet and find stakeholders! Are you interested too? I am sure you are, because CommonsNet is a great tool which helps us to build a transparent Wifi networks!

Feel free to write your comments and suggestion!

Continue ReadingCommonsNet’s main features are ready