Tutorial

Read more about the article Debugging Using Stetho in Open Event Android

Debugging Using Stetho in Open Event Android

The Open Event Android project helps event organizers to generator Apps (apk format) for their events/conferences by providing api end point or zip generated using Open Event server. In this android app data is fetched from the internet using network calls to the Open Event server and the data of the event is stored in a database. It is difficult to debug an app with so many network calls and database connectivity. A way to approach this is using  Stetho which is very helpful tool for debugging an app which deals with network calls and database.  

Stetho is Facebook’s open source project works as debug bridge for android applications which gives powerful Chrome Developers Tools for debugging android applications using Chrome desktop browser.

What can you do using stetho ?

  • Analyze network traffic
  • Inspect elements(layouts/views)  
  • View SQLite database in table format
  • Run queries on SQLite database
  • View shared preference and edit it
  • Manipulate android app from command line

Setup

1. Add Gradle dependency

To add stetho in your app add ‘com.facebook.stetho:stetho:1.5.0’ dependency in your app  module’s build.gradle file. This dependency is strictly required.

dependencies{
    compile 'com.facebook.stetho:stetho:1.5.0'
}

For network inspection add one of the following dependency according to which you will be using

'com.facebook.stetho:stetho-okhttp:1.5.0'
'com.facebook.stetho:stetho-okhttp3:1.5.0'
'com.facebook.stetho:stetho-urlconnection:1.5.0'

2. Initialize stetho

Initialize stetho in class MyApplication which extends Application class by overriding  onCreate() method. Make sure you have added MyAppication in manifest.

public class MyApplication extends Application {
    public void onCreate() {
        super.onCreate()
        Stetho.initializeWithDefaults(this);
    }
}

Stetho.initializeWithDefaults(this) initializes stetho with defaults configuration(without network inspection and more). It will be able to debug database.

Manifest file

<Application   android:name=”.MyApplication”    …   />

For enabling network inspection add StethoInterceptor  in OkHttpClient

new OkHttpClient.Builder()
    .addNetworkInterceptor(new StethoInterceptor())
    .build();

Using Chrome Developer Tools

1. Open Inspect window

Run stetho initialized app on device or emulator then start Google chrome and type chrome://inspect. You will see you device with your app’s package name. Click on “inspect”2. Network Inspection

Go to Network tab. It will show all network calls. With almost all info like url(path), method, returned status code, returned data type, size, time taken etc.

You can also see preview of image and preview, response of returned json data by clicking on Name.

3. SQLite Database

Go to “Resources”  tab and select “Web SQL”. You will see database file(.db). By clicking on database file you will see all tables in that database file. And by clicking on table name you will see data in row-column format for that table.

4. Run queries on SQLite database

Same as above go to “Resources”  tab and select “Web SQL”. You will see database file(.db). By clicking on database file you will see console on right side, where you can run queries on SQLite database. Example,

SELECT * FROM tracks ;

5. Shared Preferences Inspection

Go to “Resources”  tab and select “Local Storage”. You will show all files that your app used to save key-value pairs in shared preference and by clicking on file you will see all key-value pairs.

6. Element(View/Layout) Inspection

Go to “Elements” tab. You will see top layout/view in view hierarchy. By clicking it you will see child layout/view of that layout/view. On hover on layout/view you view will be inspected in your device/emulator.

 

In this blog the most important have been put forward, but there are still  some nice stuff available,like:

  • An integration with JavaScript Console : Enables JavaScript code execution that can interact with the application
  • Dumpapp  : It allows an integration higher than the Developer Tools, enabling the development of custom plugins.

By now, you must have realized that stetho can significantly improve your debugging experience. To learn more about stetho, refer to

Continue ReadingDebugging Using Stetho in Open Event Android

Implementing a chatbot using the SUSI.AI API

SUSI AI is an intelligent Open Source personal assistant. It is a server application which is able to interact with humans as a personal assistant. The first step in implementing a bot using SUSI AI is to specify the pathway for query response from SUSI AI server.

The steps mentioned below provide a step-by-step guide to establish communication with SUSI AI server:

    1. Given below is HTML code that demonstrates how to connect with SUSI API through an AJAX call. To put this file on a Node Js server, see Step 2.  To view the response of this call, follow Step 4.

      <!DOCTYPE html>
<body>
<h1>My Header</h1>
<p>My paragraph.</p>
//Script with source here 
//Script to be written here 
</body>
</html>

      In above code add scripts given below and end each script with closing tag </script>. In the second script we are calling SUSI API with hello query and showing data that we are receiving through call on console.

      <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js">
    2. <script>
$(function (){ 
    $.ajax({ 
        dataType: 'jsonp', 
        type:'GET', url: 'http://api.susi.ai/susi/chat.json? timezoneOffset=-300&q=hello', 
         success: function(data){ 
             console.log('success', data); 
         } 
    }); 
});
    3. Code below is in node js to setup localhost and getting the same above result on browser. Below is Node Js code to setup a server at localhost for the above created HTML file.

      var http = require('http');
 var fs = require('fs');
 http.createServer(function (req, res) {
  fs.readFile('YOURFILENAME.html', function(err, data) {
    res.writeHead(200, {'Content-Type': 'text/html'});
    res.write(data);
    res.end();
  });
 }).listen(9000);
    4. We will get following response by running this Node js code and checking results on http://localhost:9000/ To run this code install Node Js and write “node filename.js” in command line.
    5. You can open above window by right clicking on page and selecting Inspect. Go to the Network option and select the relevant api call from left section of Inspect window.

We have successfully got response from SUSI API and now we can use this response for building bots for receiving replies for user.

Continue ReadingImplementing a chatbot using the SUSI.AI API

Recyclerview in Open Event Android: A great upgradation to ListView

Recently, I was fixing a trivial bug in the Open Event Android app, an interesting thing caught my attention. Most of the lists in the app where actually not implemented using the traditional ListView, but instead used something called RecyclerView. The beautiful transitions that make this app the beauty it is, is due to the amazing and easy to code Recycler Views used in it.

A screen shot showcasing the simplicity of the FOSSASIA Open event android app.

 

List View is one among the various views that Android provides. In a nutshell, it is literally a group of generic views in Android. Why then use an entire new and more complex view, if all that it offers is a number of views displayed together? Well, it certainly has certain add-ons to the most generic set of views like Image and TextViews in Android.

I encourage you to think about what different could the ListView be as compared to multiple declarations of TextView. The thing with multiple declarations of a Text or Image View is that they aren’t “scrollable”, that is, if you want numerous Views in a single activity, without List View, you will have to fit them on one screen, else make another activity. So, if you want a scrollable view with same basic view items, List View is the way to go about it in Android.

The next question that may arise is:

How does a Listview work?

Google defines Adapters as “a bridge between an AdapterView and the underlying data for that view”. List Views use Adapters that retrieve data from Arrays, Lists and other data structures and queries, plugs them into our List View and thus, creates a beautiful looking List in no time. In Spite of all this, List View certainly lags in various fields that deal with performance, memory and compactness issues. As a great improvement Recycler View was introduced.

Using Recyclerview you are not restricted to vertical scrollable views. It has the ability to provide horizontal, vertical, grid and staggered layouts among others, with the help of different layout managers:-

recyclerView.setLayoutManager(new StaggeredGridLayoutManager(2,StaggeredGridLayoutManager.VERTICAL));
recyclerView.setLayoutManager(new LinearLayoutManager(RecyclerView_Activity.this,LinearLayoutManager.HORIZONTAL, false));

RecyclerView.ItemDecoration class allows us to add special drawings to our view. Thus, we can add dividers, borders and much more using this class. 

public class DividerItemDecoration extends RecyclerView.ItemDecoration {
 
     private Drawable mDivider;
 
     public DividerItemDecoration(Drawable divider) {
         mDivider = divider;
     }

Adding animation used to be a devil of a task in Listview days, however with the release of Recyclerview, it has become a lot more easy. RecyclerView.ItemAnimator class helps in animating items while the adapters are being changed.

Next, we have something related to the performance:-

In a List View, you might have come across the term, ‘Viewholder’, but if you haven’t, that’s okay too. However, in Recyclerview, using View Holders have been made mandatory and for a good reason. The class goes by the name RecyclerView.ViewHolder.

What exactly are View Holders?
The clean dividers and minimal design in the Recyclerview makes up for a subtle UI.

In a broader sense, it is used to provide information about the identity of an itemView and it’s position within a Recyclerview. Without View Holders we have an overhead of calling the function findViewById() for every item in our Recyclerview! This upgradation from the typical findViewById() to View Holders awards us with better and more smooth scrolling through the list.

How does Recyclerview work?

List View is an ancestor to Recyclerview. Recyclerview has everything that List View offers to us (except for a few things like setchoicemode(), which I might take up in my upcoming blog posts), and much more.

A great achievement that Android achieved through Recyclerview has been it’s ability to “recycle” views on the way. In laymen terms, if you scroll through a Recyclerview, the views that go away from the screen during scrolling are the same views that come into the screen. The only difference is that the data in the views leaving the screen is changed to the data of the views entering your screen and so they are presented as new views. In this way, a Recyclerview helps saving memory for sure as compared to List Views.

There are still a lot of other things to learn about Recycler and List Views and you find more resources here.

 

Continue ReadingRecyclerview in Open Event Android: A great upgradation to ListView

Build a simple Angular 4 message publisher

Introduction

The differences between Angular 1, Angular 2 and Angular 4

AngularJS/Angular 1 is a very popular framework which is based on MVC model, and it was released in October, 2010.
Angular 2, also called Angular, is based on component and is completely different from Angular 1 , which was released in September 2016.
Angular 4 is simply an update of Angular 2, which is released in March, 2017.

Note that Angular 3 was skipped due to the version number conflicts.

Installation

Prerequisites

1. Install TypeScript:

npm install -g typescript

2. Install code editor: vscode, sublime, intellij.

3. Use Google chrome and development tool to debug.

Install Angular CLI by command line

Angular CLI is a very useful tool which contains a brunch of commands to work with Angular 4. In order to install Angular CLI, the version of Node should be higher or equal to 6.9.0, and NPM need to higher or equal to 3.

npm install -g @angular/cli

Create a new Project

ng new loklak-message-publisher
cd loklak-message-publisher
ng serve

The command will launch web browser automatically in localhost:4200 to run the application. It will detect any change in files. Of course you can change the default port to others.
The basic project structure is like this:

package.json: standard node configuration file, which including the name, version and dependency of the configuration.

tsconfig.json: configuration file for TypeScript compiler.

typings.json: another configuration file for typescript, mainly used for type checking.

app folder contains the main ts file, boot.ts and app.component.ts.

 

Next we will show how to build a simple message publisher. The final application looks like this, we can post new tweets in the timeline, and remove any tweet by clicking ‘x’ button:

  • The first step in to create new elements in app.component.html file.

The HTML file look quite simple, we save the file and switch to browser, the application is like this:

A very simple interface! In next step we need to add some style for the elements. Edit the app.component.css file:

.block {
   width: 800px;
   border: 1px solid #E8F5FD;
   border-bottom: 0px;
   text-align: center;
   margin: 0 auto;
   padding: 50px;
   font-family: monospace;
   background-color: #F5F8FA;
   border-radius: 5px;
}

h1 {
   text-align: center;
   color: #0084B4;
   margin-top: 0px;
}

.post {
   width: 600px;
   height: 180px;
   text-align: center;
   margin: 0 auto;
   background-color: #E8F5FD;
   border-radius: 5px;
}

.addstory {
   width: 80%;
   height: 100px;
   font-size: 15px;
   margin-top: 20px;
   resize: none;
   -webkit-border-radius: 5px;
   -moz-border-radius: 5px;
   border-radius: 5px;
   border-color: #fff;
   outline: none;
}

.post button {
   height: 30px;
   width: 75px;
   margin-top: 5px;
   position: relative;
   left: 200px;
   background-color: #008CBA;
   border: none;
   color: white;
   display: inline-block;
   border-radius: 6px;
   outline: none;
}

.post button:active {
   border: 0.2em solid #fff;
   opacity: 0.6;
}

.stream-container {
   width: 600px;
   border-bottom: 1px solid #E8F5FD;
   background-color: white;
   padding: 10px 0px;
   margin: 0 auto;
   border-radius: 5px;
}

.tweet {
   border: 1px solid;
   border-color: #E8F5FD;
   border-radius: 3px;
}

p {
   text-align: left;
   text-indent: 25px;
   display: inline-block;
   width: 500px;
}

span {
   display: inline-block;
   width: 30px;
   height: 30px;
   font-size: 30px;
   cursor: pointer;
   -webkit-text-fill-color: #0084B4;
}

The app is almost finished, so excited! Now we want to add some default tweets that will be shown up in our application.
Let’s open the app.component.ts file, add an array in class AppComponent.

Then in the app.component.html file, we add ngFor to instantiates a tweets in the page.

After we finished this step, all the tweets are displayed in our application. Now we have to implement ‘tweet’ button and delete button. We define addTweet and removeTweet functions in app.component.ts file.

For the addTweet function, we need to modify html to define the behavior of textarea and tweet button:

For removeTweet function, we add id for each tweet we posted, and combine delete span with removeTweet function, which pass the id of tweet.

Conclusion

There are two things about Angular 4 we can learn from this tutorial:

  1. How to bind functions or variables to DOM elements, e.g. click function, ngModel.
  2. How to use the directives such as *ngFor to instantiate template.
Continue ReadingBuild a simple Angular 4 message publisher
Read more about the article Implementing Google Maps in Open Event ember Front-end

Implementing Google Maps in Open Event ember Front-end

In the deployment of the Open Event Front-end on eventyay.com we want to show an event’s location on a map using Google Maps. Since Eventyay.com requires Google map to be used at several web pages, we can use Google map as an ember component for reusability.

How did we do it? The following walks you through the entire process.

The first step is to: Install g-map addon-

  • ember g-map, which is an addon, is available on ember add-on library which can be found at emberobserver.com. So first import that library using ember cli command
ember install ember-g-map
  • This command adds an ember-g-map module in our node_modules and also updated package.json file of our app now our ember-g-map modules contains the following directories. The following three are required to be highlighted that serves our purpose-
  • Addon/ – this directory contains all the implementation logic which acts as a prototype for the g-map component
  • App/- this directory basically exports a copy of g-map from our Addon/ directory to the web page where it is required. The export includes the logic implement inside the Addon/ directory which can later be imported by the parent app for use.Package.json – this file holds various metadata relevant to the ember-g-map

Generate Component-

  • The next step is to generate a component event-map (we can name our component anything we want but be sure to add a hyphen between the name). The component can be generated with the help of the following command of ember cli-
ember g component public/event-map

This will add three files in our app –

event-map.js located at app/component/public/event-map.js

event-map.hbs located at app/template/component/public/event-map.hbs

event-map-test.js located at tests/integration/component/public/event-map-test.js

Define Component-

Now let us use these files to add a google map component in our app-

  • First of all, we have to tell index.hbs about our new component that we are going to add. So we add our component {{public/event-map}} in our index.hbs file which is located at app/templates/public/index.hbs And we fetch latitude, longitude and location name from model and store in the event object and pass it to our component so that this information is available in the component.
<div class="location">
<h1 id="getting-here">{{t 'Getting Here'}}</h1>
 {{public/event-map event=model.event}}
</div>
  • The event-map.js file will be used to add ui grid class to assign a class to the div of the component. This will alleviate the redundancy regarding the creation of an extra div to enclose the component and then giving it the ui grid class.
import Ember from 'ember';
 
const { Component } = Ember;
 
export default Component.extend({
 classNames: ['ui', 'stackable', 'grid']
});
  • In our event-map.hbs we have with us the latitude, longitude, and locationName available and we are good to use our g-map addon. So in our event-map.hbs we can directly add g-map and g-map-marker but make sure that we pass lat and lang attributes to it. We can use custom options like backgroundColor, draggable etc. but here we have used street view and gestureHandling. gestureHandling is set as “cooperative” so that for smaller screen device,  zooming is possible only with two fingers so that scrolling would become easy.
<div class="eight wide column event-map">
 {{#g-map lat=event.latitude lng=event.longitude 
 zoom=12 gestureHandling='cooperative' 
 streetView='StreetViewPanorama' as |context|}}
 {{g-map-marker context lat=event.latitude lng=event.longitude}}
 {{/g-map}}
</div>
<div class="eight wide column address">
 <h1>{{t 'Address'}}</h1>
 <p>{{event.locationName}}</p>
</div>
  • Next we have to decide the size of aur google map canvas on which our map is to be displayed. We define the size of our canvas in the public-event.scss file which is located under styles/pages/public-event-scss.
.event-map > .g-map {
 height: 100%;
 width: 100%;
}
 
.event-map > .g-map > .g-map-canvas {
 height: 300px;
}
  • Lastly, we have to modify our event-map-test.js file. Here we perform integration test for our component. We create a dummy object with latitude, longitude, and locationName in it and pass it to our component and then check if it renders correctly or not.
import { test } from 'ember-qunit';

import moduleForComponent from 'open-event-frontend/tests/helpers/component-helper';

import hbs from 'htmlbars-inline-precompile';

moduleForComponent('public/event-map', 'Integration | Component | public/event map');

let event = Object.create({ latitude: 37.7833, longitude: -122.4167, locationName: 'Sample event location address' });

test('it renders', function(assert) {

  this.set('event', event);

  this.render(hbs `{{public/event-map event=event}}`);

  assert.equal(this.$('.address p').text(), 'Sample event location address');

});

Now use ember s command in our ember cli and visit localhost:4200

Then open a event at the end we see our Google map integrated.

Use ember test –server command to check if all our tests (integration, acceptance, etc.) are passed.

 

Find out more at – ember-g-map

Continue ReadingImplementing Google Maps in Open Event ember Front-end
Read more about the article Set spacing in RecyclerView items by custom Item Decorator in Phimpme Android App
Spacing in RecyclerView GridLayoutManager

Set spacing in RecyclerView items by custom Item Decorator in Phimpme Android App

We have decided to shift our images Gallery code from GridView to using Grid Layout manager in RecyclerView in Phimpme Android application. RecyclerView has many advantages as compare to Grid/ List view.

  • Advantages of using layout manager either List, Grid or Staggered.
  • We can use many built in animations.
  • Item decorator for customizing the item.
  • Recycle items using the View Holder pattern

Recycler View documentation

Adding recyclerview in xml

<android.support.v7.widget.RecyclerView

    android:layout_width="match_parent"

    android:layout_height="match_parent"

    android:id="@+id/rv"

    />

Setting layout manager

mLayoutManager = new GridLayoutManager(this, 3);

recyclerView.setLayoutManager(mLayoutManager);

In phimpme we have an item as an ImageView, to show into the grid. Setup the Grid using layout manager as above.

Gallery of images is set but there is no spacing in between grid items. Padding will not help in this case.

Found a way to set offset by creating a Custom item decoration class. Add a constructor with parameter as a dimension resource.

 

public class ItemOffsetDecoration extends RecyclerView.ItemDecoration {

   private int mItemOffset;

   public ItemOffsetDecoration(int itemOffset) {

       mItemOffset = itemOffset;

   }



   public ItemOffsetDecoration(@NonNull Context context, @DimenRes int itemOffsetId) {

       this(context.getResources().getDimensionPixelSize(itemOffsetId));

   }

   @Override

   public void getItemOffsets(Rect outRect, View view, RecyclerView parent,

           RecyclerView.State state) {

       super.getItemOffsets(outRect, view, parent, state);

       outRect.set(mItemOffset, mItemOffset, mItemOffset, mItemOffset);

   }

}

Author: gist.github.com/yqritc/ccca77dc42f2364777e1

Usage:

ItemOffsetDecoration itemDecoration = new ItemOffsetDecoration(context, R.dimen.item_offset);

mRecyclerView.addItemDecoration(itemDecoration)

Pass the item_offset value in the function. Go through the material design guidelines for a clear understanding of dimensions in item offset.

Continue ReadingSet spacing in RecyclerView items by custom Item Decorator in Phimpme Android App
Read more about the article Arch Linux for Travis CI

Arch Linux for Travis CI

Travis CI does not provide Arch Linux as a testing environment but if you need to test some script or some app specifically on Arch Linux, you would have to do that manually for every change made into code and that would waste time. I had a similar problem when I was trying to test the Event Linux build scripts based on Arch Linux which required the Arch Linux environment to be able to run build scripts which take around 25 to 30 minutes to be tested for every change

How to use Arch travis script for Travis CI?

To solve the problem we need to use Arch as chroot in Travis CI. To do so we are going to use this script (link) which will deploy Arch Linux chroot on travis for us to start testing
For using it we need to configure the travis.yml like this :-

sudo: required
arch:
 repos:
  - archlinuxfr=http://repo.archlinux.fr/$arch
 packages:
  # pacman packages
  - yaourt
  - archiso
 script:

  - ./build-repo
  - sudo ./build.sh -v

script:
 - "travis_wait 30 sleep 1800 &"
 - curl -s https://raw.githubusercontent.com/fossasia/arch-travis/master/arch-travis.sh | bash

arch.packages defines a list of packages (from official repository or AUR) to be installed before running the build.
For eg:

packages:
  # pacman packages
  - python
  - perl
  # aur packages
  - go-git
  # packages from custom repo
  - yaourt

Here in arch.repos we define custom repository required for the package which is required during testing.

For eg. in above code repo : http://repo.archlinux.fr/$arch is used to install yaourt we could have used AUR too but that will increase our build testing time as the AUR will first download the pkgbuild and then build the package which can result in increase of time

arch.script defines a list of scripts to run as part of the build. Like in above example we have used ./build-repo and sudo ./build -v

Anything defined in the arch.script list will run from the base of the repository as a normal user called travis CI . sudo is available as well as any packages installed in the setup. The path of the build dir (or repository base), is stored in the TRAVIS_BUILD_DIR environment variable inside the chroot.

At the end of example we have used
script:
Which defines the scripts to be run by travis, this is where arch-travis was initialized and then we pipe it

Usage of travis-wait function

script:

- travis_wait 30 sleep 1800 &
 - curl -s https://raw.githubusercontent.com/fossasia/arch-travis/master/arch-travis.sh | bash

We cannot use travis wait function inside the chroot neither we can export it so we apply the travis wait function at the point where we have initialised the Arch-travis

Example usage for pkgbuild scripts

sudo: required # this to get sudo permissions

arch: # anything to be tested under Arch is defined below this
 script: 
  - makepkg app.pkgbuild # this will test the pkgbuild script

script: #for setting up arch chroot using the arch travis script
 - curl -s https://raw.githubusercontent.com/fossasia/arch-travis/master/arch-travis.sh | bash

Limitations

  • Increases build time by about 1-3 min.
  • Does not work on travis container-based infrastructure because sudo is required.
  • Limited configuration.
  • Does not include base group packages.
Continue ReadingArch Linux for Travis CI
Read more about the article Deploying SUSI.AI with Docker

Deploying SUSI.AI with Docker

Docker is much more efficient than VM in allocating shared resources between various containers as shown in figure. To deploy SUSI we need to create docker container. There are two ways to build it. First way is fork the SUSI project in github. Then you can signup in dockerhub and create autobuild docker container. The second way is to manually build docker file from command prompt of your computer. The following instructions needs to be executed in cloud shell or linux machine.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get -y install docker.io
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo docker build https://github.com/fossasia/susi_server.git


The first three commands install docker software to the machine. Next three lines give required permissions and execution abilities to docker. Final command builds docker container from github. Thus, We have successfully made docker container. We can deploy it on the cloud by using following command.

sudo docker run -d -p 80:80 -p 443:443 susi


Deploying Susi on cloud with kubernetes

We will use Google Cloud Service platform for demonstration. Create your GCS account. Goto dashboard and click on compute engine.

Enable Billing and Create New Project named XYZ. Open the terminal, by clicking the Google cloud shell button on the top.

Please set the compute zone to your nearest zone by running the below command.

gcloud config set compute/zone us-central1-a


Now we need to create cluster, on which we deploy susi app. We do it by this command.

gcloud container clusters create hello-cluster --num-nodes=3


We need to get docker from dockerhub and push it to our project repo.We do it by these commands.

sudo docker pull jyothiraditya/susi_server
gcloud docker -- push <image-id> gcr.io/<project-id>/<name>

We run the docker image on cluster by following commands.

kubectl run susi-server --image=gcr.io/<project-id>/<name> --port=80
kubectl get pods

We expose the container to external traffic, with help of load balancer by the following command.

kubectl expose deployment susi-server --type="LoadBalancer --port=80"


We get the external ip-address to access susi from browser. By entering

kubectl get service susi-server


You can now view the app by going to “EXTERNAL-IP:80”.

References : Docker build , Kubernetes deployment, Google cloud deployment 

 

Continue ReadingDeploying SUSI.AI with Docker
Read more about the article Using LokLak to Scrape Profiles from Quora, GitHub, Weibo and Instagram

Using LokLak to Scrape Profiles from Quora, GitHub, Weibo and Instagram

Most of us are really so curious to know about one’s social life. So taking this as a key point, LokLak has many profile scrapers in it. Profile scraper which are now available in LokLak  helps us to know about the posts, followers one has. Few of the profile scrapers available in LokLak are Quora Profile, GitHub Profile, Weibo Profile and Instagram Profile.

How do the scrapers work?

In loklak now we are using java to get the json objects of the scraped profile from different websites as mentioned above. So here is a simple explanation how one of the scraper works. In this post I am going to give you a gist about how Github Profile scraper API works:

In the github profile scraper one can search for a profile without logging in and know the contents like the followers, repositories, gists of that profile and many more.

The simple query which can be used is:

To scrape individual profiles:

https://loklak.org/api/githubprofilescraper.json?profile=kavithaenair


To scrape organization profiles:

https://loklak.org/api/githubprofilescraper.json?profile=fossasia

Jsoup is an API and it is a easiest way used by java developers for scraping the web i.e.,web scraping. This API is used for manipulating and extracting data using DOM, CSS like methods. So in here, the Jsoup API is helping us to extract the html data and with the help of the tags used in the html extracted data we are trying to get the relevant data which is needed.

How do we get the matching elements?

We here are using special methods like getElementsByAttributeValueContaining() of the org.jsoup.nodes.Element class to get the data. For instance, to get the email from the extracted data the code is written as:

String email = html.getElementsByAttributeValueContaining("itemprop", "email").text();
            if (!email.contains("@"))
                  email = "";
            githubProfile.put("email", email);

Code:

Here is the java code which imports and extracts data:

Imports the html file:

html = Jsoup.connect("https://github.com/" + profile).get();

Extracts the html file for individual user:

/*If Individual*/
           if (html.getElementsByAttributeValueContaining("class", "user-profile-nav").size() != 0) {
               scrapeGithubUser(githubProfile, terms, profile, html);
           }
           if (terms.contains("gists") || terms.contains("all")) {
               String gistsUrl = GITHUB_API_BASE + profile + GISTS_ENDPOINT;
               JSONArray gists = getDataFromApi(gistsUrl);
               githubProfile.put("gists", gists);
           }
           if (terms.contains("subscriptions") || terms.contains("all")) {
               String subscriptionsUrl = GITHUB_API_BASE + profile + SUBSCRIPTIONS_ENDPOINT;
               JSONArray subscriptions = getDataFromApi(subscriptionsUrl);
               githubProfile.put("subscriptions", subscriptions);
           }
           if (terms.contains("repos") || terms.contains("all")) {
               String reposUrl = GITHUB_API_BASE + profile + REPOS_ENDPOINT;
               JSONArray repos = getDataFromApi(reposUrl);
               githubProfile.put("repos", repos);
           }
           if (terms.contains("events") || terms.contains("all")) {
               String eventsUrl = GITHUB_API_BASE + profile + EVENTS_ENDPOINT;
               JSONArray events = getDataFromApi(eventsUrl);
               githubProfile.put("events", events);
          }
          if (terms.contains("received_events") || terms.contains("all")) {
              String receivedEventsUrl = GITHUB_API_BASE + profile + RECEIVED_EVENTS_ENDPOINT;
              JSONArray receivedEvents = getDataFromApi(receivedEventsUrl);
              githubProfile.put("received_events", receivedEvents);
          }

Extracts the html file for organization:

/*If organization*/
if (html.getElementsByAttributeValue("class", "orgnav").size() != 0) {
    scrapeGithubOrg(profile, githubProfile, html);
}

And this is the sample output:

For query: https://loklak.org/api/githubprofilescraper.json?profile=kavithaenair
 
{
  "data": [{
    "joining_date": "2016-04-12",
    "gists_url": "https://api.github.com/users/kavithaenair/gists",
    "repos_url": "https://api.github.com/users/kavithaenair/repos",
    "user_name": "kavithaenair",
    "bio": "GSoC'17 @loklak @fossasia ; Developer @fossasia ; Intern @amazon",
    "subscriptions_url": "https://api.github.com/users/kavithaenair/subscriptions",
    "received_events_url": "https://api.github.com/users/kavithaenair/received_events",
    "full_name": "Kavitha E Nair",
    "avatar_url": "https://avatars0.githubusercontent.com/u/18421291",
    "user_id": "18421291",
    "events_url": "https://api.github.com/users/kavithaenair/events",
    "organizations": [
      {
        "img_link": "https://avatars1.githubusercontent.com/u/6295529?v=3&s=70",
        "link": "https://github.com/fossasia",
        "label": "fossasia",
        "img_Alt": "@fossasia"
      },
      {
        "img_link": "https://avatars0.githubusercontent.com/u/10620750?v=3&s=70",
        "link": "https://github.com/coala",
        "label": "coala",
        "img_Alt": "@coala"
      },
      {
        "img_link": "https://avatars1.githubusercontent.com/u/11370631?v=3&s=70",
        "link": "https://github.com/loklak",
        "label": "loklak",
        "img_Alt": "@loklak"
      },
      {
        "img_link": "https://avatars2.githubusercontent.com/u/24720168?v=3&s=70",
        "link": "https://github.com/bvrit-wise-django-team",
        "label": "bvrit-wise-django-team",
        "img_Alt": "@bvrit-wise-django-team"
      }
    ],
    "home_location": "\n    Hyderabad, India\n",
    "works_for": "",
    "special_link": "https://www.overleaf.com/read/ftnvcphnwzhp",
    "email": "",
    "atom_feed_link": "https://github.com/kavithaenair.atom"
  }],
  "metadata": {"count": 1},
  "session": {"identity": {
    "type": "host",
    "name": "162.158.46.18",
    "anonymous": true
  }}
}
Continue ReadingUsing LokLak to Scrape Profiles from Quora, GitHub, Weibo and Instagram

Linking Codecov to your Angular2 Project

As a developer, the importance of testing code is well known. Testing source code helps to prevent bugs and syntax errors by cross-checking it with an expected output.
As stated on their official website: “Code coverage provides a visual measurement of what source code is being executed by a test suite”. This information indicates to the software developer where they should write new tests in the effort to achieve higher coverage, and consequently a lower chance of being buggy. Hence nearly every public repository on git uses codecov, a tool to measure the coverage of their source code.

In this blog, we shall see how to link codecov, with a public repository on Github when the code has been written in Angular2(Typescript). We shall assume that the repository uses Travis CI for integration.

STEP 1:
Go to https://codecov.io/gh/ and login to your Github account.

It will now give you the option to chose a repository to add for coverage. Select your repository.

STEP 2:
Navigate to Settings Tab, you should see something like this:

Follow the above-mentioned instructions.

STEP 3:
We now come to one of the most important parts of Codecov integration. Writing the files in our repo to enable this.
We will need three main files:
Travis.yml- which will ensure continuous integration services on your git hosted project
Codecov.yml- to personalise your settings and override the default settings in codecov.”The Codecov Yaml file is the single point of configuration, providing the developers with a transparent and version controlled file to adjust all Codecov settings.” as mentioned in the official website
Package.json- to inform npm of the new dependencies related to codecov, in addition to providing all the metadata to the user.

In .travis.yml, Add the following line:
after_success:

 - bash <(curl -s https://codecov.io/bash)

In codecov.yml, Add the following

Source: https://github.com/codecov/support/wiki/Codecov-Yaml#
 codecov:
 url: "string" # [enterprise] your self-hosted Codecov endpoint
 # ex. https://codecov.company.com
 slug: "owner/repo" # [enterprise] the project's name when using the global upload tokens
 branch: master # the branch to show by default, inherited from your git repository settings
 # ex. master, stable or release
 # default: the default branch in git/mercurial
 bot: username # the username that will consume any oauth requests
 # must have previously logged into Codecov
 ci: # [advanced] a list of custom CI domains
 - "ci.custom.com"
 notify: # [advanced] usage only
 after_n_builds: 5 # how many build to wait for before submitting notifications
 # therefore skipping status checks
 countdown: 50 # number of seconds to wait before checking CI status
 delay: 100 # number of seconds between each CI status check

coverage:
 precision: 2 # how many decimal places to display in the UI: 0 <= value <= 4 round: down # how coverage is rounded: down/up/nearest range: 50...100 # custom range of coverage colors from red -> yellow -> green

notify:
 irc:
 default: # -> see "sections" below
 server: "chat.freenode.net" #*S the domain of the irc server
 branches: null # -> see "branch patterns" below
 threshold: null # -> see "threshold" below
 message: "template string" # [advanced] -> see "customized message" below

gitter:
 default: # -> see "sections" below
 url: "https://webhooks.gitter.im/..." #*S unique Gitter notifications url
 branches: null # -> see "branch patterns" below
 threshold: null # -> see "threshold" below
 message: "template string" # [advanced] -> see "customized message" below

status:
 project: # measuring the overall project coverage
 default: # context, you can create multiple ones with custom titles
 enabled: yes # must be yes|true to enable this status
 target: auto # specify the target coverage for each commit status
 # option: "auto" (must increase from parent commit or pull request base)
 # option: "X%" a static target percentage to hit
 branches: # -> see "branch patterns" below
 threshold: null # allowed to drop X% and still result in a "success" commit status
 if_no_uploads: error # will post commit status of "error" if no coverage reports we uploaded
 # options: success, error, failure
 if_not_found: success # if parent is not found report status as success, error, or failure
 if_ci_failed: error # if ci fails report status as success, error, or failure


patch: # pull requests only: this commit status will measure the
 # entire pull requests Coverage Diff. Checking if the lines
 # adjusted are covered at least X%.
 default:
 enabled: yes # must be yes|true to enable this status
 target: 80% # specify the target "X%" coverage to hit
 branches: null # -> see "branch patterns" below
 threshold: null # allowed to drop X% and still result in a "success" commit status
 if_no_uploads: error # will post commit status of "error" if no coverage reports we uploaded
 # options: success, error, failure
 if_not_found: success
 if_ci_failed: error

changes: # if there are any unexpected changes in coverage
 default:
 enabled: yes # must be yes|true to enable this status
 branches: null # -> see "branch patterns" below
 if_no_uploads: error
 if_not_found: success
 if_ci_failed: error

ignore: # files and folders that will be removed during processing
 - "tests/*"
 - "demo/*.rb"

fixes: # [advanced] in rare cases the report tree is invalid, specify adjustments here
 - "old_path::new_path"

# comment: false # to disable comments
 comment:
 layout: "header, diff, changes, sunburst, suggestions, tree"
 branches: null # -> see "branch patterns" below
 behavior: default # option: "default" posts once then update, posts new if delete
 # option: "once" post once then updates, if deleted do not post new
 # option: "new" delete old, post new
 # option: "spammy" post new

Your package.json should look like this:

{
 "name": "example-typescript",
 "version": "1.0.0",
 "description": "Codecov Example Typescript",
 "main": "index.js",
 "devDependencies": {
 "chai": "^3.5.0",
 "codecov": "^1.0.1",
 "mocha": "^2.5.3",
 "nyc": "^6.4.4",
 "tsd": "^0.6.5",
 "typescript": "^1.8.10"
 },
 "scripts": {
 "postinstall": "tsd install",
 "pretest": "tsc test/*.ts --module commonjs --sourcemap",
 "test": "nyc mocha",
 "posttest": "nyc report --reporter=json && codecov -f coverage/*.json"
 },
 "repository": {
 "type": "git",
 "url": "git+https://github.com/Myname/example-typescript.git"
 },
 /*Optional*/
 "author": "Myname",
 "license": "Lic.name",
 "bugs": {
 "url": "https://github.com/example-typescript-path"
 },
 "homepage": "https://github.com/Myname/example-typescript#readme"
 }

Most of the code in package.json is metadata.
Two major parts of the code above are the devDependencies and the scripts.
In devDependencies, make sure to include the latest versions of all the packages your repository is using.
In scripts:

  • Postinstall – indicates the actions to be performed, once installation is complete.
  • Pretest – is for just before running ng test.
  • Test – indicates what is used while testing.
  • Posttest – is what is run just after testing is complete.

Check this repository for the sample files to generate the reports to be uploaded for Codecov: https://github.com/codecov/example-typescript

Check https://docs.codecov.io/v4.3.6/docs/codecov-yaml for detailed step by step instructions on writing codecov.yaml and https://docs.codecov.io/v4.3.6/docs for any general information

Continue ReadingLinking Codecov to your Angular2 Project