How to use Digital Ocean and Docker to setup Test CMS for Phimpme

One of the core feature of Phimpme app is sharing images to other different accounts, including various open source CMS such as WordPress, Drupal etc and other open source data storage account such as OwnCloud, NextCloud etc.

One can not have everything at place, but for development and testing purpose it is required in our end. So problem I was facing to get things done in most optimize way. I thought setting things on hosted server would be good, because it saves lots of time in setting locally in our system, adding all the dependencies. And also we cannot share the account as it is limited to our local environment.

Digital Ocean caught my attention in providing hosting service. It is very easy to use with their droplet creation. Select as per your system requirement and service requirement, the droplet will be ready in few moments and we can use it anywhere.

Note: DigitalOcean is a paid service. Student can use Github Education Pack for free credits on Digital Ocean. I used the same.

I currently worked on Nextcloud integration so here in this blog I will tell how to quickly create nextcloud server using Digital Ocean and Docker.

Step 1: Creating Droplet

DigitalOcean completely work on droplets and one can anytime create and destroy different droplets associated with their account.

Choose an Image

So there are three options of choosing the image of Droplet.

Distributions : Which is other operating systems you want to use

One Click app: It is a very good feature as it creates everything for use in just one click. But again, it doesn’t provide everything, like there is no NextCloud. That’s why I used docker to take its image.

Snapshots: This is if you saved your droplet already, so it will pick it and creates similar to the saved image. Here I selected Docker from one-click apps section.

Selecting the size

This is for selecting the size of the server we are creating, For small development purpose $5 plan is good. There is a good thing in DigitalOcean as it didn’t charge on the monthly basis to the use. It works on hourly basis and charge according to that. Also providing the SSD Disk for fast IO operations.

Choose a datacenter Region

Add SSH

This is very important to add a ssh key. Otherwise you have to setup root password or used the shell they provide. To work on your computer terminal, its good that you setup an ssh key already and it to the account.

How to get ssh key in your system: https://help.github.com/

Rename the number of droplet and name of the droplet and create.

Now it will show there in your droplet section

Step 2: Access the Server

As we have already added the ssh key to our droplet. Now we can access it from our terminal. Open the terminal and type this

➜  ~ ssh root@<your IP> 

It will logged in to you

root@docker-512mb-blr1-01:~# 

Our objective is setting a NextCloud Account.

Here now I will use Docker. Firstly, What is Docker?

Go here to read: https://www.docker.com/what-docker

I will explain docker in other words. Like I setted up everything which I need. Now If I have to destroy this all and want to use it after some days. Or if my friends wants to use the setted platform. What is the option here?

Recreate and everything everytime? NO.

Just create docker image, save it pull the image when you want, and run it to serve on the serve. Your friends need, provide them the docker image.

Isn’t it cool and much time saving.

Browse the Docker Hub

In the hub we can find docker images for various platforms officially maintained by the authors.

Nextcloud have their official account on Docker to provide latest images to the developers.

Here is the link : https://hub.docker.com/_/nextcloud/

Pull the image in your server.

root@docker-512mb-blr1-01:~# docker pull nextcloud
Using default tag: latest
latest: Pulling from library/nextcloud
9f0706ba7422: Pull complete
4c407763908f: Pull complete
82e2bc3a45c1: Pull complete
c84e1013aed1: Pull complete
a3b5e03d7e24: Pull complete
917f836a88be: Pull complete
b2dc54431819: Pull complete
a60b574790b8: Pull complete
49ef0f1aff88: Pull complete
7773a865ee49: Pull complete
9e0e5cc56a9d: Pull complete
bfade1c7421e: Pull complete
ece8ceb33bed: Pull complete
c691d2747a3e: Pull complete
4b5e96bf54c9: Pull complete
6fbe30ae456b: Pull complete
e0c534b35a6b: Pull complete
4d2687f4b6f3: Pull complete
00197422846a: Pull complete
6ab57168c49c: Pull complete
9e1260db005f: Pull complete
Digest: sha256:1bb5c256f19dcec60d8468c00bc7dc74efdf93390666cb82e20bcacbbbd9746c
Status: Downloaded newer image for nextcloud:latest
root@docker-512mb-blr1-01:~#

Following the documentation

I need to run this command $ docker run -d -p 8080:80 nextcloud

It serves the account on localhost.

Check on your https://<IP>:8080

So in this way I easily setup different account for testing and integration purpose in Phimpme Android app. It really saves lots of time and speed up the process.

You can easily destroy the droplet after work is done.

Student can use the free credits from GitHub Education Pack.

Source:

 

Implementing the Message Response Status Indicators In SUSI WebChat

SUSI Web Chat now has indicators reflecting the message response status. When a user sends a message, he must be notified that the message has been received and has been delivered to server. SUSI Web Chat implements this by tagging messages with ticks or waiting clock icons and loading gifs to indicate delivery and response status of messages ensuring good UX.

This is implemented as:

  • When the user sends a message, the message is tagged with a `clock` icon indicating that the message has been received and delivered to server and is awaiting response from the server
  • When the user is waiting for a response from the server, we display a loading gif
  • Once the response from the server is received, the loading gif is replaced by the server response bubble and the clock icon tagged to the user message is replaced by a tick icon.

Lets visit SUSI WebChat and try it out.

Query : Hey

When the message is sent by the user, we see that the displayed message is tagged with a clock icon and the left side response bubble has a loading gif indicating that the message has been delivered to server and are awaiting response.

When the response from server is delivered, the loading gif disappears and the user message tagged with a tick icon.

 

How was this implemented?

The first step is to have a boolean flag indicating the message delivery and response status.

let _showLoading = false;

getLoadStatus(){
  return _showLoading;
},

The `showLoading` boolean flag is set to true when the user just sends a message and is waiting for server response.  When the user sends a message, the CREATE_MESSAGE action is triggered. Message Store listens to this action and along with creating the user message, also sets the showLoading flag as true.

case ActionTypes.CREATE_MESSAGE: {

  let message = action.message;
  _messages[message.id] = message;
  _showLoading = true;
  MessageStore.emitChange();
  
  break;
}

The showLoading flag is used in MessageSection to display the loading gif. We are using a saved gif to display the loading symbol. The loading gif is displayed at the end after displaying all the messages in the message store. Since this loading component must be displayed for every user message, we don’t save this component in MessageStore as a loading message as that would lead to repeated looping thorugh the messages in message store to add and delete loading component.

import loadingGIF from '../../images/loading.gif';

function getLoadingGIF() {

  let messageContainerClasses = 'message-container SUSI';

  const LoadingComponent = (
    <li className='message-list-item'>
      <section className={messageContainerClasses}>
        <img src={loadingGIF}
          style={{ height: '10px', width: 'auto' }}
          alt='please wait..' />
      </section>
    </li>
  );
  return LoadingComponent;
}

We then use this flag in MessageListItem class to tag the user messages with the clock icons. We used Material UI SVG Icons to display the clock and tick messages. We display these beside the time in the messages.

import ClockIcon from 'material-ui/svg-icons/action/schedule';

statusIndicator = (
  <li className='message-time' style={footerStyle}>
    <ClockIcon style={indicatorStyle}
      color={UserPreferencesStore.getTheme()==='light' ? '#90a4ae' : '#7eaaaf'}/>
  </li>
);

When the response from server is received, the CREATE_SUSI_MESSAGE action is triggered to render the server response. This action is again collected in MessageStore where the `showLoading` boolean flag is reset to false. This event also triggers the state of MessageSection where we are listening to showLoading value from MessageStore, hence triggering changes in MessageSection and accordingly in MessageListItem where showLoading is passed as props, removing the loading gif component and displaying the server response and replacing the clock icon with tick icon on the user message.

case ActionTypes.CREATE_SUSI_MESSAGE: {
  
  let message = action.message;
  MessageStore.resetVoiceForThread(message.threadID);
  _messages[message.id] = message;
  _showLoading = false;
  MessageStore.emitChange();
  
  break;
}

This is how the status indicators were implemented for messages. The complete code can be found at SUSI WebChat Repo.

Resources

Introducing Priority Kaizen Harvester for loklak server

In the previous blog post, I discussed the changes made in loklak’s Kaizen harvester so it could be extended and other harvesting strategies could be introduced. Those changes made it possible to introduce a new harvesting strategy as PriorityKaizen harvester which uses a priority queue to store the queries that are to be processed. In this blog post, I will be discussing the process through which this new harvesting strategy was introduced in loklak.

Background, motivation and approach

Before jumping into the changes, we first need to understand that why do we need this new harvesting strategy. Let us start by discussing the issue with the Kaizen harvester.

The produce consumer imbalance in Kaizen harvester

Kaizen uses a simple hash queue to store queries. When the queue is full, new queries are dropped. But numbers of queries produced after searching for one query is much higher than the consumption rate, i.e. the queries are bound to overflow and new queries that arrive would get dropped. (See loklak/loklak_server#1156)

Learnings from attempt to add blocking queue for queries

As a solution to this problem, I first tried to use a blocking queue to store the queries. In this implementation, the producers would get blocked before putting the queries in the queue if it is full and would wait until there is space for more. This way, we would have a good balance between consumers and producers as the consumers would be waiting until producers can free up space for them –

public class BlockingKaizenHarvester extends KaizenHarvester {
   ...
   public BlockingKaizenHarvester() {
       super(new KaizenQueries() {
           ...
           private BlockingQueue<String> queries = new ArrayBlockingQueue<>(maxSize);

           @Override
           public boolean addQuery(String query) {
               if (this.queries.contains(query)) {
                   return false;
               }
               try {
                   this.queries.offer(query, this.blockingTimeout, TimeUnit.SECONDS);
                   return true;
               } catch (InterruptedException e) {
                   DAO.severe("BlockingKaizen Couldn't add query: " + query, e);
                   return false;
               }
           }
           @Override
           public String getQuery() {
               try {
                   return this.queries.take();
               } catch (InterruptedException e) {
                   DAO.severe("BlockingKaizen Couldn't get any query", e);
                   return null;
               }
           }
           ...
       });
   }
}

[SOURCE, loklak/loklak_server#1210]

But there is an issue here. The consumers themselves are producers of even higher rate. When a search is performed, queries are requested to be appended to the KaizenQueries instance for the object (which here, would implement a blocking queue). Now let us consider the case where queue is full and a thread requests a query from the queue and scrapes data. Now when the scraping is finished, many new queries are requested to be inserted to most of them get blocked (because the queue would be full again after one query getting inserted).

Therefore, using a blocking queue in KaizenQueries is not a good thing to do.

Other considerations

After the failure of introducing the Blocking Kaizen harvester, we looked for other alternatives for storing queries. We came across multilevel queues, persistent disk queues and priority queues.

Multilevel queues sounded like a good idea at first where we would have multiple queues for storing queries. But eventually, this would just boil down to how much queue size are we allowing and the queries would eventually get dropped.

Persistent disk queues would allow us to store greater number of queries but the major disadvantage was lookup time. It would terribly slow to check if a query already exists in the disk queue when the queue is large. Also, since the queries would always increase practically, the disk queue would also go out of hand at some point in time.

So by now, we were clear that not dropping queries is not an alternative. So what we had to use the limited size queue smartly so that we do not drop queries that are important.

Solution: Priority Queue

So a good solution to our problem was a priority queue. We could assign a higher score to queries that come from more popular Tweets and they would go higher in the queue and do not drop off until we have even higher priority queried in the queue.

Assigning score to a Tweet

Score for a tweet was decided using the following formula –

α= 5* (retweet count)+(favourite count)

score=α/(α+10*exp(-0.01*α))

This equation generates a score between zero and one from the retweet and favourite count of a Tweet. This normalisation of score would ensure we do not assign an insanely large score to Tweets with a high retweet and favourite count. You can see the behaviour for the second mentioned equation here.

Graph?

Changes required in existing Kaizen harvester

To take a score into account, it became necessary to add an interface to also provide a score as a parameter to the addQuery() method in KaizenQueries. Also, not all queries can have a score associated with it, for example, if we add a query that would search for Tweets older than the oldest in the current timeline, giving it a score wouldn’t be possible as it would not be associated with a single Tweet. To tackle this, a default score of 0.5 was given to these queries –

public abstract class KaizenQueries {

   public boolean addQuery(String query) {
       return this.addQuery(query, 0.5);
   }

   public abstract boolean addQuery(String query, double score);
   ...
}

[SOURCE]

Defining appropriate KaizenQueries object

The KaizenQueries object for a priority queue had to define a wrapper class that would hold the query and its score together so that they could be inserted in a queue as a single object.

ScoreWrapper and comparator

The ScoreWrapper is a simple class that stores score and query object together –

private class ScoreWrapper {

   private double score;
   private String query;

   ScoreWrapper(String m, double score) {
       this.query = m;
       this.score = score;
   }

}

[SOURCE]

In order to define a way to sort the ScoreWrapper objects in the priority queue, we need to define a Comparator for it –

private Comparator<ScoreWrapper> scoreComparator = (scoreWrapper, t1) -> (int) (scoreWrapper.score - t1.score);

[SOURCE]

Putting things together

Now that we have all the ingredients to declare our priority queue, we can also declare the strategy to getQuery and putQuery in the corresponding KaizenQueries object –

public class PriorityKaizenHarvester extends KaizenHarvester {

   private static class PriorityKaizenQueries extends KaizenQueries {
       ...
       private Queue<ScoreWrapper> queue;
       private int maxSize;

       public PriorityKaizenQueries(int size) {
           this.maxSize = size;
           queue = new PriorityQueue<>(size, scoreComparator);
       }

       @Override
       public boolean addQuery(String query, double score) {
           ScoreWrapper sw = new ScoreWrapper(query, score);
           if (this.queue.contains(sw)) {
               return false;
           }
           try {
               this.queue.add(sw);
               return true;
           } catch (IllegalStateException e) {
               return false;
           }
       }

       @Override
       public String getQuery() {
           return this.queue.poll().query;
       }
       ...
}

[SOURCE]

Conclusion

In this blog post, I discussed the process in which PriorityKaizen harvester was introduced to loklak. This strategy is a flavour of Kaizen harvester which uses a priority queue to store queries that are to be processed. These changes were possible because of a previous patch which allowed extending of Kaizen harvester.

The changes were introduced in pull request loklak/loklak#1240 by @singhpratyush (me).

Resources

Fetching URL for Embedded Twitter Videos in loklak server

The primary web service that loklak scrapes is Twitter. Being a news and social networking service, Twitter allows its users to post videos directly to Twitter and they convey more thoughts than what text can. But for an automated scraper, getting the links is not a simple task.

Let us see that what were the problems we faced with videos and how we solved them in the loklak server project.

Previous setup and embedded videos

In the previous version of loklak server, the TwitterScraper searched for videos in 2 ways –

  1. Youtube links
  2. HTML5 video links

To fetch the video URL from HTML5 video, following snippet was used –

if ((p = input.indexOf("<source video-src")) >= 0 && input.indexOf("type=\"video/") > p) {
   String video_url = new prop(input, p, "video-src").value;
   videos.add
   continue;
}

Here, input is the current line from raw HTML that is being processed and prop is a class defined in loklak that is useful in parsing HTML attributes. So in this way, the HTML5 videos were extracted.

The Problem – Embedded videos

Though the previous setup had no issues, it was useless as Twitter embeds the videos in an iFrame and therefore, can’t be fetched using simple HTML5 tag extraction.

If we take the following Tweet for example,

the requested HTML from the search page contains video in following format –

<src="https://twitter.com/i/videos/tweet/881946694413422593?embed_source=clientlib&player_id=0&rpc_init=1" allowfullscreen="" id="player_tweet_881946694413422593" style="width: 100%; height: 100%; position: absolute; top: 0; left: 0;">

So we needed to come up with a better technique to get those videos.

Parsing video URL from iFrame

The <div> which contains video is marked with AdaptiveMedia-videoContainer class. So if a Tweet has an iFrame containing video, it will also have the mentioned class.

Also, the source of iFrame is of the form https://twitter.com/i/videos/tweet/{Tweet-ID}. So now we can programmatically go to any Tweet’s video and parse it to get results.

Extracting video URL from iFrame source

Now that we have the source of iFrame, we can easily get the video source using the following flow –

public final static Pattern videoURL = Pattern.compile("video_url\\\":\\\"(.*?)\\\"");

private static String[] fetchTwitterIframeVideos(String iframeURL) {
   // Read fron iframeURL line by line into BufferReader br
   while ((line = br.readLine()) != null ) {
       int index;
       if ((index = line.indexOf("data-config=")) >= 0) {
           String jsonEscHTML = (new prop(line, index, "data-config")).value;
           String jsonUnescHTML = HtmlEscape.unescapeHtml(jsonEscHTML);
           Matcher m = videoURL.matcher(jsonUnescHTML);
           if (!m.find()) {
               return new String[]{};
           }
           String url = m.group(1);
           url = url.replace("\\/", "/");  // Clean URL
           /*
            * Play with url and return results
            */
       }
   }
}

MP4 and M3U8 URLs

If we encounter mp4 URLs, we’re fine as it is the direct link to video. But if we encounter m3u8 URL, we need to process it further before we can actually get to the videos.

For Twitter, the hosted m3u8 videos contain link to further m3u8 videos which are of different resolution. These m3u8 videos again contain link to various .ts files that contain actual video in parts of 3 seconds length each to support better streaming experience on the web.

To resolve videos in such a setup, we need to recursively parse m3u8 files and collect all the .ts videos.

private static String[] extractM3u8(String url) {
   return extractM3u8(url, "https://video.twimg.com/");
}

private static String[] extractM3u8(String url, String baseURL) {
   // Read from baseURL + url line by line
   while ((line = br.readLine()) != null) {
       if (line.startsWith("#")) {  // Skip comments in m3u8
           continue;
       }
       String currentURL = (new URL(new URL(baseURL), line)).toString();
       if (currentURL.endsWith(".m3u8")) {
           String[] more = extractM3u8(currentURL, baseURL);  // Recursively add all
           Collections.addAll(links, more);
       } else {
           links.add(currentURL);
       }
   }
   return links.toArray(new String[links.size()]);
}

And then in fetchTwitterIframeVideos, we can return the all .ts URLs for the video –

if (url.endsWith(".mp4")) {
   return new String[]{url};
} else if (url.endsWith(".m3u8")) {
   return extractM3u8(url);
}

Putting things together

Finally, the TwitterScraper can discover the video links by tweaking a little –

if (input.indexOf("AdaptiveMedia-videoContainer") > 0) {
   // Fetch Tweet ID
   String tweetURL = props.get("tweetstatusurl").value;
   int slashIndex = tweetURL.lastIndexOf('/');
   if (slashIndex < 0) {
       continue;
   }
   String tweetID = tweetURL.substring(slashIndex + 1);
   String iframeURL = "https://twitter.com/i/videos/tweet/" + tweetID;
   String[] videoURLs = fetchTwitterIframeVideos(iframeURL);
   Collections.addAll(videos, videoURLs);
}

Conclusion

This blog post explained the process of extracting video URL from Twitter and the problem faced. The discussed change enabled loklak to extract and serve URLs to video for tweets. It was introduced in PR loklak/loklak_server#1193 by me (@singhpratyush).

The service was further enhanced to collect single mp4 link for videos (see PR loklak/loklak_server#1206), which is discussed in another blog post.

Resources

How SUSI WebChat Implements RSS Action Type

SUSI.AI now has a new action type called RSS. As the name suggests, SUSI is now capable of searching the internet to answer user queries. This web search can be performed either on the client side or the server side. When the web search is to be performed on the client side, it is denoted by websearch action type. When the web search is performed by the server itself, it is denoted by rss action type. The server searches the internet and using RSS feeds, returns an array of objects containing :

  • Title
  • Description
  • Link
  • Count

Each object is displayed as a result tile and all the results are rendered as swipeable tiles.

Lets visit SUSI WebChat and try it out.

Query : Google
Response: API response

SUSI WebChat uses the same code abstraction to render websearch and rss results as both are results of websearch, only difference being where the search is being performed i.e client side or server side.

How does the client know that it is a rss action type response?

"actions": [
  {
    "type": "answer",
    "expression": "I found this on the web:"
  },
  {
    "type": "rss",
    "title": "title",
    "description": "description",
    "link": "link",
    "count": 3
  }
],

The actions attribute in the JSON API response has information about the action type and the keys to be parsed for title, link and description.

  • The type attribute tells the action type is rss.
  • The title attribute tells that title for each result is under the key – title for each object in answers[0].data.
  • Similarly keys to be parsed for description and link are description and link respectively.
  • The count attribute tells the client how many results to display.

We then loop through the objects in answers,data[0] and from each object we extract title, description and link.

let rssKeys = Object.assign({}, data.answers[0].actions[index]);

delete rssKeys.type;

let count = -1;

if(rssKeys.hasOwnProperty('count')){
  count = rssKeys.count;
  delete rssKeys.count;
}

let rssTiles = getRSSTiles(rssKeys,data.answers[0].data,count);

We use the count attribute and the length of answers[0].data to fix the number of results to be displayed.

// Fetch RSS data

export function getRSSTiles(rssKeys,rssData,count){

  let parseKeys = Object.keys(rssKeys);
  let rssTiles = [];
  let tilesLimit = rssData.length;

  if(count > -1){
    tilesLimit = Math.min(count,rssData.length);
  }

  for(var i=0; i<tilesLimit; i++){
    let respData = rssData[i];
    let tileData = {};

    parseKeys.forEach((rssKey,j)=>{
      tileData[rssKey] = respData[rssKeys[rssKey]];
    });

    rssTiles.push(tileData);
  }

return rssTiles;

}

We now have our list of objects with the information parsed from the response.We then pass this list to our renderTiles function where each object in the rssTiles array returned from getRSSTiles function is converted into a Paper tile with the title and description and the entire tile is hyperlinked to the given link using Material UI Paper Component and few CSS attributes.

// Draw Tiles for Websearch RSS data

export function drawTiles(tilesData){

let resultTiles = tilesData.map((tile,i) => {

  return(
    <div key={i}>
      <MuiThemeProvider>
        <Paper zDepth={0} className='tile'>
          <a rel='noopener noreferrer'
          href={tile.link} target='_blank'
          className='tile-anchor'>
            {tile.icon &&
            (<div className='tile-img-container'>
               <img src={tile.icon}
               className='tile-img' alt=''/>
             </div>
            )}
            <div className='tile-text'>
              <p className='tile-title'>
                <strong>
                  {processText(tile.title,'websearch-rss')}
                </strong>
              </p>
              {processText(tile.description,'websearch-rss')}
            </div>
          </a>
        </Paper>
      </MuiThemeProvider>
    </div>
  );

});

return resultTiles;
}

The tile title and description is processed for HTML special entities and emojis too using the processText function.

case 'websearch-rss':{

let htmlText = entities.decode(text);
processedText = <Emojify>{htmlText}</Emojify>;
break;

}

We now display our result tiles as a carousel like swipeable display using react-slick. We initialise our slider with few default options specifying the swipe speed and the slider UI.

import Slider from 'react-slick';

// Render Websearch RSS tiles

export function renderTiles(tiles){

  if(tiles.length === 0){
    let noResultFound = 'NO Results Found';
    return(<center>{noResultFound}</center>);
  }

  let resultTiles = drawTiles(tiles);
  
  var settings = {
    speed: 500,
    slidesToShow: 3,
    slidesToScroll: 1,
    swipeToSlide:true,
    swipe:true,
    arrows:false
  };

  return(
    <Slider {...settings}>
      {resultTiles}
    </Slider>
  );
}

We finally add CSS attributes to style our result tile and add overflow for text maintaining standard width for all tiles.We also add some CSS for our carousel display to show multiple tiles instead of one by default. This is done by adding some margin for child components in the slider.

.slick-slide{
  margin: 0 10px;
}

.slick-list{
  max-height: 100px;
}

We finally have our swipeable display of rss data tiles each tile hyperlinked to the source of the data. When the user clicks on a tile, he is redirected to the link in a new window i.e the entire tile is hyperlinked. And when there are no results to display, we show a `NO Results Found` message.

The complete code can be found at SUSI WebChat Repo. Feel free to contribute

Resources

 

Landing Page for Phimpme Android

Landing page for any app is very important for its distribution. Its provide all the relevant informations for the app, its download/ source link and screenshot of the app for preview of its view and features. As our Phimpme app is now reached to a milestone, so we decided to bring out a landing for it.

Landing page is a simple static website which gives relevant informations and their links on a page. To develop a landing page, we can use multiple frameworks available for it. For example using Bootstrap, basic pages using html and css. As GitHub provides a free hosting service such as Github pages. I decided to work on jekyll which is easy to customize and hosted with GitHub pages.

How I did in Phimpme

  • Search for Open Source theme

There are various open source themes present for jekyll, mainly you will get for blogs. I got this awesome app landing page. It fulfils our requirements of showing a screenshot, brief description and links.

  • Create separate branch for Landing page

The GitHub pages works from a separate branch. So I firstly created a orphan branch in main Phimpme Repository. And Clear the working directory. Orphan branches are those branch which do not have any commit history. It is highly required that branch should not have any commit history, because that will irrelevant for gh-pages branch.

git checkout --orphan gh-pages
git rm --cached -r .
  • Test on Fork repo

Now rebase your branch with the gh-pages branch created on the main repository. Add the complete jekyll code here. To reflect changes on github.io page. You need to enable that in settings.

  • Enable gh-pages from settings

Go to the Settings of your repo

Scroll down to gh-pages section

Here things are clear, as we need to select the source branch from where the gh-pages is hosted. We can directly choose some open source themes built over jekyll. Also there is a option to add custom domain, such phimp.me in our case.

This all I’m writing as per the Developer perspective, where the developer have limited access to the repository and can’t access the settings of main repo. In that case push your code to by selecting the gh-pages base branch.

  • Problems I faced

The usual problem any Developer here faces is creating a orphan branch in main repo. This required removing all the files as well. Also to face less problem I recommend using some other editor such as Sublime Text, Atom etc. Or any other IDE other than Android Studio. Because in Android Studio after changing branch, it sync and rebuild again which actually takes lots of time.

Resources

 

Documenting Open Event API Using API-Blueprint

FOSSASIA‘s Open Event Server API documentation is done using an api-blueprint. The API Blueprint language is a format used to describe API in an API blueprint file, where a blueprint file (or a set of files) is such that describes an API using the API Blueprint language. To follow up with the blueprint, an apiary editor is used. This editor is responsible for rendering the API blueprint and printing the result in user readable API documented format. We create the API blueprint manually.

Using API Blueprint:-
We create the API blueprint by first adding the name and metadata for the API we aim to design. This step looks like this :-

FORMAT: V1
HOST: https://api.eventyay.com

# Open Event API Server

The Open Event API Server

# Group Authentication

The API uses JWT Authentication to authenticate users to the server. For authentication, you need to be a registered user. Once you have registered yourself as an user, you can send a request to get the access_token.This access_token you need to then use in Authorization header while sending a request in the following manner: `Authorization: JWT <access_token>`


API blueprint starts with the metadata, here FORMAT and HOST are defined metadata. FORMAT keyword specifies the version of API Blueprint . HOST defines the host for the API.

The heading starts with # and the first heading is regarded as the name of the API.

NOTE – Also all the heading starts with one or more # symbol. Each symbol indicates the level of the heading. One # symbol followed by heading serves as the top level i.e. one # = Top Level. Similarly for  ## = second level and so on. This is in compliance with normal markdown format.
        Following the heading section comes the description of the API. Further, headings are used to break up the description section.

Resource Groups:
—————————–
    By using group keyword at the starting of a heading , we create a group of related resources. Just like in below screenshot we have created a Group Users.

# Group Users

For using the API you need(mostly) to register as an user. Registering gives you access to all non admin API endpoints. After registration, you need to create your JWT access token to send requests to the API endpoints.


| Parameter | Description | Type | Required |
|:----------|-------------|------|----------|
| `name`  | Name of the user | string | - |
| `password` | Password of the user | string | **yes** |
| `email` | Email of the user | string | **yes** |

 

Resources:
——————
    In the Group Users we have created a resource Users Collection. The heading specifies the URI used to access the resource inside of the square brackets after the heading. We have used here parameters for the resource URI which takes us into how to add parameters to the URI. Below code shows us how to add parameters to the resource URI.

## Users Collection [/v1/users{?page%5bsize%5d,page%5bnumber%5d,sort,filter}]
+ Parameters
    + page%5bsize%5d (optional, integer, `10`) - Maximum number of resources in a single paginated response.
    + page%5bnumber%5d (optional, integer, `2`) - Page number to fetchedfor the paginated response.
    + sort (optional, string, `email`) - Sort the resources according to the given attribute in ascending order. Append '-' to sort in descending order.
    + filter(optional, string, ``) - Filter according to the flask-rest-jsonapi filtering system. Please refer: http://flask-rest-jsonapi.readthedocs.io/en/latest/filtering.html for more.

 

Actions:
————–
    An action is specified with a sub-heading inside of  a resource as the name of Action, followed by HTTP method inside the square brackets.
    Before we get on further, let us discuss what a payload is. A payload is an HTTP transaction message including its discussion and any additional assets such as entity-body validation schema.

There are two payloads inside an Action:

  1. Request: It is a payload containing one specific HTTP Request, with Headers and an optional body.
  2. Response: It is a payload containing one HTTP Response.

A payload may have an identifier-a string for a request payload or an HTTP status code for a response payload.

+ Request

    + Headers

            Accept: application/vnd.api+json

            Authorization: JWT <Auth Key>

+ Response 200 (application/vnd.api+json)


Types of HTTP methods for Actions:

  • GET – In this action, we simply send the header data like Accept and Authorization and no body. Along with it we can send some GET parameters like page[size]. There are two cases for GET: List and Detail. For example, if we consider users, a GET for List helps us retrieve information about all users in the response, while Details, helps us retrieve information about a particular user.

The API Blueprint examples implementation of both GET list and detail request and response are as follows.

### List All Users [GET]
Get a list of Users.

+ Request

    + Headers

            Accept: application/vnd.api+json

            Authorization: JWT <Auth Key>

+ Response 200 (application/vnd.api+json)

        {
            "meta": {
                "count": 2
            },
            "data": [
                {
                    "attributes": {
                        "is-admin": true,
                        "last-name": null,
                        "instagram-url": null,

 

### Get Details [GET]
Get a single user.

+ Request

    + Headers

            Accept: application/vnd.api+json

            Authorization: JWT <Auth Key>

+ Response 200 (application/vnd.api+json)

        {
            "data": {
                "attributes": {
                    "is-admin": false,
                    "last-name": "Doe",
                    "instagram-url": "http://instagram.com/instagram",

 

  • POST – In this action, apart from the header information, we also need to send a data. The data must be correct with jsonapi specifications. A POST body data for an users API would look something like this:
### Create User [POST]
Create a new user using an email, password and an optional name.

+ Request (application/vnd.api+json)

    + Headers

            Authorization: JWT <Auth Key>

    + Body

            {
              "data":
              {
                "attributes":
                {
                  "email": "example@example.com",
                  "password": "password",


A POST request with this data, would create a new entry in the table and then return in jsonapi format the particular entry that was made into the table along with the id assigned to this new entry.

  • PATCH – In this action, we change or update an already existing entry in the database. So It has a header data like all other requests and a body data which is almost similar to POST except that it also needs to mention the id of the entry that we are trying to modify.
### Update User [PATCH]
+ `id` (integer) - ID of the record to update **(required)**

Update a single user by setting the email, email and/or name.

Authorized user should be same as user in request body or must be admin.

+ Request (application/vnd.api+json)

    + Headers

            Authorization: JWT <Auth Key>

    + Body

            {
              "data": {
                "attributes": {
                  "password": "password1",
                  "avatar_url": "http://example1.com/example1.png",
                  "first-name": "Jane",
                  "last-name": "Dough",
                  "details": "example1",
                  "contact": "example1",
                  "facebook-url": "http://facebook.com/facebook1",
                  "twitter-url": "http://twitter.com/twitter1",
                  "instagram-url": "http://instagram.com/instagram1",
                  "google-plus-url": "http://plus.google.com/plus.google1",
                  "thumbnail-image-url": "http://example1.com/example1.png",
                  "small-image-url": "http://example1.com/example1.png",
                  "icon-image-url": "http://example1.com/example1.png"
                },
                "type": "user",
                "id": "2"
              }
            }

Just like in POST, after we have updated our entry, we get back as response the new updated entry in the database.

  • DELETE – In this action, we delete an entry from the database. The entry in our case is soft deleted by default. Which means that instead of deleting it from the database, we set the deleted_at column with the time of deletion. For deleting we just need to send header data and send a DELETE request to the proper endpoint. If deleted successfully, we get a response as “Object successfully deleted”.
### Delete User [DELETE]
Delete a single user.

+ Request

    + Headers

            Accept: application/vnd.api+json

            Authorization: JWT <Auth Key>

+ Response 200 (application/vnd.api+json)

        {
          "meta": {
            "message": "Object successfully deleted"
          },
          "jsonapi": {
            "version": "1.0"
          }
        }


How to check after manually entering all these? We can use the
apiary website to render it, or simply use different renderer to do it. How? Checkout for my next blog on apiary and aglio.

Learn more about api blueprint here: https://apiblueprint.org/

Creating Multiple Device Compatible Layouts in PSLab Android

The developer’s goal is that PSLab Android App as an app should run smoothly on all the variety of Android devices out in the market. There are two aspects of it – the app should be able to support maximum number of Android versions possible which is related to the core software part and the other being the app should be able to generate the same user experience on all sizes of screens. This post focuses on the later.

There are a whole range of android devices available in the market right from 4 inch mobile phones to 12 inch tablets and the range in the screen sizes is quite large. So, the challenge in front of app designers is to make the app compatible with the maximum  number of devices without doing any specific tweaks related to a particular resolution range. Android has its mechanism of scaling the app as per the screen size and it does a good job almost all the time, however, still there are cases where android fails to scale up or scale down the app leading to distorted layout of the app.

This blog discusses some of the tricks that needs to be kept in mind while designing layouts that work independent of screen sizes.

Avoid using absolute dimensions

It is one of the most common things to keep in mind before starting any UI design. Use of absolute dimensions like px, inch etc. must be avoided every time as they are fixed in size and don’t scale up or scale down while screen sizes are changed. Instead relative dimensions like dp should be used which depend on the resolution and scale up or scale down. ( It’s a fair assumption that bigger screens will have better resolution compared to the smaller ones although exceptions do exist) .

Ensure the use of correct layout/View group

Since, android provides a variety of layouts like Linearlayout, Constrainedlayout, Relativelayout, Tablelayout and view groups like ScrollView, RecyclerView, ListView etc. it is often confusing to know which layout/viewgroup should be used. The following list gives a rough idea of when to use a particular layout or view group.

  • Linearlayout – Mostly used for simple designs when the elements are stacked in ordered horizontal/vertical fashion and it needs explicit declaration of orientation.
  • Relativelayout – Mostly used when the elements need to defined relative to the parent or the neighbouring elements. Since, the elements are relative, there is no need to define the orientation.
  • Constraintlayout – It has all the features of Relativelayout and in addition a feature of adding constraints to the child elements or neighbouring elements.
  • Tablelayout – Tablelayout is helpful to when all the views/widgets are arranged in an ordered fashion.

All the above layouts can be used interchangeably most of the times, however, certain cases make some more favourable than others like when than views/ widgets are not present in an organised manner, it is better to stick to Linearlayout or Relativelayout.

  • ListView – Used when the views/ widgets in a screen are repeated, so using a listview ensures that the volume of the code is reduced and all the repetitive views are identical in nature.
  • RecyclerView – More of an improved version of ListView. It is recommended to use this view over ListView. Additionally this view group supports features like swipe to refresh.
  • ScrollView – Used when the UI screen cannot fit within the given screen space. ScrollView supports one direct child layout. So, to implement a scrollview, all the views must be under a particular layout and then masked by scrollview.

Choosing the correct layout or view group would help to create a better UI.

Use of layout_weight

Ensuring the layout width assigned in XML file covers the entire width on the screen. For ensuring this, one possible solution is to use layout_weight instead of layout_width.

Example –

<TextView
   android:id="@+id/tv_control_read9"
   android:layout_width="0dp"
   android:layout_weight="1"
   android:layout_height="30dp"
   android:layout_marginTop="10dp"
/>

 

In order to use layout_weight, layout_width must be set to 0 else it would interfere with the width and as layout_width is a compulsory parameter it cannot be omitted. Layout weight can be any number and the space is allocated to each view in proportion to the weights assigned. Since it does not involve numerical dimensions, the distribution would be uniform for all types of screens. The result is clearly evident here. The same UI in different screen sizes is displayed below.

blog_post_8_2

Fig: Screenshot taken on a 6” phone and on a 4” phone. Although the screen area of 4” phone is 44% that of the 6” phone, the UIs are identically the same.

Create different layout directories for different resolutions

  • Creating different layouts for different screen sizes ensures that the limitations of smaller screen sizes are taken care of and the advantages offered by bigger screen sizes are put to the best use.
  • The Android documentation here mentions the conventions to be followed while designing.
  • Although over the years, android has become better at auto-adjusting layouts for different screen sizes. However, if the no. of views and widgets are high, auto-adjusting does not work well as in case of PSLab and it is better to create different sets of layouts.
  • As evident from the picture of the 8” tablet, although the auto-adjusted layout is manageable, the layout looks stretched and does not utilize the entire screen space, so it a better UI can be made by creating a dedicated layout directory for bigger screens.

Additional resources

 

Using Sensors with PSLab Android App

The PSLab Android App as of now supports quite a few sensors. Sensors are an essential part of many science experiments and therefore PSLab has a feature to support plug & play sensors. The list of sensors supported by PSLab can be found here.

  • AD7718 – 24-bit 10-channel Low voltage Low power Sigma Delta ADC
  • AD9833 – Low Power Programmable Waveform generator
  • ADS1115 – Low Power 16 bit ADC
  • BH1750 – Light Intensity sensor
  • BMP180 – Digital Pressure Sensor
  • HMC5883L – 3-axis digital magnetometer
  • MF522 – RFID Reader
  • MLX90614 – Infrared thermometer
  • MPU6050 – Accelerometer & gyroscope
  • MPU925x – Accelerometer & gyroscope
  • SHT21 – Humidity sensor
  • SSD1306 – Control for LED matrix
  • Sx1276 – Low Power Long range Transceiver
  • TSL2561 – Digital Luminosity Sensor

All the sensors except Sx1276 communicate using the I2C protocol whereas the Sx1276 uses the SPI protocol for communication. There is a dedicated set of ports on the PSLab board for the communication under the label I2C with the ports named 3.3V, GND, SCL & SDA.

blog_post_7_1

Fig; PSLab board sketch

Any I2C sensor has ports named 3.3V/VCC, GND, SCL, SDA at least along with some other ports in some sensors. The connections are as follows:

  1. 3.3V on PSLab – 3.3V/VCC on sensor
  2. GND on PSLab – GND on sensor
  3. SCL on PSLab – SCL on sensor
  4. SDA on PSLab – SDA on sensor

The diagram here shows the connections

For using the sensors with the Android App, there is a dedicated I2C library written in communication in Java for the communication. Each sensor has its own specific set of functionalities and therefore has its own library file. However, all these sensors share some common features like each one of them has a getRaw method which fetches the raw sensor data. For getting the data from a sensor, the sensor is initially connected to the PSLab board.

The following piece of code is responsible for detecting any devices that are connected to the PSLab board through the I2C bus. Each sensor has it’s own unique address and can be identified using it. So, the AutoScan function returns the addresses of all the connected sensors and the sensors can be uniquely identified using those addresses.

public ArrayList<Integer> scan(Integer frequency) throws IOException {
	if (frequency == null) frequency = 100000;
	config(frequency);
	ArrayList<Integer> addresses = new ArrayList<>();
	for (int i = 0; i < 128; i++) {
		int x = start(i, 0);
		if ((x & 1) == 0) {
			addresses.add(i);
		}
		stop();
	}
	return addresses;
}

 

As per the addresses fetched, the sensor library corresponding to that particular sensor can be imported and the getRaw method can be called. The getRaw method will return the raw sensor data. For example here is the getRaw method of ADS1115.

public int[] getRaw() throws IOException, InterruptedException {
	String chan = typeSelection.get(channel);
	if (channel.contains("UNI"))
		return new int[]{(int) readADCSingleEnded(Integer.parseInt(chan))};
	else if (channel.contains("DIF"))
		return new int[]{readADCDifferential(chan)};
	return new int[0];
}

Here the raw data is returned in the form of voltages in mV.

Similarly, the other sensors return some values like luminosity sensor TSL2561 returns values of luminosity in Lux, the accelerometer & gyroscope MPU6050 returns the angles of the 3-axes.

In order to initiate the process of getting raw data from the sensor in Sensor Activity, the object for the sensor is created and the method of getRaw is called. The following is the implementation for ADS1115. The rest of the sensors also have an implementation similar to this. There are try-catch statements in the code to handle some of the exceptions thrown during process of method calls.

ADS1115 ADS1115 = null;
try {
	ADS1115 = new ADS1115(i2c);
} catch (IOException | InterruptedException e) {
	e.printStackTrace();
}

int[] dataADS1115 = null;
String datadispADS1115 = null;
try {
	if (ADS1115 != null) {
		dataADS1115 = ADS1115.getRaw();
	}
} catch (IOException | InterruptedException e) {
	e.printStackTrace();
}

if (dataADS1115 != null) {
	for(int i = 0; i < dataADS1115.length; i++)
		datadispADS1115 += String.valueOf(dataADS1115[i]);
	}

tvSensorGetRaw.setText(datadispADS1115);

 

Additional Resources

  1. Sensor implementation in PSLab Python repository – https://github.com/fossasia/pslab-python/tree/development/PSL/SENSORS
  2. Using the sensors with Arduino in case you have worked with Arduino before – The basic connections are same as PSLab http://www.instructables.com/id/Arduino-MPU-6050-Getting-It-to-Work/

Creating Custom Components in the PSLab Android App

PSLab Android App supports a lot of features and each of these features need components & views for their implementation. A typical UI of PSLab is shown in the figure below. Considering the number of views & components used in the figure, implementation of each view & component separately would lead to a huge volume of repetitive and inefficient code. As it is evident that the EditText and two buttons beside it keep repeating a lot, it is wiser to create a single custom component consisting of an EditText and two buttons. This not only leads to efficient code but also results in a drastic reduction of the volume of code.

Android has a feature which allows creating components. For almost all the cases, the pre-defined views in Android serve our purpose of creating the UIs. However, sometimes there is a need to create custom components to reduce code volume and improve quality. Custom components are used when a particular set of component needed by us is not present in the Android view collection or when a pattern of components is frequently repeated or when we need to reduce the code complexity.

The above set can be replaced by defining a custom component which includes an edittext and two buttons and then treating it like just any other component. To get started with creating a custom component, the steps are the following:

Create a layout for the custom component to be designed

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
   android:orientation="horizontal" android:layout_width="match_parent"
   android:layout_height="match_parent">

   <Button
       android:id="@+id/button_control_plus"
       android:layout_width="0dp"
       android:layout_weight="0.5"
       android:layout_height="20dp"
       android:background="@drawable/button_minus" />

   <EditText
       android:id="@+id/edittext_control"
       android:layout_width="0dp"
       android:layout_weight="2"
       android:layout_height="24dp"
       android:layout_marginTop="@dimen/control_margin_small"
       android:inputType="numberDecimal"
       android:padding="@dimen/control_edittext_padding"
       android:background="@drawable/control_edittext" />

   <Button
       android:id="@+id/button_control_minus"
       android:layout_width="0dp"
       android:layout_weight="0.5"
       android:layout_height="20dp"
       android:background="@drawable/button_plus" />
</LinearLayout>

The layout file edittext_control.xml is created with three views and each one of them has been assigned an ID along with all the other relevant parameters.

Incorporate the newly created custom layout in the Activity/Fragment layout file

<org.fossasia.pslab.others.Edittextwidget
       android:id="@+id/etwidget_control_advanced1"
       android:layout_height="wrap_content"
       android:layout_width="0dp"
       android:layout_weight="2"
       android:layout_marginLeft="@dimen/control_margin_small"
       android:layout_marginStart="@dimen/control_margin_small"
/>

The custom layout can be added the activity/fragment layout just like any other view and can be assigned properties similarly.

Create the activity file for the custom layout

public class Edittextwidget extends LinearLayout{

   private EditText editText;
   private Button button1;
   private Button button2;
   private double leastCount;
   private double maxima;
   private double minima;

 
   public Edittextwidget(Context context, AttributeSet attrs, int defStyle) {
       super(context, attrs, defStyle);
       applyAttrs(attrs);
   }

   public Edittextwidget(Context context, AttributeSet attrs) {
       super(context, attrs);
       applyAttrs(attrs);
   }

   public Edittextwidget(Context context) {
       super(context);
   }

  public void init(Context context, final double leastCount, final double minima, final double maxima) {
       View.inflate(context, R.layout.edittext_control, this);
       editText = (EditText) findViewById(R.id.edittext_control);
       button1 = (Button) findViewById(R.id.button_control_plus);
       button2 = (Button) findViewById(R.id.button_control_minus);

       button1.setOnClickListener(new OnClickListener() {
           @Override
           public void onClick(View v) {
               Double data = Double.valueOf(editText.getText().toString());
               data = data - leastCount;
               data = data > maxima ? maxima : data;
               data = data < minima ? minima : data;
               editText.setText(String.valueOf(data));
           }
       });

       button2.setOnClickListener(new OnClickListener() {
           @Override
           public void onClick(View v) {
               Double data = Double.valueOf(editText.getText().toString());
               data = data + leastCount;
               data = data > maxima ? maxima : data;
               data = data < minima ? minima : data;
               editText.setText(String.valueOf(data));
           }
       });
   }

   private void applyAttrs(AttributeSet attrs) {
       TypedArray a = getContext().obtainStyledAttributes(attrs, R.styleable.Edittextwidget);
       final int N = a.getIndexCount();
       for (int i = 0; i < N; ++i) {
           int attr = a.getIndex(i);
           switch (attr) {
               case R.styleable.Edittextwidget_leastcount:
                   this.leastCount = a.getFloat(attr, 1.0f);
                   break;
               case R.styleable.Edittextwidget_maxima:
                   this.maxima = a.getFloat(attr, 1.0f);
                   break;
               case R.styleable.Edittextwidget_minima:
                   this.minima = a.getFloat(attr, 1.0f);
           }
       }
       a.recycle();
   }
}

In the activity file Editextwidget.java, the views of the custom layout are defined and functionalities are assigned to them. For example, here there are two buttons which work as increment/decrement buttons and an edittext which takes numeric input. The buttons are initiated just like the way they are done in other activity/fragment using OnClickListener.

Define the attributes for the custom layout

<declare-styleable name="Edittextwidget">
     <attr name="leastcount" format="float" />
     <attr name="maxima" format="float" />
     <attr name="minima" format="float" />
</declare-styleable>

The attributes for the custom layout are defined in the attrs.xml file. Each attribute is assigned a name and a format which can be int, float, double, string etc.

Finally call the methods of the custom layout from the desired activity/fragment

Edittextwidget etwidgetControlAdvanced1 = (Edittextwidget)view.findViewById(R.id.etwidget_control_advanced1);

etwidgetControlAdvanced1.init(getContext(), 1.0, 10.0, 5000.0);

The init method of Edittextwidget.java is called while passing the relevant parameters like context, least count, maxima and minima.

Additional Resources on Custom Components

  1. Official Android Guide on Custom components – https://developer.android.com/guide/topics/ui/custom-components.html
  2. Simple example of creating a custom component to get started – https://www.tutorialspoint.com/android/android_custom_components.htm