Caching Elasticsearch Aggregation Results in loklak Server

To provide aggregated data for various classifiers, loklak uses Elasticsearch aggregations. Aggregated data speaks a lot more than a few instances from it can say. But performing aggregations on each request can be very resource consuming. So we needed to come up with a way to reduce this load.

In this post, I will be discussing how I came up with a caching model for the aggregated data from the Elasticsearch index.

Fields to Consider while Caching

At the classifier endpoint, aggregations can be requested based on the following fields –

  • Classifier Name
  • Classifier Classes
  • Countries
  • Start Date
  • End Date

But to cache results, we can ignore cases where we just require a few classes or countries and store aggregations for all of them instead. So the fields that will define the cache to look for will be –

  • Classifier Name
  • Start Date
  • End Date

Type of Cache

The data structure used for caching was Java’s HashMap. It would be used to map a special string key to a special object discussed in a later section.

Key

The key is built using the fields mentioned previously –

private static String getKey(String index, String classifier, String sinceDate, String untilDate) {
    return index + "::::"
        + classifier + "::::"
        + (sinceDate == null ? "" : sinceDate) + "::::"
        + (untilDate == null ? "" : untilDate);
}

[SOURCE]

In this way, we can handle requests where a user makes a request for every class there is without running the expensive aggregation job every time. This is because the key for such requests will be same as we are not considering country and class for this purpose.

Value

The object used as key in the HashMap is a wrapper containing the following –

  1. json – It is a JSONObject containing the actual data.
  2. expiry – It is the expiry of the object in milliseconds.

class JSONObjectWrapper {
    private JSONObject json; 
    private long expiry;
    ... 
}

Timeout

The timeout associated with a cache is defined in the configuration file of the project as “classifierservlet.cache.timeout”. It defaults to 5 minutes and is used to set the eexpiryof a cached JSONObject –

class JSONObjectWrapper {
    ...
    private static long timeout = DAO.getConfig("classifierservlet.cache.timeout", 300000);

    JSONObjectWrapper(JSONObject json) {
        this.json = json;
        this.expiry = System.currentTimeMillis() + timeout;
    }
    ...
}

 

Cache Hit

For searching in the cache, the previously mentioned string is composed from the parameters requested by the user. Checking for a cache hit can be done in the following manner –

String key = getKey(index, classifier, sinceDate, untilDate);
if (cacheMap.keySet().contains(key)) {
    JSONObjectWrapper jw = cacheMap.get(key);
    if (!jw.isExpired()) {
        // Do something with jw
    }
}
// Calculate the aggregations
...

But since jw here would contain all the data, we would need to filter out the classes and countries which are not needed.

Filtering results

For filtering out the parts which do not contain the information requested by the user, we can perform a simple pass and exclude the results that are not needed.

Since the number of fields to filter out, i.e. classes and countries, would not be that high, this process would not be that resource intensive. And at the same time, would save us from requesting heavy aggregation tasks from the user.

Since the data about classes is nested inside the respective country field, we need to perform two level of filtering –

JSONObject retJson = new JSONObject(true);
for (String key : json.keySet()) {
    JSONArray value = filterInnerClasses(json.getJSONArray(key), classes);
    if ("GLOBAL".equals(key) || countries.contains(key)) {
        retJson.put(key, value);
    }
}

Cache Miss

In the case of a cache miss, the helper functions are called from ElasticsearchClient.java to get results. These results are then parsed from HashMap to JSONObject and stored in the cache for future usages.

JSONObject freshCache = getFromElasticsearch(index, classifier, sinceDate, untilDate);
cacheMap.put(key, new JSONObjectWrapper(freshCache));

The getFromElasticsearch method finds all the possible classes and makes a request to the appropriate method in ElasticsearchClient, getting data for all classifiers and all countries.

Conclusion

In this blog post, I discussed the need for caching of aggregations and the way it is achieved in the loklak server. This feature was introduced in pull request loklak/loklak_server#1333 by @singhpratyush (me).

Resources

Continue ReadingCaching Elasticsearch Aggregation Results in loklak Server

Upload Images to OwnCloud and NextCloud in Phimpme Android

As increasing the stack of account manager in Phimpme Android. We have now two new items OwnCloud and NextCloud to add. Both are open source storage services. Provides complete source code of their official apps and libraries on Github. You can check below

OwnCloud: https://github.com/owncloud

NextCloud: https://github.com/nextcloud

This requires a hosting server, where you can deploy it and access it through their web app and Mobile apps. I added a feature in Phimpme to upload images directly to the server right from the app using their android-library.

Steps (How I did in Phimpme)

  • Add library in Application gradle file

Firstly, to work with, we need to add the android-library they provide.

compile "com.github.nextcloud:android-library:$rootProject.nextCloudVersion"

Check the new version from here and apply over it: https://github.com/nextcloud/android-library/releases

  • Login from Account Manager

As per our Phimpme app flow, User first connect itself from the account manager and then share image from app using these credentials. Added a new Login activity for OwnCloud and NextCloud both.

          

  • Saved credentials in Database

To use that further in android-library, I store the credentials in Realm database.

account.setServerUrl(data.getStringExtra(getString(R.string.server_url)));
account.setUsername(data.getStringExtra(getString(R.string.auth_username)));
account.setPassword(data.getStringExtra(getString(R.string.auth_password)));
  • Uploading image using library

As per the official guide of OwnCloud, used Created an object of OwnCloudClient. Set the username and password.

private OwnCloudClient mClient;
mClient = OwnCloudClientFactory.createOwnCloudClient(serverUri, this, true);
mClient.setCredentials(
       OwnCloudCredentialsFactory.newBasicCredentials(
               username,
               password
       )
);

Passed the image path which we are getting in the SharingActivity. Modified with adding the separator.

File fileToUpload = new File(saveFilePath);
String remotePath = FileUtils.PATH_SEPARATOR + fileToUpload.getName();

Used the UploadRemoteOperation Class and just need to pass the path, mimeType and timeStamp. The library have already defined functions to execute the upload operations.

UploadRemoteFileOperation uploadOperation =
       new UploadRemoteFileOperation(fileToUpload.getAbsolutePath(), remotePath, mimeType, timeStamp);
uploadOperation.execute(mClient, this, mHandler);

  • Setup Account using Docker and Digital Ocean

I have already a previous blog post on how to setup NextCloud or OwnCloud account on server using Digital Ocean and Docker.

Link: https://blog.fossasia.org/how-to-use-digital-ocean-and-docker-to-setup-test-cms-for-phimpme/

Resource:

  1. NextCloud Developer Mannual: https://docs.nextcloud.com/server/9/developer_manual/index.html
  2. OwnCloud Library installation: https://doc.owncloud.org/server/9.0/developer_manual/android_library/library_installation.html
  3. Examples: https://doc.owncloud.org/server/9.0/developer_manual/android_library/examples.html
Continue ReadingUpload Images to OwnCloud and NextCloud in Phimpme Android

Common Utility classes Progress Bar and Snack Bar in Phimpme Android

As the Phimpme Android is scaling very fast on its features, code gets redundant sometimes. Some of the widely used design widgets in Android are Progress Bar and Snack Bar. Progress Bar is shown to user when some process is happening in the background. Snackbar is a feedback operation to user of its recent process. In other words we can say Snackbar is the new toast in Android with a cool feature of setting action on them. So that User can interact with the feedback received on the process.

As In Phimpme lots of account Login and Logout progress happens. Uploading success and failure required Snackbar to show to the Users. So to remove the redundancy of the boilerplate of these codes, I added two Utilities class one is Phimpme ProgressbarHandler and other is SnackbarHandler in the app. Below is one by one code and explanation of both.

Progress Bar Handler

In the constructor I passed Context as parameter. Created a ViewGroup object and set view of android. Setting the progress bar style and length using Android core attributes such as progressBarStyleLarge and duration to setIndeterminate true.

private ProgressBar mProgressBar;

public PhimpmeProgressBarHandler(Context context) {
   ViewGroup layout = (ViewGroup) ((Activity) context).findViewById(android.R.id.content)
           .getRootView();

   mProgressBar = new ProgressBar(context, null, android.R.attr.progressBarStyleLarge);
   mProgressBar.setIndeterminate(true);



   RelativeLayout.LayoutParams params = new
           RelativeLayout.LayoutParams(RelativeLayout.LayoutParams.MATCH_PARENT,
           RelativeLayout.LayoutParams.MATCH_PARENT);

   RelativeLayout rl = new RelativeLayout(context);

   rl.setGravity(Gravity.CENTER);
   rl.addView(mProgressBar);

   layout.addView(rl, params);

   hide();

}

Next is used dynamically created Relative Layout object and setup the parameters for width and height as MATCH_PARENT. Setting gravity of the layout to center and added the progress bar view on it using the addView method. So basically we have a progress bar ready and we dynamically created a relative layout and added the view over it.

The function used in setting up the views and progress bar are from AOSP only.

After that a Progressbar is set, we now need functions to show and hide the progress bar in the code. Created two functions show() and hide().

public void show() {
   mProgressBar.setVisibility(View.VISIBLE);
}

public void hide() {
   mProgressBar.setVisibility(View.INVISIBLE);
}

These functions set the visibility of the the progress bar.

Usage:

Now in any class we can create object of our Progressbar handler class pass the context on it and use the show() and hide() methods wherever we want to show this and hide. Below is the code snippet to show the illustration.

phimpmeProgressBarHandler = new PhimpmeProgressBarHandler(this);

phimpmeProgressBarHandler.show();

phimpmeProgressBarHandler.hide();

Snackbar Handler

To do this, I created a separate class as Snackbar Handler. What we can do is to create a static function show() and inside the declaration, we can create an object of Snackbar and apply the styles to that.

As you can see in the code snippet below, I Created a static function with parameters such as View (to take the view instance), String (to show the message) and duration  of the Snackbar. Set Up the text, textsize and action on the snackbar. An “OK” action is predefined in the function only.

public static void show(View view, String text, int duration) {
   final Snackbar snackbar = Snackbar.make(view, text, duration);
   View sbView = snackbar.getView();
   TextView textView = (TextView)sbView.findViewById(android

.support.design.R.id.snackbar_text);
   textView.setTextColor(Color.WHITE);
   textView.setTextSize(12);
   snackbar.setAction("OK", new View.OnClickListener() {
       @Override
       public void onClick(View view) {
           snackbar.dismiss();
       }
   });
   snackbar.show();
}

Usage:

To use this directly call the show method pass the view and String of the message which you want to show on Snackbar. There are overloaded methods as well in which you can pass the durations. See the below code as example.

SnackBarHandler.show(parentLayout, getString(R.string.no_account_signed_in));

Resources

Continue ReadingCommon Utility classes Progress Bar and Snack Bar in Phimpme Android

Use of Flux Architecture to Switch between Themes in SUSI Web Chat

While we were developing the SUSI Web Chat we got a requirement to build a dark theme. And to build a way that user can switch between dark and light theme.

SUSI Web Chat application is made according to the Flux architecture, So we had to build sub components according to that architecture.

What is flux:

Flux is not a framework. It is an Architecture/pattern that we can use to build applications using react and some other frameworks. Below figure shows the way how that architecture works and how it communicate.

How flux works:

Flux has four types of components. Those are views, actions, dispatcher and stores. We use JSX to build and integrate views into our JavaScript code.

When someone triggers an event from view, it triggers an action and that action sends particular action details  such as Actiontype, action name  and data to dispatcher. Dispatcher broadcasts those details to every store which are registered with the particular dispatcher. That means every store gets all the action details and data which are broadcasting from dispatchers which they are registered.

Let’s say we have triggered an action from view that is going to change the value of the store. Those action details are coming to dispatcher. Then dispatcher broadcasts those data to every store that registered with it. Matching action will trigger and update its value. If there is any change happened in store, view automatically updates corresponding view.

How themes are changing:

We have a store called “SettingStore.js”. This “SettingStore” contains theme values like current theme. We store other settings of the application such as: Mic input settings, Custom server details, Speech Output details, Default Language, etc.it is responsible to provide these details to corresponding view.

let CHANGE_EVENT = 'change';
class SettingStore extends EventEmitter {
   constructor() {
       super();
       this.theme = true; 
   }

We use “this.theme = true” in constructor to switch light theme as the default theme.

getTheme() { //provides current value of theme
       return this.theme;
   }

This method returns the value of the current theme when it triggers.

   changeTheme(themeChanges) {
       this.theme = !this.theme;
       this.emit(CHANGE_EVENT);
   }

We use “changeTheme” method to change the selected theme.

   handleActions(action) {
       switch (action.type) {
           case ActionTypes.THEME_CHANGED: {
               this.changeTheme(action.theme);
               break;
           }
           default: {
               // do nothing
           }
       }
   }
}

This switch case is the place that store gets actions distributed from the dispatcher and executes relevant method.

const settingStore = new SettingStore();
ChatAppDispatcher.register(settingStore.handleActions.bind(settingStore));
export default settingStore;

Here we registered our store(SettingStore) to “ChatAppDispatcher” .

This is how Store works.
Now we need to get the default light theme to the view. For that we have to call ”getTheme()” function. We can use the value it returns to update the state of the application.
Now we are going to change the theme. To do that we have to trigger “changeTheme” method of “Settingstrore” from view ( MessageSection.react.js ).
We trigger below method on click of the “Change Theme” button. It triggers the action called “themeChanged”.

 themeChanger(event) {
   Actions.themeChanged(!this.state.darkTheme);
 }

Previous method executes “themeChanged()” function of the actions.js file.

export function themeChanged(theme) {
 ChatAppDispatcher.dispatch({
   type: ActionTypes.THEME_CHANGED,
   theme //data came from parameter
 });
};

In above function we collect data from the view and send those data, method details to dispatcher.
Dispatcher sends those details to each and every registered store. In our case we have “SettingStore” and update the state to new dark theme.
This is how themes are changing in SUSI AI Web Chat application. Check this link to see the preview.

Resources:

  • Read About Flux: https://facebook.github.io/flux/
  • GitHub repository: https://github.com/fossasia/chat.susi.ai
  • Live Web Chat: http://chat.susi.ai/
Continue ReadingUse of Flux Architecture to Switch between Themes in SUSI Web Chat

Using Elasticsearch Aggregations to Analyse Classifier Data in loklak Server

Loklak uses Elasticsearch to index Tweets and other social media entities. It also houses a classifier that classifies Tweets based on emotion, profanity and language. But earlier, this data was available only with the search API and there was no way to get aggregated data out of it. So as a part of my GSoC project, I proposed to introduce a new API endpoint which would allow users to access aggregated data from these classifiers.

In this blog post, I will be discussing how aggregations are performed on the Elasticsearch index of Tweets in the loklak server.

Structure of index

The ES index for Twitter is called messages and it has 3 fields related to classifiers –

  1. classifier_emotion
  2. classifier_language
  3. classifier_profanity

With each of these classifiers, we also have a probability attached which represents the confidence of the classifier for assigned class to a Tweet. The name of these fields is given by suffixing the emotion field by _probability (e.g. classifier_emotion_probability).

Since I will also be discussing aggregation based on countries in this blog post, there is also a field named place_country_code which saves the ISO 3166-1 alpha-2 code for the country of creation of Tweet.

Requesting aggregations using Elasticsearch Java API

Elasticsearch comes with a simple Java API which can be used to perform any desired task. To work with data, we need an ES client which can be built from a ES Node (if creating a cluster) or directly as a transport client (if connecting remotely to a cluster) –

// Transport client
TransportClient tc = TransportClient.builder()
                        .settings(mySettings)
                        .build();

// From a node
Node elasticsearchNode = NodeBuilder.nodeBuilder()
                            .local(false).settings(mySettings)
                            .node();
Client nc = elasticsearchNode.client();

[SOURCE]

Once we have a client, we can use ES AggregationBuilder to get aggregations from an index –

SearchResponse response = elasticsearchClient.prepareSearch(indexName)
                            .setSearchType(SearchType.QUERY_THEN_FETCH)
                            .setQuery(QueryBuilders.matchAllQuery())  // Consider every row
                            .setFrom(0).setSize(0)  // 0 offset, 0 result size (do not return any rows)
                            .addAggregation(aggr)  // aggr is a AggregatoinBuilder object
                            .execute().actionGet();  // Execute and get results

[SOURCE]

AggregationBuilders are objects that define the properties of an aggregation task using ES’s Java API. This code snippet is applicable for any type of aggregation that we wish to perform on an index, given that we do not want to fetch any rows as a response.

Performing simple aggregation for a classifier

In this section, I will discuss the process to get results from a given classifier in loklak’s ES index. Here, we will be targeting a class-wise count of rows and stats (average and sum) of probabilities.

Writing AggregationBuilder

An AggregationBuilder for this task will be a Terms AggregationBuilder which would dynamically generate buckets for all the different values of fields for a given field in index –

AggregationBuilder getClassifierAggregationBuilder(String classifierName) {
    String probabilityField = classifierName + "_probability";
    return AggregationBuilders.terms("by_class").field(classifierName)
        .subAggregation(
            AggregationBuilders.avg("avg_probability").field(probabilityField)
        )
        .subAggregation(
            AggregationBuilders.sum("sum_probability").field(probabilityField)
        );
}

[SOURCE]

Here, the name of aggregation is passed as by_class. This will be used while processing the results for this aggregation task. Also, sub-aggregation is used to get average and sum probability by the name of avg_probability and sum_probability respectively. There is no need to specify to count rows as this is done by default.

Processing results

Once we have executed the aggregation task and received the SearchResponse as sr (say), we can use the name of top level aggregation to get Terms aggregation object –

Terms aggrs = sr.getAggregations().get("by_class");

After that, we can iterate through the buckets to get results –

for (Terms.Bucket bucket : aggrs.getBuckets()) {
String key = bucket.getKeyAsString();
long docCount = bucket.getDocCount(); // Number of rows
// Use name of sub aggregations to get results
Sum sum = bucket.getAggregations().get("sum_probability");
Avg avg = bucket.getAggregations().get("avg_probability");
// Do something with key, docCount, sum and avg
}

[SOURCE]

So in this manner, results from aggregation response can be processed.

Performing nested aggregations for different countries

The previous section described the process to perform aggregation over a single field. For this section, we’ll aim to get results for each country present in the index given a classifier field.

Writing a nested aggregation builder

To get the aggregation required, AggregationBuilder from previous section can be added as a sub-aggregation for the AggregationBuilder for country code field –

AggregationBuilder aggrs = AggregationBuilders.terms("by_country").field("place_country_code")
    .subAggregation(
        AggregationBuilders.terms("by_class").field(classifierName)
            .subAggregation(
                AggregationBuilders.avg("avg_probability").field(probabilityField)
            )
            .subAggregation(
                AggregationBuilders.sum("sum_probability").field(probabilityField)
            );
    );

[SOURCE]

Processing the results

Again, we can get the results by processing the AggregationBuilders by name in a top-to-bottom fashion –

Terms aggrs = response.getAggregations().get("by_country");
for (Terms.Bucket bucket : aggr.getBuckets()) {
    String countryCode = bucket.getKeyAsString();
    Terms classAggrs = bucket.getAggregations().get("by_class");
    for (Terms.Bucket classBucket : classAggr.getBuckets()) {
        String key = classBucket.getKeyAsString();
        long docCount = classBucket.getDocCount();
        Sum sum = classBucket.getAggregations().get("sum_probability");
        Avg avg = classBucket.getAggregations().get("avg_probability");
        ...
    }
    ...
}

[SOURCE]

And we have the data about classifier for each country present in the index.

Conclusion

This blog post explained about Elasticsearch aggregations and their usage in the loklak server project. The changes discussed here were introduced over a series of patches to ElasticsearchClient.java by @singhpratyush (me).

Resources

Continue ReadingUsing Elasticsearch Aggregations to Analyse Classifier Data in loklak Server

Setup SUSI Assistant on Raspberry Pi in under 30 minutes

With our ever growing list of list of platforms supported by Susi AI, we now have a client that can run on Raspberry Pi and you can access it hands-free!! Here is a video that you can refer for its working.

But it might have left you wondering how you can replicate such a setup yourself? It is fairly easy and will be done fairly easy. Just follow the following instructions.

You need to have following hardware in order to have your own SUSI Assistant running on Raspberry Pi.

  • A Raspberry Pi (prefer 2 or 3) with Raspbian Jessie OS.
  • A stable internet connection.  ( Recommended 4 Mbps )
  • A USB Microphone /  USB Webcam with Microphone. You may buy one like this.
  • A Speaker that connects through 3.5mm jack. You may buy one like this.

After you get all the above items in order, you need to get access to a terminal of your Raspberry Pi. You can have that by either connecting a monitor to Raspberry Pi temporarily or by connecting to Raspberry Pi over SSH.

Once this is done, next step is the installation of the dependencies. The installation of the SUSI on Raspberry is automated after dependencies are installed. Run the following command on Raspberry Pi terminal.

sudo apt install git swig3.0 portaudio19-dev pulseaudio libpulse-dev unzip sox libatlas-dev libatlas-base-dev libsox-fmt-all python3

After this, you may check if your output and input devices are working alright. For this, run rec recording.wav . It will start recording audio and saving it to a file named recording.wav. Play back the file using play recording.wav If you hear your audio clearly, setup is done right else you need to configure your Audio Devices correctly.  Most of the time the configuration of Audio works out the box and devices are plug and play so you would not encounter any errors. If you are successful in configuring your devices, install extra dependencies for SUSI Hardware by running the automated install script. In your terminal run,

$ git clone https://github.com/fossasia/susi_hardware.git
$ cd susi_hardware
$ ./install.sh 

This will install all the remaining dependencies. After the above step is complete, you may run configuration file generator script to choose the Text to Speech and Speech to Text service according to your wish. For doing so, you need to run

$ python3 config_generator.py

Follow the instructions in the script. It will ask you to configure the default service for Text to Speech and Speech to Text and other options. After the configuration is complete, you can simply run the following command to start SUSI.

$ python3 main.py

This will start SUSI in a continuously listening mode. You may invoke SUSI anytime, just by saying SUSI followed by a query. The query will be answered by SUSI subsequently.

Since configurations for different hardware devices may vary, you may encounter some problems. In such a scenario, you may refer to the following resources to solve the issues.

Resources:

Continue ReadingSetup SUSI Assistant on Raspberry Pi in under 30 minutes

How to Store and Retrieve User Settings from SUSI Server in SUSI iOS

Any user using the SUSI iOS client can set preferences like enabling or disabling the hot word recognition or enabling input from the microphone. These settings need to be stored, in order to be used across all platforms such as web, Android or iOS. Now, in order to store these settings and maintain a synchronization between all the clients, we make use of the SUSI server. The server provides an endpoint to retrieve these settings when the user logs in.

First, we will focus on storing settings on the server followed by retrieving settings from the server. The endpoint to store settings is as follows:

http://api.susi.ai/aaa/changeUserSettings.json?key=key&value=value&access_token=ACCESS_TOKEN

This takes the key value pair for storing a settings and an access token to identify the user as parameters in the GET request. Let’s start by creating the method that takes input the params, calls the API to store settings and returns a status specifying if the executed successfully or not.

 let url = getApiUrl(UserDefaults.standard.object(forKey: ControllerConstants.UserDefaultsKeys.ipAddress) as! String, Methods.UserSettings)

        _ = makeRequest(url, .get, [:], parameters: params, completion: { (results, message) in
            if let _ = message {
                completion(false, ResponseMessages.ServerError)
            } else if let results = results {
                guard let response = results as? [String : AnyObject] else {
                    completion(false, ResponseMessages.ServerError)
                    return
                }
                if let accepted = response[ControllerConstants.accepted] as? Bool, let message = response[Client.UserKeys.Message] as? String {
                    if accepted {
                        completion(true, message)
                        return
                    }
                    completion(false, message)
                    return
                }
            }
        })

Let’s understand this function line by line. First we generate the URL by supplying the server address and the method. Then, we pass the URL and the params in the `makeRequest` method which has a completion handler returning a results object and an error object. Inside the completion handler, check for any error, if it exists mark the request completed with an error else check for the results object to be a dictionary and a key `accepted`, if this key is `true` our request executed successfully and we mark the request to be executed successfully and finally return the method. After making this method, it needs to be called in the view controller, we do so by the following code.

Client.sharedInstance.changeUserSettings(params) { (_, message) in
  DispatchQueue.global().async {
    self.view.makeToast(message)
  }
}

The code above takes input params containing the user token and key-value pair for the setting that needs to be stored. This request runs on a background thread and displays a toast message with the result of the request.

Now that the settings have been stored on the server, we need to retrieve these settings every time the user logs in the app. Below is the endpoint for the same:

http://api.susi.ai/aaa/listUserSettings.json?access_token=ACCESS_TOKEN

This endpoint accepts the user token which is generated when the user logs in which is used to uniquely identify the user and his/her settings are returned. Let’s create the method that would call this endpoint and parse and save the settings data in the iOS app’s User Defaults.

if let _ = message {
  completion(false, ResponseMessages.ServerError)
} else if let results = results {
  guard let response = results as? [String : AnyObject] else {
    completion(false, ResponseMessages.ServerError)
    return
  }
  guard let settings = 
response[ControllerConstants.Settings.settings.lowercased()] as? [String:String] else {
    completion(false, ResponseMessages.ServerError)
    return
  }
  for (key, value) in settings {
    if value.toBool() != nil {
      UserDefaults.standard.set(value.toBool()!, forKey: key)
    } else {
      UserDefaults.standard.set(value, forKey: key)
    }
  }
  completion(true, response[Client.UserKeys.Message] as? String ?? "error")
}

Here, the creation of the URL is same as we created above the only difference being the method passed. We parse the settings key value into a dictionary followed by a loop which loop’s through all the keys and stores the value in the User Defaults for that key. We simply call this method just after user log in as follows:

Client.sharedInstance.fetchUserSettings(params as [String : AnyObject]) { (success, message) in
  DispatchQueue.global().async {
    print("User settings fetch status: \(success) : \(message)")
  }
}

That’s all for this tutorial where we learned how to store and retrieve settings on the SUSI Server.

References

Continue ReadingHow to Store and Retrieve User Settings from SUSI Server in SUSI iOS

Save Chat Messages using Realm in SUSI iOS

Fetching data from the server each time causes a network load which makes the app depend on the server and the network in order to display data. We use an offline database to store chat messages so that we can show messages to the user even if network is not present which makes the user experience better. Realm is used as a data storage solution due to its ease of usability and also, since it’s faster and more efficient to use. So in order to save messages received from the server locally in a database in SUSI iOS, we are using Realm and the reasons for using the same are mentioned below.

The major upsides of Realm are:

  • It’s absolutely free of charge,
  • Fast, and easy to use.
  • Unlimited use.
  • Work on its own persistence engine for speed and performance

Below are the steps to install and use Realm in the iOS Client:

Installation:

  • Install Cocoapods
  • Run `pod repo update` in the root folder
  • In your Podfile, add use_frameworks! and pod ‘RealmSwift’ to your main and test targets.
  • From the command line run `pod install`
  • Use the `.xcworkspace` file generated by Cocoapods in the project folder alongside `.xcodeproj` file

After installation we start by importing `Realm` in the `AppDelegate` file and start configuring Realm as below:

func initializeRealm() {
        var config = Realm.Configuration(schemaVersion: 1,
            migrationBlock: { _, oldSchemaVersion in
                if (oldSchemaVersion < 0) {
                    // Nothing to do!
                }
        })
        config.fileURL = config.fileURL?.deletingLastPathComponent().appendingPathComponent("susi.realm")
        Realm.Configuration.defaultConfiguration = config
}

Next, let’s head over to creating a few models which will be used to save the data to the DB as well as help retrieving that data so that it can be easily used. Since Susi server has a number of action types, we will cover some of the action types, their model and how they are used to store and retrieve data. Below are the currently available data types, that the server supports.

enum ActionType: String {
  case answer
  case websearch
  case rss
  case table
  case map 
  case anchor
}

Let’s start with the creation of the base model called `Message`. To make it a RealmObject, we import `RealmSwift` and inherit from `Object`

class Message: Object {
  dynamic var queryDate = NSDate()
  dynamic var answerDate = NSDate()
  dynamic var message: String = ""
  dynamic var fromUser = true
  dynamic var actionType = ActionType.answer.rawValue
  dynamic var answerData: AnswerAction?
  dynamic var mapData: MapAction?
  dynamic var anchorData: AnchorAction?
}

Let’s study these properties of the message one by one.

  • `queryDate`: saves the date-time the query was made
  • `answerDate`: saves the date-time the query response was received
  • `message`: stores the query/message that was sent to the server
  • `fromUser`: a boolean which keeps track who created the message
  • `actionType`: stores the action type
  • `answerData`, `rssData`, `mapData`, `anchorData` are the data objects that actually store the respective action’s data

To initialize this object, we need to create a method that takes input the data received from the server.

// saves query and answer date
if let queryDate = data[Client.ChatKeys.QueryDate] as? String,
let answerDate = data[Client.ChatKeys.AnswerDate] as? String {
  message.queryDate = dateFormatter.date(from: queryDate)! as NSDate
  message.answerDate = dateFormatter.date(from: answerDate)! as NSDate}if let type = action[Client.ChatKeys.ResponseType] as? String,
  let data = answers[0][Client.ChatKeys.Data] as? [[String : AnyObject]] {
  if type == ActionType.answer.rawValue {
     message.message = action[Client.ChatKeys.Expression] as! String
     message.actionType = ActionType.answer.rawValue
    message.answerData = AnswerAction(action: action)
  } else if type == ActionType.map.rawValue {
    message.actionType = ActionType.map.rawValue
    message.mapData = MapAction(action: action)
  } else if type == ActionType.anchor.rawValue {
    message.actionType = ActionType.anchor.rawValue
    message.anchorData = AnchorAction(action: action)
    message.message = message.anchorData!.text
  }
}

Since, the response from the server for a particular query might contain numerous action types, we create loop inside a method to capture all those action types and save each one of them. Since, there are multiple action types thus we need a list containing all the messages created for the action types. For each action in the loop, corresponding data is saved into their specific objects.

Let’s discuss the individual action objects now.

  • AnswerAction
class AnswerAction: Object {
  dynamic var expression: String = ""
  convenience init(action: [String : AnyObject]) {
    self.init()
    if let expression = action[Client.ChatKeys.Expression] as? String {
      self.expression = expression
    }
  }
}

 This is the simplest action type implementation. It contains a single property `expression` which is a string type. For initializing it, we take the action object and extract the expression key-value and save it.

if type == ActionType.answer.rawValue {
  message.message = action[Client.ChatKeys.Expression] as! String
  message.actionType = ActionType.answer.rawValue
  // pass action object and save data in `answerData`
  message.answerData = AnswerAction(action: action)
}

Above is the way an answer action is checked and data saved inside the `answerData` variable.

2)   MapAction

class MapAction: Object {
  dynamic var latitude: Double = 0.0
  dynamic var longitude: Double = 0.0
  dynamic var zoom: Int = 13

  convenience init(action: [String : AnyObject]) {
    self.init()
    if let latitude = action[Client.ChatKeys.Latitude] as? String,
    let longitude = action[Client.ChatKeys.Longitude] as? String,
    let zoom = action[Client.ChatKeys.Zoom] as? String {
      self.longitude = Double(longitude)!
      self.latitude = Double(latitude)!
      self.zoom = Int(zoom)!
    }
  }
}

This action implementation contains three properties, `latitude` `longitude` `zoom`. Since the server responds the values inside a string, each of them need to be converted to their respective type using force-casting. Default values are provided for each property in case some illegal value comes from the server.

3)   AnchorAction

class AnchorAction: Object {
  dynamic var link: String = ""
  dynamic var text: String = ""

  convenience init(action: [String : AnyObject]) {
    self.init()if let link = action[Client.ChatKeys.Link] as? String,
    let text = action[Client.ChatKeys.Text] as? String {
      self.link = link
      self.text = text
    }
  }
}

Here, the link to the openstreetmap website is saved in order to retrieve the image for displaying.

Finally, we need to call the API and create the message object and use the `write` clock of a realm instance to save it into the DB.

if success {
  self.collectionView?.performBatchUpdates({
    for message in messages! {
    // real write block
      try! self.realm.write {
        self.realm.add(message)
        self.messages.append(message)
        let indexPath = IndexPath(item: self.messages.count - 1, section: 0)
        self.collectionView?.insertItems(at: [indexPath])
      }
   }
}, completion: { (_) in
    self.scrollToLast()
  })
}

list of message items and inserted into the collection view.Below is the output of the Realm Browser which is a UI for viewing the database.

References:

Continue ReadingSave Chat Messages using Realm in SUSI iOS

Realm database in Loklak Wok Android for Persistent view

Loklak Wok Android provides suggestions for tweet searches. The suggestions are stored in local database to provide a persistent view, resulting in a better user experience. The local database used here is Realm database instead of sqlite3 which is supported by Android SDK. The proper way to use an sqlite3 database is to first create a contract where the schema of the database is defined, then a database helper class which extends from SQLiteOpenHelper class where the schema is created i.e. tables are created and finally write ContentProvider so that you don’t have to write long SQL queries every time a database operation needs to be performed. This is just a lot of hard work to do, as this includes a lot of steps, debugging is also difficult. A solution to this can be using an ORM that provides a simple API to use sqlite3, but the currently available ORMs lack in terms of performance, they are too slow. A reliable solution to this problem is realm database, which is faster than raw sqlite3 and has really simple API for database operations. This blog explains the use of realm database for storing tweet search suggestions.

Adding Realm database to Android project

In project level build.gradle

buildscript {
   repositories {
       jcenter()
   }
   dependencies {
       classpath 'com.android.tools.build:gradle:2.3.3'
       classpath "io.realm:realm-gradle-plugin:3.3.1"

       // NOTE: Do not place your application dependencies here; they belong
       // in the individual module build.gradle files
   }
}

 

And at the top of app/build.gradle “apply plugin: ‘realm-android'”  is added.

Using Realm Database

Let’s start with a simple example. We have a Student class that has only two attributes name and age. To create the model for the database, the Student class is simply extended to RealmObject.

public class Student extends RealmObject {

   private String name;
   private int age;

   // A constructor needs to be explicitly defined, be it an empty constructor
   public Student(String name, int age) {
       this.name = name;
       this.age = age;
   }

   // getters and setters
}

 

To push data to the database, Java objects are created, a transaction is initialized, then copyToRealm method is used to push the data and finally the transaction is committed. But before all this, the database is initialized and a Realm instance is obtained.

Realm.init(context); // Database initialized
Realm realm = Realm.getDefaultInstance(); // realm instance obtained
      
Student student = new Student("Rahul Dravid", 22); // Simple java object created
realm.beginTransaction() // initialization of transaction
realm.copyToRealm(student); // pushed to database
realm.commitTransaction(); // transaction committed

 

copyToRealm takes only a single parameter, the parameter can be an object or an Iterable. Off course, the passed parameter should extend RealmObject. A List of Student can be passed as a parameter to copyToRealm to push multiple data into the database.

The above way of inserting data is synchronous. Realm also supports asynchronous transactions, you guessed it right, you don’t have to depend on AsyncTaskLoader. The same operation can be performed asynchronously as

realm.executeTransaction(new Realm.Transaction() { 
    @Override public void execute(Realm realm) {
        Student student = new Student("Rahul Dravid", 22);
        realm.copyToRealm(student);
    }
});

 

Now, querying the database is as easy as inserting.

RealmResults<Student> studentList = realm.where(Student.class).findAll();

 

No, transaction is required as we are not manipulating the database, as data is just read from the database. RealmResults extend Java List, so List methods which don’t manipulate the List can be used on studentList e.g. get(int index) to obtain object at the index.

The result can also be a filtered one, for example filtering students who are 22 years old.

RealmResults<Student> studentList = realm.where(Student.class).equalTo(age, 22).findAll();

 

Now, removing data from database. deleteAllFromRealm method can be executed on the obtained RealmResults or to completely remove data of a model class delete(Model.class) method on the realm instance is invoked. The operations should be enclosed between beginTransaction and commitTransaction if synchronous behaviour is required else for asynchronous behaviour the operation is done in execute method of an anonymous object of Realm.Transaction.

studentList.deleteAllFromRealm(); // removes the filtered result
realm.delete(Student.class); // removes all data of model class

 

Storing Tweet Search Suggestions in Loklak Wok Android for Persistent view

Loklak Wok Android uses Retrofit2 for sending network requests, for which POJO classes are already created so that it becomes easy for parsing the obtained JSON from network request. Due to this using Realm database becomes more easier, as the defined POJOs can be simply extended to RealmObject to create the model class of the data e.g. Query class extends RealmObject, one of the attribute is the suggestion query i.e. mQuery.

The database is initialized in LoklakWokApplication, the application class, this way the database is initialized only once which persists throughout the app lifecycle.

@Override
public void onCreate() {
   super.onCreate();
   Realm.init(this);
   RealmConfiguration realmConfiguration = new RealmConfiguration.Builder()
           .name(Realm.DEFAULT_REALM_NAME)
           .deleteRealmIfMigrationNeeded()
           .build();
   Realm.setDefaultConfiguration(realmConfiguration);
}

 

deleteRealmIfMigrationNeeded removes the old realm database and creates a new one if any of the Model class get changed i.e. an attribute is removed, added or simply the name of attribute is changed. This is done as we are not storing user generated data, we are just using database to provide persistent view. So, the previously kept data is not important.

The database is closed in onTerminate callback of the application

@Override
public void onTerminate() {
   Realm.getDefaultInstance().close();
   super.onTerminate();
}

 

Now that database is initialized. We fetch the previously stored data and display it in RecyclerView. If the network request is successful the queries from database are replaced by the queries fetched in network request, else the queries from database are displayed providing a persistent view. The way it is implemented in onCreateView of SuggestFragment

mRealm = Realm.getDefaultInstance();
...
// old queries obtained from database
RealmResults<Query> queryRealmResults = mRealm.where(Query.class).findAll();
List<Query> queries = mRealm.copyFromRealm(queryRealmResults);
// RecyclerView adapter created with old queries
mSuggestAdapter = new SuggestAdapter(queries, this);
tweetSearchSuggestions.setLayoutManager(new LinearLayoutManager(getActivity()));
tweetSearchSuggestions.setAdapter(mSuggestAdapter);

 

The old queries are replaced in onSuccessfulRequest

private void onSuccessfulRequest(SuggestData suggestData) {
   // suggestData contains suggestion queries
   if (suggestData != null) {
       // old queries replaced with new ones
       mSuggestAdapter.setQueries(suggestData.getQueries());
   }
   setAfterRefreshingState();
}

 

Now suggestion queries needs to be inserted into the database. Only the latest suggestions are inserted i.e. queries present when onStop lifecycle method of fragment is called, and as previous queries are not needed anymore, they are deleted. The operation is performed in a synchronous way.

@Override
public void onStop() {
   ...
   mRealm.beginTransaction();
   // old queries deleted
   mRealm.delete(Query.class);
   // new queries inserted
   mRealm.copyToRealm(mSuggestAdapter.getQueries());
   mRealm.commitTransaction();
   // fragment lifecycle called i.e. a new fragment/activity opens
   super.onStop();
}

 

Conclusion: Sqlite3 and Realm comparison

Operations Sqlite3 Realm
Table creation CREATE TABLE … extends RealmObject
Inserting data INSERT INTO … copyToRealm
Searching data SELECT … realm.where(Model.class)
Deleting data DELETE FROM … realmResults.deleteAllFromRealm() or realm.delete(Model.class)

Resources:

 

 

Continue ReadingRealm database in Loklak Wok Android for Persistent view

Importing the Open Event format in Giraffe

Giraffe is a personal conference schedule tool for Android. Most conferences, bar camps and similar events offer their plan of sessions and talks in the iCal format for importing into your calendar. However importing a whole session plan into your standard calendar renders it pretty much useless for anything else. Giraffe allows users to import the schedule into a separate list giving you a simple overview on what happens on the conference. Besides the session, title, date and  time it also lists the speaker, location and description if available in the iCal URL. Sessions can be bookmarked and the list can be filtered by favourites and upcoming talks.

Recently I added the support for Open Event JSON format along with iCal. In this blog I describe the simple steps you need to follow to see the event that is created in the Open Event server in the Giraffe app. The initial steps are similar to Giggity app,

 1. Go to your event dashboard

2. Click on the export button.

3. Select sessions from the dashboard and copy the URL.

 

4. Click on the “Giraffe” button on the toolbar and paste the link in the box following. App will ask you to paste it when the first time you open it. Here the app loads the data and checks few initial character to see which kind of data is received. Find my other blog post to solve that problem here.

The app uses separate data models for iCal and JSON to store the informations received and then save them in SQL database for CRUD options. See the database activity here

   

5. Now you can see the sessions. Click on them to see more information or bookmark them if needed. The data is loaded from the database so when app is offline so we don’t need to worry about connection once the data is being loaded.

   

Resources:

 

Continue ReadingImporting the Open Event format in Giraffe