Using Picasso to Show Images in SUSI Android

Important skills of SUSI.AI are to display web search queries, maps of any location and provide a list of relevant information of a topic. This blog post will cover why Glide is replaced by Picasso to show images related to these action types and how it is implemented in SUSI AndroidPicasso is a powerful image downloading and caching open source library developed by Square.

Why Glide is replaced by Picasso to show images in SUSI Android?

Previously we used Glide library to show preview in SUSI Android but we replace it because it was creating an error continuously.

java.lang.IllegalArgumentException: You cannot start a load for a destroyed activity at com.bumptech.glide.manager.RequestManagerRetriever  at Com.bumptech.glide.manager.RequestManagerRetriever.get(RequestManagerRetriever.java:102) at com.bumptech.glide.manager.RequestManagerRetriever.get(RequestManagerRetriever.java:87)at com.bumptech.glide.Glide.with(Glide.java:629)
atnorg.fossasia.susi.ai.adapters.recycleradapters.WebSearchAdapter.onBindViewHolder(WebSearchAdapter.java:74)

Reason for this error is when activity destroyed and again recreated the context used by glide is old one and  that activity already destroyed .

Glide.with(context).load(imageList.get(0))

One solution of this error is to use context.getApplicationContext()  but it is a bad idea. Another solution is to replace glide by picasso and later one is good because picasso is also a very good image downloading and caching library.

To use Picasso in your project you have to add dependency in build.gradle(Module) file.

dependencies {
  
  compile “com.squareup.picasso:picasso:2.4.0”
  
}

How Picasso is used in different actiontype

Map

“actions”: [
     {
       “type”: “map”,
       “latitude”: “1.2896698812440377”,
       “longitude”: “103.85006683126556”,
       “zoom”: “13”
     }
   ]

Link we used to retrieve image url is

http://staticmap.openstreetmap.de/statucmap.php?center=latitude,longitude& zoom=zoom&size=lengthXbreadth

Picasso will load image from this url and show image in the imageview. Here mapImage is the imageview in which map image is shown.

Picasso.with(currContext).load(mapHelper.getMapURL())
                       .into(mapImage, new  com.squareup.picasso.Callback() {    
                           @Override
                           public void onSuccess() {
                               pointer.setVisibility(View.VISIBLE);
                           }
                           @Override
                           public void onError() {
                               Log.d(“Error”, “map image can’t loaded”);
                           }
                       });

WebSearch

When we query like “Search for fog” we get ‘query’ in reply from server

“query”: “fog”

Now we use this query to retrieve image url which we used in Picasso to show images.Picasso load this image into previewImageView imageview. Image url is retrieved using  DuckDuckGo api. We are using url

https://api.duckduckgo.com/?format=json&pretty=1&q=query&ia=meanings

It gives a json response which contains image url

Picasso.with(context).load(iconUrl)
      .into(holder.previewImageView, new  com.squareup.picasso.Callback() {
                  @Override
                   public void onSuccess() {
                         Log.d(“Sucess”,“image loaded successfully”);
                   }
                   @Override
                   public void onError() {
                       holder.previewImageView.setVisibility(View.GONE);
                     }
                });     

Here also com.squareup.picasso.Callback is use to find that image is loaded successfully or not.

RSS

When we query any like “dhoni” we get ‘link’ in reply from server

“title”: “Dhoni”,

“description”: “”,
“link”: “http://de.wikipedia.org/wiki/Dhoni”

We use this link in android-link-preview library to retrieve relevant image url and then Picasso use this url to load image into imageview previewImageView.

Picasso.with(currContext).load(imageList.get(0))
      .fit().centerCrop()
      .into(previewImageView);

Reference

Continue ReadingUsing Picasso to Show Images in SUSI Android

Semantic-UI Validations for Forms in Open Event Frontend

Open Event Frontend requires forms at several places like at the time of login, for creation of events, taking the details of the user, creating discount codes for tickets etc.. Validations for these forms is a must, like in the above picture, we can see that many fields like discount code, amount etc. have been left empty, these null values when stored at backend can induce errors.

Semantic-UI makes our life easier and provides us with it’s own validations. Its form validation behavior checks data against a set of criteria or rules before passing it along to the server.

Let’s now dive deeper into Semantic validations, we’ll take discount code form as our example. The discount code form has many input fields and we have put checks at all of them, these checks are called rules here. We’ll discuss all the rules used in this form one by one

  1. Empty

Here we check if the input box with the identifier discount_amount is empty or not, if it is empty, a prompt is shown with the given message.

         identifier : ‘discount_amount’,
         rules      : [
           {
             type   : ’empty’,
             prompt : this.l10n.t(‘Please enter the discount amount’)
           }
         ]

2. Checked
Here, we validate whether the checkbox is checked or not and if it is not, show corresponding message

rules      : [
   {
     type   : ‘checked’,
     prompt : this.l10n.t(‘Please select the appropriate choices’)
   }]

3. RegExp

These checks are very important in input fields requiring passwords and codes, they specify the allowed input characters

rules      : [{
   type  : ‘regExp’,
   value : ‘^[a-zA-Z0-9_-]*$’
}]

4.Custom rules

Many a times, we require some rules which are by default not given by semantic, here we can create custom rules.

Like here, we want to check whether the user has not set max value lower than min.

$.fn.form.settings.rules.checkMaxMin = () => {
     if (this.get(‘data.minQuantity’) > this.get(‘data.maxQuantity’)) {
       return false;
     }
     return true;
   };

Here, we are creating our own custom rule checkMaxMin which returns boolean value depending upon minQuantity and maxQuantity. Now, this can be directly used as a rule

identifier : ‘min_order’,
optional   : true,
rules      : [
 {
  type   : ‘checkMaxMin’,
  prompt : this.l10n.t(‘Minimum value should not be greater than maximum’)
 }]

You can find the above code here

Additional Resources

Continue ReadingSemantic-UI Validations for Forms in Open Event Frontend

Search Functionalities in SUSI Android App Using Android SearchView Widget

Searching is a common feature that is required in most applications. But the problem in implementing searching functionality is that there is no common way to do that. People fight over whose way is best to implement search functionality. In this blog post we’ll be looking at how search functionality works in SUSI Android App and how is it implemented. We have used Android’s SearchView widget to do that. There are many other ways to do so but this one is best suited for our requirements. Let’s see how it works.

UI Components used for Searching

1. Search icon (magnifying glass icon)

In the action bar, you can see a small icon. Clicking on the icon initiates search.

2. Edit text

An Obvious requirement is an edit test to enter search query.

3. Up and Down arrow keys

Required to search through the whole app. Simply use the up and down arrow keys to navigate through the app and find out each occurrence of the word you want to search.

 

 

 

 

 

 

 

4. Cross Button

Last but not the least, a close or cross button to close the search action.

Implementation

We have used Android’s inbuilt Widget SearchView. According to official android documentation

A widget that provides a user interface for the user to enter a search query and submit a request to a search provider. Shows a list of query suggestions or results, if available, and allows the user to pick a suggestion or result to launch into.

This widget makes searching a lot easier. It provides all methods and listeners which are actually required for searching. Let’s cover them one by one.

  1. Starting the search: searchView.setOnSearchClickListener Listener simply activates when a user clicks on search icon in the toolbar. Do all your work which needs to be done at the starting of the search like, hiding some other UI elements of doing an animation inside the listener
searchView.setOnSearchClickListener({
   chatPresenter.startSearch()
})
  1. Stop the Search: searchView.setOnCloseListener Listener gets activated when a user clicks on the cross icon to close the search. Add all the code snippet you want which is needed to be executed when the search is closed inside this like maybe notify the adapter about data set changes or closing the database etc.
searchView.setOnCloseListener({
   chatPresenter.stopSearch()
   false
})
  1.  Searching a query:  searchView.setOnQueryTextListener Listener overrides 2 methods:

3.1 onQueryTextSubmit: As the name suggests, this method is called when the query to be searched is submitted.

3.2 onQueryTextChange: This method is called when query you are writing changes.

We, basically wanted same thing to happen if user has submitted the query or if he is still typing and that is to take the query at that particular moment, find it in database and highlight it. So, chatPresenter.onSearchQuerySearched(query) this method is called in both onQueryTextSubmit and onQueryTextSubmit  to do that.

 searchView.setOnQueryTextListener(object : SearchView.OnQueryTextListener {
 
      override fun onQueryTextSubmit(query: String): Boolean {
           //Handle Search Query
           chatPresenter.onSearchQuerySearched(query)
           recyclerAdapter.query = query
           return false
       }

       override fun onQueryTextChange(newText: String): Boolean {
           if (TextUtils.isEmpty(newText)) {
               modifyMenu(false)
               recyclerAdapter.highlightMessagePosition = -1
               recyclerAdapter.notifyDataSetChanged()
               if (!editText.isFocused) {
                   editText.requestFocus()
               }
           } else {
               chatPresenter.onSearchQuerySearched(newText)
               recyclerAdapter.query = newText
           }
           return false
       }
   })
   return true
}
  1. Finding query in database: Now we have a query to be searched, we can just use a database operation to do that. The below code snippet finds all the messages which has the query present in it and work on it. If the query is not found, it simply displays a toast saying “Not found”
override fun onSearchQuerySearched(query: String) {
   chatView?.displaySearchElements(true)
   results = databaseRepository.getSearchResults(query)
   offset = 1
   if (results.size > 0) {
       chatView?.modifyMenu(true)
       chatView?.searchMovement(results[results.size - offset].id.toInt())
   } else {
       chatView?.showToast(utilModel.getString(R.string.not_found))
   }
}

This is the database operation.

override fun getSearchResults(query: String): RealmResults<ChatMessage> {
   return realm.where(ChatMessage::class.java).contains(Constant.CONTENT,
           query, Case.INSENSITIVE).findAll()
}

  1. Highlighting the part of message: Now, we know which message has the query, we just want to highlight it with a bright color to display the result. For that, we used SpannableString to highlight a part of complete string.
String text = chatTextView.getText().toString();
SpannableString modify = new SpannableString(text);
Pattern pattern = Pattern.compile(query, Pattern.CASE_INSENSITIVE);
Matcher matcher = pattern.matcher(modify);
while (matcher.find()) {
   int startIndex = matcher.start();
   int endIndex = matcher.end();
   modify.setSpan(new BackgroundColorSpan(Color.parseColor("#ffff00")), startIndex, endIndex, Spannable.SPAN_EXCLUSIVE_EXCLUSIVE);
}
chatTextView.setText(modify);

Summary

The whole point of this blog post was to educate about SearchView widget of android and how it makes it easy to search queries. All the methods you need are already implemented. You just need to call them and add database operation.

Resources

  1. The link to official android documentation explaining about different methods in SearchView Class https://developer.android.com/reference/android/widget/SearchView.html
  2. Another tutorial about SearchView http://www.journaldev.com/12478/android-searchview-example-tutorial
Continue ReadingSearch Functionalities in SUSI Android App Using Android SearchView Widget

Sorting Photos in Phimpme Android

The Phimpme Android application features a fully fledged gallery interface with an option to switch to all photos mode, albums mode and to sort photos according to various sort actions. Sorting photos via various options helps the user to get to the desired photo immediately without having to scroll down till the end in case it is the last photo in the list generated automatically by scanning the phone for images using the Android’s mediaStore class. In this tutorial, I will be discussing how we achieved the sorting option in the Phimpme application with the help of some code snippets.

To sort the all photos list, first of all we need a list of all the photos by scanning the phone using the media scanner class via the code snippet provided below:

uri = android.provider.MediaStore.Images.Media.EXTERNAL_CONTENT_URI;
      String[] projection = {MediaStore.MediaColumns.DATA};
      cursor = activity.getContentResolver().query(uri, projection, null, null, null);

In the above code we are using a cursor to point to each photos and then we are extracting the path of the images and storing it in a list using a while loop. After we generate the list of path of all the images, we have to convert the into a list of media using the file path using the code below:

for (String path : listOfAllImages) {
          list.add(new Media(new File(path)));
      }
      return list;
  }

After generating the list of all images we can sort the photos using the Android’s collection class. In Phimpme Android we provide the option to sort photos in different categories namely:

  1. Name Sort action
  2. Date Sort action
  3. Size Sort action
  4. Numeric Sort action

As sorting is somewhat heavy task so doing this in the main thread will result in freezing UI of the application so we have to put this into an AsyncTask with a progress dialog to sort the photos. After putting the above four options in the menu options. We can define an Asynctask to load the images and in the onPreExecute method of the AsyncTask, we are displaying the progress dialog to the user to depict that the sorting process is going on as depicted in the code snippet below

AlertDialog.Builder progressDialog = new AlertDialog.Builder(LFMainActivity.this, getDialogStyle());
dialog = AlertDialogsHelper.getProgressDialog(LFMainActivity.this, progressDialog,
      getString(R.string.loading_numeric), all_photos ? getString(R.string.loading_numeric_all) : getAlbum().getName());
dialog.show();

In the doInBackgroundMethod of the AsyncTask, we are sorting the list of all photos using the Android’s collection class and using the static sort method defined in that class which takes the list of all the media files as a parameter and the MediaComparator which takes the sorting mode as the first parameter and the sorting order as the second. The sorting order decides whether to arrange the list in ascending or in descending order.

getAlbum().setDefaultSortingMode(getApplicationContext(), NUMERIC);
Collections.sort(listAll, MediaComparators.getComparator(getAlbum().settings.getSortingMode(), getAlbum().settings.getSortingOrder()));

After sorting, we have to update the data set to reflect the changes of the list in the UI. This we are doing in the onPostExecute method of the AsyncTask after dismissing the progress Dialog to avoid the window leaks in the application. You can read more about the window leaks in Android due to progressdialog here.

dialog.dismiss();
mediaAdapter.swapDataSet(listAll);

To get the full source code, you can refer the Phimpme Android repository listed in the resources below.

Resources

  1. Android developer guide to mediastore class: https://developer.android.com/reference/android/provider/MediaStore.html
  2. GitHub LeafPic repository: https://github.com/HoraApps/LeafPic
  3. Stackoverflow – Preventing window leaks in Android: https://stackoverflow.com/questions/6614692/progressdialog-how-to-prevent-leaked-window
  4. Blog – Sorting lists in Android: http://www.worldbestlearningcenter.com/tips/Android-sort-ListView.htm
Continue ReadingSorting Photos in Phimpme Android

Visualising Tweet Statistics in MultiLinePlotter App for Loklak Apps

MultiLinePlotter app is now a part of Loklak apps site. This app can be used to compare aggregations of tweets containing a particular query word and visualise the data for better comparison. Recently there has been a new addition to the app. A feature for showing tweet statistics like the maximum number of tweets (along with date) containing the given query word and the average number of tweets over a period of time. Such statistics is visualised for all the query words for better comparison.

Related issue: https://github.com/fossasia/apps.loklak.org/issues/236

Obtaining Maximum number of tweets and average number of tweets

Before visualising the statistics we need to obtain them. For this we simply need to process the aggregations returned by the Loklak API. Let us start with maximum number of tweets containing the given keyword. What we actually require is what is the maximum number of tweets that were posted and contained the user given keyword and on which date the number was maximum. For this we can use a function which will iterate over all the aggregations and return the largest along with date.

$scope.getMaxTweetNumAndDate = function(aggregations) {
        var maxTweetDate = null;
        var maxTweetNum = -1;

        for (date in aggregations) {
            if (aggregations[date] > maxTweetNum) {
                maxTweetNum = aggregations[date];
                maxTweetDate = date;
            }
        }

        return {date: maxTweetDate, count: maxTweetNum};
    }

The above function maintains two variables, one for maximum number of tweets and another for date. We iterate over all the aggregations and for each aggregation we compare the number of tweets with the value stored in the maxTweetNum variable. If the current value is more than the value stored in that variable then we simply update it and keep track of the date. Finally we return an object containing both maximum number of tweets and the corresponding date.Next we need to obtain average number of tweets. We can do this by summing up all the tweet frequencies and dividing it by number of aggregations.

$scope.getAverageTweetNum = function(aggregations) {
        var avg = 0;
        var sum = 0;

        for (date in aggregations) {
            sum += aggregations[date];
        }

        return parseInt(sum / Object.keys(aggregations).length);
    }

The above function calculates average number of tweets in the way mentioned before the snippet.

Next for every tweet we need to store these values in a format which can easily be understood by morris.js. For this we use a list and store the statistics values for individual query words as objects and later pass it as a parameter to morris.

var maxStat = $scope.getMaxTweetNumAndDate(aggregations);
        var avg = $scope.getAverageTweetNum(aggregations);

        $scope.tweetStat.push({
            tweet: $scope.tweet,
            maxTweetCount: maxStat.count,
            maxTweetOn: maxStat.date,
            averageTweetsPerDay: avg,
            aggregationsLength: Object.keys(aggregations).length
        });

We maintain a list called tweetStat and the list contains objects which stores the query word and the corresponding values.

Apart from plotting these statistics, the app also displays the statistics when user clicks on an individual treat present in the search record section. For this we filter tweetStat list mentioned above and get the required object corresponding to the query word the user selected bind it to angular scope. Next we display it using HTML.

<div class="tweet-stat max-tweet">
                  <div class="stat-label"> <h4>Maximum number of tweets containing '{{modalHeading}}' :</h4></div>
                  <div class="stat-value"> <strong>{{selectedTweetStat.maxTweetCount}}</strong> tweets on
                    <strong>{{selectedTweetStat.maxTweetOn}}</strong>
                  </div>
                </div>

Finally we need to plot the statistics. For this we use a function called plotStatGraph dedicated only for plotting statistics graph. We pass the tweetStat list as a parameter to morris and configure all the other parameters.

$scope.plotStatGraph = function() {
        $scope.plotStat = new Morris.Bar({
            element: 'graph',
            data: $scope.tweetStat,
            xkey: 'tweet',
            ykeys: ['maxTweetCount', 'averageTweetsPerDay'],
            labels: ['Maximum no. of tweets : ', 'Average no. of tweets/day'],
            parseTime: false,
            hideHover: 'auto',
            resize: true,
            stacked: true,
            barSizeRatio: 0.40
        });
        $scope.graphLoading = false;
    }

But now we have two graphs. One for showing variations in aggregation and the other for showing statistics. How do we manage them? Somehow we need to show them in the same page as this is a single page app. Also we need to avoid vertical scrolling as it would degrade both UI and UX. So we need to implement a switching mechanism. The user should be able to switch between the two graph views as per their wish. How to achieve that? Well, for this we maintain a global variable which will keep track of the current plot type. If the current graph type is aggregations then we call the function to plot aggregations otherwise we call the above mentioned function to plot statistics.

$scope.plotData = function() {
        $(".plot-data").html("");
        if ($scope.currentGraphType === "aggregations") {
            $scope.plotAggregationGraph();
        } else {
            $scope.plotStatGraph();
        }
    }

Lastly we integrate this state variable (currentGraphType) with the UI so that users can easily toggle between graph views with just a click.

<div class="switch" ng-click="toggle()">
                <span ng-if="queryRecords.length !== 0" class="glyphicon glyphicon-stats"></span>
              </div>

Important resources

Continue ReadingVisualising Tweet Statistics in MultiLinePlotter App for Loklak Apps

Developing MultiLinePlotter App for Loklak

MultiLinePlotter is a web application which uses Loklak API under the hood to plot multiple tweet aggregations related to different user provided query words in the same graph. The user can give several query words and multiple lines for different queries will be plotted in the same graph. In this way, users will be able to compare tweet distribution for various keywords and visualise the comparison. All the searched queries are shown under the search record section. Clicking on a record causes a dialogue box to pop up where the individual tweets related to the query word is displayed. Users can also remove a series from the plot dynamically by just pressing the Remove button beside the query word in record section. The app is presently hosted on Loklak apps site.

Related issue – https://github.com/fossasia/apps.loklak.org/issues/225

Getting started with the app

Let us delve into the working of the app. The app uses Loklak aggregation API to get the data.

A call to the API looks something like this:

http://api.loklak.org/api/search.json?q=fossasia&source=cache&count=0&fields=created_at

A small snippet of the aggregation returned by the above API request is shown below.

"aggregations": {"created_at": {
    "2017-07-03": 3,
    "2017-07-04": 9,
    "2017-07-05": 12,
    "2017-07-06": 8,
}}

The API provides a nice date v/s number of tweets aggregation. Now we need to plot this. For plotting Morris.js has been used. It is a lightweight javascript library for visualising data.

One of the main features of this app is addition and removal of multiple series from the graph dynamically. How do we achieve that? Well, this can be achieved by manipulating the morris.js data list whenever a new query is made. Let us understand this in steps.

At first, the data is fetched using angular HTTP service.

$http.jsonp('http://api.loklak.org/api/search.json?callback=JSON_CALLBACK',
            {params: {q: $scope.tweet, source: 'cache', count: '0', fields: 'created_at'}})
                .then(function (response) {
                    $scope.getData(response.data.aggregations.created_at);
                    $scope.plotData();
                    $scope.queryRecords.push($scope.tweet);
                });

Once we get the data, getData function is called and the aggregation data is passed to it. The query word is also stored in queryRecords list for future use.

In order to plot a line graph morris.js requires a data object which will contain the required values for a series. Given below is an example of such a data object.

data: [
    { x: '2006', a: 100, b: 90 },
    { x: '2007', a: 75,  b: 65 },
    { x: '2008', a: 50,  b: 40 },
    { x: '2009', a: 75,  b: 65 },
],

For every ‘x’, ‘a’ and ‘b’ will be plotted. Thus two lines will be drawn. Our app will also maintain a data list like the one shown above, however, in our case, the data objects will have a variable number of keys. One key will determine the ‘x’ value and other keys will determine the ordinates (number of tweets).

All the data objects present in the data list needs to be updated whenever a new search is done.

The getData function does this for us.

var value = $scope.tweet;
        for (date in aggregations) {
            var present = false;
            for (var i = 0; i < $scope.data.length; i++) {
                var item = $scope.data[i];
                if (item['day'] === date) {
                    item[[value]] = aggregations[date];
                    $scope.data[i] = item
                    present = true;
                    break;
                }
            }
            if (!present) {
                $scope.data.push({day: date, [value]: aggregations[date]});
            }
        }


The for loop in the above code snippet updates the global data list used by morris.js. It simply iterates over the dates in the aggregation, extracts the object corresponding to a particular date, adds the new query word as a key and, the number of tweets on that date as the value.If a date is not already present in the list, then it inserts a new object corresponding to the date and query word. Once our data list is updated, we are ready to redraw the graph with the updated data. This is done using plotData function. The plotData function simply checks the user selected graph type. If the selected type is aggregations then it calls plotAggregationGraph() to redraw the aggregations plot.

$scope.remove = function(record) {
        $scope.queryRecords = $scope.queryRecords.filter(function(e) {
            return e !== record });

        $scope.data.forEach(function(item) {
            delete item[record];
        });

        $scope.data = $scope.data.filter(function(item) {
            return Object.keys(item).length !== 1;
        });

        $scope.ykeys = $scope.ykeys.filter(function(item) {
            return item !== record;
        });

        $scope.labels = $scope.labels.filter(function(item) {
            return item !== record;
        });

        $scope.plotData();
}

The above function simply scans the data list, filters the objects which contains selected record as a key and removes them using filter method of javascript arrays. It also removes the corresponding labels and entries from labels and ykeys arrays. Finally, it once again calls plotData function to redraw the plot.

Given below is a sample plot generated by this app with the query words – google, android, microsoft, samsung.

 

Conclusion

This blog post explained how multiple series are plotted dynamically in the MultiLinePlotter app. Apart from aggregations plot it also plots tweet statistics like maximum tweets and average tweets containing a query word and visualises them using stacked bar chart. I will be discussing about them in my subsequent blogs.

Important resources

Continue ReadingDeveloping MultiLinePlotter App for Loklak

Managing Related Endpoints in Permission Manager of Open Event API Server

Open Event API Server has its permission manager to manage all permission to different endpoints and some of the left gaps were filled by new helper method has_access. The next challenge for permission manager was to incorporate a feature many related endpoints points to the same resource.
Example:

  • /users-events-roles/<int:users_events_role_id>/user or
  • /event-invoices/<int:event_invoice_id>/user

Both endpoints point to Users API where they are fetching the record of a single user and for this, we apply the permission “is_user_itself”. This permission ensures that the logged in user is the same user whose record is asked through the API and for this we need the “user_id” as the “id” in the permission function, “is_user_itself”
Thus there is need to add the ability in permission manager to fetch this user_id from different models for different endpoints. For example, if we consider above endpoints then we need the ability to get user_id from UsersEventsRole and EventInvoice models and pass it to permission function so that it can use it for the check.

Adding support

To add support for multiple keys, we have to look for two things.

  • fetch_key_url
  • model

These two are key attributes to add this feature, fetch_key_url will take the comma separated list which will be matched with view_kwargs and model receives the array of the Model Classes which will be used to fetch the related records from the model
This snippet provides the main logic for this:

for index, mod in enumerate(model):
   if is_multiple(fetch_key_url):
       f_url = fetch_key_url[index]
   else:
       f_url = fetch_key_url
   try:
       data = mod.query.filter(getattr(mod, fetch_key_model) == view_kwargs[f_url]).one()
   except NoResultFound, e:
       pass
   else:
       found = True

if not found:
   return NotFoundError({'source': ''}, 'Object not found.').respond()

From the above snippet we are:

  • We iterate through the models list
  • Check if fetch_key_url has multiple keys or not
  • Get the key from fetch_key_url on the basis of multiple keys or single key in it.
  • We try to attempt to get object from model for the respective iteration
  • If there is any record/object in the database then it’s our data. Skipping further process
  • Else continue iteration till we get the object or to the end.

To use multiple mode

Instead of providing the single model to the model option of permission manager, provide an array of models. Also, it is optional to provide comma separated values to fetch_key_url
Now there can be scenario where you want to fetch resource from database model using different keys present on your view_kwargs
for example, consider these endpoints

  1. `/notifications/<notification_id>/event`
  2. `/orders/<order_id>/event`

Since they point to same resource and if you want to ensure that logged in user is organizer then you can use these two things as:

  1. fetch_key_url=”notification_id, order_id”
  2. model=[Notification, Order]

Permission manager will always match indexes in both options, the first key of fetch_key_url will be only used for the first key of the model and so on.
Also, fetch_key_url is an optional parameter and even in multiple mode you can provide a single value as well.  But if you provide multiple commas separated values make sure you provide all values such that no of values in fetch_key_url and model must be equal.

Resources

Continue ReadingManaging Related Endpoints in Permission Manager of Open Event API Server

Custom Data Layer in Open Event API Server

Open Event API Server uses flask-rest-jsonapi module to implement JSON API. This module provides a good logical abstraction in the data layer.
The data layer is a CRUD interface between resource manager and data. It is a very flexible system to use any ORM or data storage. The default layer you get in flask-rest-jsonapi is the SQLAlchemy ORM Layer and API Server makes use of default alchemy layer almost everywhere except the case where I worked on email verification part.

To add support for adding user’s email verification in API Server, there was need to create an endpoint for POST /v1/users/<int:user_id>/verify
Clearly here we are working on a single resource i.e, specific user record. This requires us to use ResourceDetail and the only issue was there is no any POST method or view in ResourceDetail class. To solve this I created a custom data layer which enables me to redefine all methods and views by inheriting abstract class. A custom data layer must inherit from flask_rest_jsonapi.data_layers.base.Base.

Creating Custom Layer

To solve email verification process, a custom layer was created at app/api/data_layers/VerifyUserLayer.py

def create_object(self, data, view_kwargs):
   user = safe_query(self, User, 'id', view_kwargs['user_id'], 'user_id')
   s = get_serializer()
   try:
       data = s.loads(data['token'])
   except Exception:
       raise UnprocessableEntity({'source': 'token'}, "Invalid Token")

   if user.email == data[0]:
       user.is_verified = True
       save_to_db(user)
       return user
   else:
       raise UnprocessableEntity({'source': 'token'}, "Invalid Token")

Using custom layer in API

We can easily provide custom layer in API Resource using one of the properties of the Resource Class

data_layer = {
   'class': VerifyUserLayer,
   'session': db.session
}

This is all we have to provide in the custom layer, now all CRUD method will be directed to our custom data layer.

Solution to our issue
Setting up custom layer provides us the ability to create our custom resource methods, i.e, modifying the view for POST request and allowing us to verify the registered users in API Server.
On Setting up the data layer all I need to do is create a ResourceList with using this layer and with permissions

class VerifyUser(ResourceList):

   methods = ['POST', ]
   decorators = (jwt_required,)
   schema = VerifyUserSchema
   data_layer = {
       'class': VerifyUserLayer,
       'session': db.session
   }

This enables me to use the custom layer, VerifyUserLayer for ResourceList resource.

Resources

Continue ReadingCustom Data Layer in Open Event API Server

A guide to use Permission Manager in Open Event API Server

This article provides a simple guide to use permission manager in Open Event API Server. Permission manager is constantly being improved and new features are being added into it. To ensure that all co-developers get to know about it and make use of them, this blog posts describes every part of permission manager.

Bootstrapping

Permission manager as a part of flask-rest-jsonapi works as a decorator for different resources of the API. There are two ways to provide the permission decorator to any view

  • First one is to provide it in the list of decorators
decorators = (api.has_permission('is_coorganizer', fetch="event_id",
                                fetch_as="event_id", model=StripeAuthorization),)
    • Second way is to explicitly provide it as a decorator to any view
@api.has_permission('custom_arg', custom_kwargs='custom_kwargs')
    def get(*args, **kwargs):
        return 'Hello world !'

In the process of booting up, we first need to understand the flow of Resources in API. All resources even before doing any schema check, call the decorators. So this way you will not get any request data in the permission methods. All you will receive is a dict of the URL parameters but again it will not include the filter parameters.
Permission Manager receives five parameters as: 

def permission_manager(view, view_args, view_kwargs, *args, **kwargs):

First three are provided into it implicitly by flask-rest-jsonapi module

  • view: This is the resource’s view method which is called through the API. For example, if I go to /events then the get method of ResourceList will be called.
  • view_args: These are args associated with that view.
  • view_kwargs: These are kwargs associated with that resource view. It includes all your URL parameters as well.
  • args: These are the custom args which are provided when calling the permission manager. Here at permission manager is it expected that the first index of args will be the name of permission to check for.
  • kwargs: This is the custom dict which is provided on calling the permission manager. The main pillar of the permission manager. Described below in usage.

Using Permission Manager

Using permission manager is basically understanding the different options you can send through the kwargs so here is the list of the things you can send to permission manager
These are all described in the order of priority in permission manager

  • method (string): You can provide a string containing the methods where permission needs to be checked as comma separated values of different methods in a string.
    For example: method=”GET,POST”
  • leave_if (lambda): This receives a lambda function which should return boolean values. Based on returned value if is true then it will skip the permission check. The provided lambda function receives only parameter, “view_kwargs”
    Example use case can be the situation where you can leave the permission for any specifically related endpoint to some resource and would like to do a manual check in the method itself.
  • check (lambda): Opposite to leave_if. It receives a lambda function that will return boolean values. Based on returned value, If it is true then only it will go further and check the request for permissions else will throw forbidden error.
  • fetch (string): This is the string containing the name of the key which has to be fetched for the fetch_as key (described below). Permission manager will first look for this value in view_kwargs dict object. If it is not there then it will make the query to get one(described below at model )
  • fetch_as (string): This is the string containing the name of a key. The value of fetch key will be sent to the permission functions by this name.
  • model (string): This is one most interesting concept here. To get the value of the fetch key. Permission manager first looks into view_kwargs and if there no such value then you can still get one through the model. The model attribute here receives the class of the database model which will be used to get the value of the fetch key.
    It makes the query to get the single resource from this model and look for the value of the fetch key and then pass it to the permission functions/methods.
    The interesting part is that by default it uses <id> from view_kwargs to get the resource from the model but in any case if there is no specific ID with name <id> on the view_kwargs. You can use these two options as:
  • fetch_key_url (string): This is the name of the key whose value will be fetched from view_kwargs and will be used to match the records in database model to get the resource.
  • fetch_key_model (string): This is the name of the match column in the database model for the fetch_key_url, The value of it will be matched with the column named as the value of fetch_key_model.
    In case there is no record found in the model then permission manager will throw NotFound 404 Error.

A helper for permissions

The next big thing in permission manager is the addition of new helper function “has_access”

def has_access(access_level, **kwargs):
   if access_level in permissions:
       auth = permissions[access_level](lambda *a, **b: True, (), {}, (), **kwargs)
       if type(auth) is bool and auth is True:
           return True
   return False

This method allows you to check the permission at the mid of any method of any view and of any resource. Just provide the name of permission in the first parameter and then the additional options needed by the permission function as the kwargs values.
This does not throw any exception. Just returns the boolean value so take care of throwing any exception by yourselves.

Anything to improve on?

I will not say this exactly as the improvement but I would really like to make it more meaningful and interesting to add permission. May be something like this below:

permission = "Must be co_organizer OR track_organizer, fetch event_id as event_id, use model Event"

This clearly needs time to make it. But I see this as an interesting way to add permission. Just provide meaningful text and rest leave it to the permission manager.

Continue ReadingA guide to use Permission Manager in Open Event API Server

Showing RSS and Table Type Replies in SUSI Messenger Bots

All the messengers have a “plain text” reply support. To show web search (RSS) or table type replies, either:

  1. We need a “list type” (as in Facebook messenger) or “table type” reply support built in the messenger itself.
                                                                  or
  2. We need to convert the RSS or table type reply by SUSI API to plain text, so that we can send it, due to the “plain text” reply support available in almost every messenger.

The second point is the most favorable approach, as that way, replying with RSS or table type results is dependent only on the text support feature in the messenger. This way RSS or table type replies can be shown in messengers like REST API Gitter (which do not provide any other reply type support except text).

In SUSI web app, the UI of the web search and table type results:

As the SUSI web app is made in React js, it provided the app with necessary features to show the results this way. The messengers may not be having these required features.

So the task is we need to convert the RSS or table type replies by SUSI API to plain text to send it to the messenger.

Let’s work it out.

Converting RSS results to text:

First get familiar with the SUSI API reply to query “why” by visiting this url – http://api.susi.ai/susi/chat.json?q=why.

The JSON object returned will be constituting an array of objects as the value of the data key like:

"data": [
        {
        "title": "Why is Oracle male?",
        "description": "Why is Oracle male?. http dba oracle com why is male htm. Oracle Why is masculine?. ",
        "link": "http://dba-oracle.com/t_why_is_oracle_male.htm"
      }
]

If we notice carefully each object has 3 main keys namely “title”, “description” and “link”. So extracting these 3 properties from each object and binding them together into 1 string is the task we need to do.

So we traverse each object (i.e. rss result) in the data array and we keep on appending the values of “title”, “description” and “link” key values to the ans variable. At the end we send this resultant string to the messenger bot as a reply.  

Suppose we have the returned JSON object, in the data variable.

// storing the number of rss results
var metaCnt = data.answers[0].metadata.count;
    for(var i=0;i<metaCnt;i++){
        ans += ('Title : ');
ans += data.answers[0].data[i].title+', ';
        ans += ('Link : ');
        ans += data.answers[0].data[i].link+', ';
        ans += '\n\n';
    }

// send the message in ans variable to the messenger

Converting table type replies to text:

First get familiar with the SUSI API reply to query “why” by visiting this url – http://api.susi.ai/susi/chat.json?q=universities+in+australia.

The JSON object returned will be constituting an array of objects representing universities as the value of the data key in this form:

{
    "alpha_two_code": "AU",
    "name": "Australian Correspondence Schools",
    "country": "Australia",
    "web_page": "http://www.acs.edu.au/"
}

Here too, each object has 3 main keys namely “name”, “country” and “web_page”. So extracting these 3 properties from each object and binding them together into 1 string can make the things work.

Again traverse each object (i.e. university object) in the data array and we keep on appending the values of “name”, “country” and “web_page” key values to the ans variable. At the end we send this resultant string to the messenger bot as a reply.

Suppose we have the returned JSON object in the data variable.

    // the 3 main columns which we need are stored in colNames variable
    var colNames = data.answers[0].actions[0].columns;
    
    // storing the number of table entries
    var metaCnt = data.answers[0].metadata.count;
    for(var i=0;i<metaCnt;i++){
        for(var cN in colNames){
            // The column name
            ans += (colNames[cN]+' : ');
            // value for that column name
            ans += data.answers[0].data[i][cN]+', ';
        }
        ans += '\n\n';
    }

    // send the message in ans variable to the messenger

Resources

  1. By Slobodan Stojanović from smashing magazineDevelop a chat bot with node js.
  2. By Mikhail Larionov from Facebook blogsList templates and check box plugin
Continue ReadingShowing RSS and Table Type Replies in SUSI Messenger Bots