Change Background of Message Section in SUSI.AI Web Chat

In SUSI.AI Web Chat we pay special attentions to the UI to make it easy to use and attract more users. Many chat apps offer users customization such as changing the colors and background. As this is very popular we decided to give the option to customize the background of message section of SUSI.AI web chat. The UI of the message section had a default gray background which could not be modified by the user. The goal was now to allow the user to customize the look of his or her own client starting with the background of the message section.

We added the settings to change the background image to Custom theme menu which occurs only when the user is logged in.The option looks like this:

User can add URL of any image of his/her choice and it will be set as the background of message section.
Now let’s take a look at the implementation of this option. We added messageBackgroundImage to our state of message section and initialised it to ‘ ’ (empty string). The TextField component looks like this:

<TextField
   name="messageImg"
   style={{display:component.component==='body'?'block':'none'}}
   ref={(input) => { this.backImage = input }}
   onChange={
     (e,value)=>
     this.handleChangeMessageBackground(value) }
    value={this.state.messageBackgroundImage}
    floatingLabelText="Message Background Image URL"
 />

OnChange method handles the input URL and this calls the function handleChangeMessageBackground. This function is having following implementation:

handleChangeMessageBackground(backImage){
    this.setState({messageBackgroundImage:backImage});
  }

It sets the messageBackgroundImage equal to the URL entered by the user. Now to change the background of message section we made a custom React style object messageBackgroundStyles:

const messageBackgroundStyles = {
        backgroundImage: `url(${this.state.messageBackgroundImage})`,
        backgroundRepeat: 'no-repeat',
        backgroundSize: '100% 100%'
    }

In the above object,

backgroundImage: sets the message section background to the image on URL entered by user

backgroundImage: Does not allows to the image to be repeated.

backgroundSize: Fills the entire message section with the background image.

Now this style was added to the component of message section :

<div className='message-section'>
     <ul
      className='message-list'
      ref={(c) => { this.messageList = c; }}
      style={messageBackgroundStyles}> // Styles added
       // Some relevant code    
      </ul>
</div>

Now we have following UI, where the user can modify the message section background with image of his or her own choice. User can add any image as background of message section by adding appropriate URL. The image will cover the entire message section without repeating itself.

Background Image source:link

This gives our user custom look according to his/her own choice.

Resources

Testing Link

http://chat.susi.ai

Background Image source of featured image : link

Continue ReadingChange Background of Message Section in SUSI.AI Web Chat

Sorting Photos in Phimpme Android

The Phimpme Android application features a fully fledged gallery interface with an option to switch to all photos mode, albums mode and to sort photos according to various sort actions. Sorting photos via various options helps the user to get to the desired photo immediately without having to scroll down till the end in case it is the last photo in the list generated automatically by scanning the phone for images using the Android’s mediaStore class. In this tutorial, I will be discussing how we achieved the sorting option in the Phimpme application with the help of some code snippets.

To sort the all photos list, first of all we need a list of all the photos by scanning the phone using the media scanner class via the code snippet provided below:

uri = android.provider.MediaStore.Images.Media.EXTERNAL_CONTENT_URI;
      String[] projection = {MediaStore.MediaColumns.DATA};
      cursor = activity.getContentResolver().query(uri, projection, null, null, null);

In the above code we are using a cursor to point to each photos and then we are extracting the path of the images and storing it in a list using a while loop. After we generate the list of path of all the images, we have to convert the into a list of media using the file path using the code below:

for (String path : listOfAllImages) {
          list.add(new Media(new File(path)));
      }
      return list;
  }

After generating the list of all images we can sort the photos using the Android’s collection class. In Phimpme Android we provide the option to sort photos in different categories namely:

  1. Name Sort action
  2. Date Sort action
  3. Size Sort action
  4. Numeric Sort action

As sorting is somewhat heavy task so doing this in the main thread will result in freezing UI of the application so we have to put this into an AsyncTask with a progress dialog to sort the photos. After putting the above four options in the menu options. We can define an Asynctask to load the images and in the onPreExecute method of the AsyncTask, we are displaying the progress dialog to the user to depict that the sorting process is going on as depicted in the code snippet below

AlertDialog.Builder progressDialog = new AlertDialog.Builder(LFMainActivity.this, getDialogStyle());
dialog = AlertDialogsHelper.getProgressDialog(LFMainActivity.this, progressDialog,
      getString(R.string.loading_numeric), all_photos ? getString(R.string.loading_numeric_all) : getAlbum().getName());
dialog.show();

In the doInBackgroundMethod of the AsyncTask, we are sorting the list of all photos using the Android’s collection class and using the static sort method defined in that class which takes the list of all the media files as a parameter and the MediaComparator which takes the sorting mode as the first parameter and the sorting order as the second. The sorting order decides whether to arrange the list in ascending or in descending order.

getAlbum().setDefaultSortingMode(getApplicationContext(), NUMERIC);
Collections.sort(listAll, MediaComparators.getComparator(getAlbum().settings.getSortingMode(), getAlbum().settings.getSortingOrder()));

After sorting, we have to update the data set to reflect the changes of the list in the UI. This we are doing in the onPostExecute method of the AsyncTask after dismissing the progress Dialog to avoid the window leaks in the application. You can read more about the window leaks in Android due to progressdialog here.

dialog.dismiss();
mediaAdapter.swapDataSet(listAll);

To get the full source code, you can refer the Phimpme Android repository listed in the resources below.

Resources

  1. Android developer guide to mediastore class: https://developer.android.com/reference/android/provider/MediaStore.html
  2. GitHub LeafPic repository: https://github.com/HoraApps/LeafPic
  3. Stackoverflow – Preventing window leaks in Android: https://stackoverflow.com/questions/6614692/progressdialog-how-to-prevent-leaked-window
  4. Blog – Sorting lists in Android: http://www.worldbestlearningcenter.com/tips/Android-sort-ListView.htm
Continue ReadingSorting Photos in Phimpme Android

Teaching Susi

In this blog post, I’ll explain how you can add a skill to SUSI.

Skills to be added in this tutorial:

  1. Ask SUSI for conversion of decimal into Binary .
  2. Ask SUSI to tell a quote.
  3. Skill development in SUSI is very interesting and easy. A comprehensive guide for skill development can be found here.

We have a Susi Skill development environment based on an Etherpad. Are you unaware what an Etherpad is? It is a blank web page where you can just put in your text and everyone can collaborate.

Here is a screenshot of what etherpad looks like:

  • open http://dream.susi.ai
  • name a dream (just pick a name for your tests in lower case letters)
  • the etherpad is filled with a welcome message, just delete the content completely

Ask SUSI for conversion of decimal into Binary

“*” here represents any decimal number number.Suppose we enter a decimal number and want susi to return it’s binary equivalent. So to make our skill, first of all we should form a general query.

Query: Convert * into binary or * in binary

After defining our query we want susi to reply with relevant answer.

For taking out the conversion from decimal to binary we can use JavaScript.

We define Javascript syntax in etherpad as follows :

!javascript:Binary for $1$ = $!$
(+$1$).toString(2);
eol

Explanation :

!javascript” allows us to print like javascript console. We can add JavaScript code by using this and do mathematical calculations to convert our decimal into binary.

“Binary for $1$ = $!$”  represents output format where $1$ stores the value in “*” in query given by user to susi. $!$ will print the result of javascript code below.

“(+$1$).toString(2);” This is a single line javascript code while converts value in “$1$” to binary, It’s output is reflected in “$!$”

“eol” represents end of line, which signifies our code for skill is finished.

In etherpad our skill would look like :

“#” – used for commenting out lines in skill

#The following code returns binary equivalent of a decimal number given by user

convert * into binary || binary of *
!javascript:Binary for $1$ = $!$
(+$1$).toString(2);
eol

Screenshot:

Ask SUSI to tell a quote

We can use external API’s for providing answer to user’s query. The external API used for telling the quote is Quotes Rest API created by They Said So. It offers a very good quotes platform and also the quotes are not repeated when several requests are made continuously.

We need a query for our skill.

Query: Tell me a quote.

Now let’s say that we use this JSON response for fetching quote data.

Our JSON object looks like:

{
 success: {
  total: 1
 },
contents: {
quotes: [
{
quote: "Let our advance worrying become advance thinking and planning.",
length: "62",
author: "Winston Churchill",
tags: [
"anxiety",
"inspire",
"planning",
"time-management"
],
category: "inspire",
date: "2017-05-31",
permalink: "https://theysaidso.com/quote/o3aMOSUwOPqUnJv9YyfYHweF/winston-churchill-let-our-advance-worrying-become-advance-thinking-and-planning",
title: "Inspiring Quote of the day",
background: "https://theysaidso.com/img/bgs/man_on_the_mountain.jpg",
id: "o3aMOSUwOPqUnJv9YyfYHweF"
}
],
copyright: "2017-19 theysaidso.com"
}
}

Now we are interested in getting value corresponding to  quotes key, which takes the path:  “path”:”$.contents.quotes“.

We make output query as follows:

!console:$quote$
     {
        "url" : "http://quotes.rest/qod.json",
        "path":"$.contents.quotes"
      }
eol

Explanation :

“!console” prints the output and this is returned by SUSI.AI.

$quote$” is the key whose value will be printed from parsed JSON.It contains our quote and this is a random quote. This is the string which will be responded by SUSI.AI

“url” generated JSON response from external API.

“path” is parsing path of JSON object which we follow for getting correct response.

Our skill will look like:

tell me a quote
!console:$quote$
     {
        "url" : "http://quotes.rest/qod.json",
        "path":"$.contents.quotes"
      }
eol

Screenshot :

Resources:

Continue ReadingTeaching Susi

Visualising Tweet Statistics in MultiLinePlotter App for Loklak Apps

MultiLinePlotter app is now a part of Loklak apps site. This app can be used to compare aggregations of tweets containing a particular query word and visualise the data for better comparison. Recently there has been a new addition to the app. A feature for showing tweet statistics like the maximum number of tweets (along with date) containing the given query word and the average number of tweets over a period of time. Such statistics is visualised for all the query words for better comparison.

Related issue: https://github.com/fossasia/apps.loklak.org/issues/236

Obtaining Maximum number of tweets and average number of tweets

Before visualising the statistics we need to obtain them. For this we simply need to process the aggregations returned by the Loklak API. Let us start with maximum number of tweets containing the given keyword. What we actually require is what is the maximum number of tweets that were posted and contained the user given keyword and on which date the number was maximum. For this we can use a function which will iterate over all the aggregations and return the largest along with date.

$scope.getMaxTweetNumAndDate = function(aggregations) {
        var maxTweetDate = null;
        var maxTweetNum = -1;

        for (date in aggregations) {
            if (aggregations[date] > maxTweetNum) {
                maxTweetNum = aggregations[date];
                maxTweetDate = date;
            }
        }

        return {date: maxTweetDate, count: maxTweetNum};
    }

The above function maintains two variables, one for maximum number of tweets and another for date. We iterate over all the aggregations and for each aggregation we compare the number of tweets with the value stored in the maxTweetNum variable. If the current value is more than the value stored in that variable then we simply update it and keep track of the date. Finally we return an object containing both maximum number of tweets and the corresponding date.Next we need to obtain average number of tweets. We can do this by summing up all the tweet frequencies and dividing it by number of aggregations.

$scope.getAverageTweetNum = function(aggregations) {
        var avg = 0;
        var sum = 0;

        for (date in aggregations) {
            sum += aggregations[date];
        }

        return parseInt(sum / Object.keys(aggregations).length);
    }

The above function calculates average number of tweets in the way mentioned before the snippet.

Next for every tweet we need to store these values in a format which can easily be understood by morris.js. For this we use a list and store the statistics values for individual query words as objects and later pass it as a parameter to morris.

var maxStat = $scope.getMaxTweetNumAndDate(aggregations);
        var avg = $scope.getAverageTweetNum(aggregations);

        $scope.tweetStat.push({
            tweet: $scope.tweet,
            maxTweetCount: maxStat.count,
            maxTweetOn: maxStat.date,
            averageTweetsPerDay: avg,
            aggregationsLength: Object.keys(aggregations).length
        });

We maintain a list called tweetStat and the list contains objects which stores the query word and the corresponding values.

Apart from plotting these statistics, the app also displays the statistics when user clicks on an individual treat present in the search record section. For this we filter tweetStat list mentioned above and get the required object corresponding to the query word the user selected bind it to angular scope. Next we display it using HTML.

<div class="tweet-stat max-tweet">
                  <div class="stat-label"> <h4>Maximum number of tweets containing '{{modalHeading}}' :</h4></div>
                  <div class="stat-value"> <strong>{{selectedTweetStat.maxTweetCount}}</strong> tweets on
                    <strong>{{selectedTweetStat.maxTweetOn}}</strong>
                  </div>
                </div>

Finally we need to plot the statistics. For this we use a function called plotStatGraph dedicated only for plotting statistics graph. We pass the tweetStat list as a parameter to morris and configure all the other parameters.

$scope.plotStatGraph = function() {
        $scope.plotStat = new Morris.Bar({
            element: 'graph',
            data: $scope.tweetStat,
            xkey: 'tweet',
            ykeys: ['maxTweetCount', 'averageTweetsPerDay'],
            labels: ['Maximum no. of tweets : ', 'Average no. of tweets/day'],
            parseTime: false,
            hideHover: 'auto',
            resize: true,
            stacked: true,
            barSizeRatio: 0.40
        });
        $scope.graphLoading = false;
    }

But now we have two graphs. One for showing variations in aggregation and the other for showing statistics. How do we manage them? Somehow we need to show them in the same page as this is a single page app. Also we need to avoid vertical scrolling as it would degrade both UI and UX. So we need to implement a switching mechanism. The user should be able to switch between the two graph views as per their wish. How to achieve that? Well, for this we maintain a global variable which will keep track of the current plot type. If the current graph type is aggregations then we call the function to plot aggregations otherwise we call the above mentioned function to plot statistics.

$scope.plotData = function() {
        $(".plot-data").html("");
        if ($scope.currentGraphType === "aggregations") {
            $scope.plotAggregationGraph();
        } else {
            $scope.plotStatGraph();
        }
    }

Lastly we integrate this state variable (currentGraphType) with the UI so that users can easily toggle between graph views with just a click.

<div class="switch" ng-click="toggle()">
                <span ng-if="queryRecords.length !== 0" class="glyphicon glyphicon-stats"></span>
              </div>

Important resources

Continue ReadingVisualising Tweet Statistics in MultiLinePlotter App for Loklak Apps

Integrating Swagger with SUSI Server

The goal of Swagger is to define a standard interface for REST APIs which allows humans to understand the capabilities of the APIs without access to source code or documentation.

SUSI Server is now completely API centric. As more and more people make their own bots using SUSI Server they will be needing documentation for the APIs. We can use swagger so that without looking at the javadocs or documentation people can consume the REST APIs.

In this I post will walk you through the steps to integrate Swagger with SUSI Server which is running on Jetty.

Add the following dependencies to build.gradle file. These add Swagger Annotations, Swagger Models, Swagger UI and Glassfish containers.

  compile group: 'io.swagger', name: 'swagger-annotations',version: swaggerVersion
  compile group: 'io.swagger', name: 'swagger-core', version: swaggerVersion
  compile group: 'io.swagger', name: 'swagger-jersey2-jaxrs', version: swaggerVersion
  compile group: 'io.swagger', name: 'swagger-models', version: swaggerVersion
  compile group: 'org.glassfish.jersey.core', name: 'jersey-server', version: versionJersey
  compile group: 'org.glassfish.jersey.containers', name: 'jersey-container-servlet-core', version: versionJersey
  compile group: 'org.webjars', name: 'swagger-ui', version: '2.1.4'

Now SusiServer.class is the main file which initializes all the servlets and server handlers.

Here, we need to tell the SusiServer to look for the swagger annotations and use them to build the documentation.

In the main function, before starting the server we set the serverHandlers from setServerHandler function.

public static void main(String[] args) throws Exception {
  // init the http server
        try {
            setupHttpServer(httpPort, httpsPort);
        } catch (Exception e) {
            Log.getLog().warn(e.getMessage());
            System.exit(-1);
        }
        setServerHandler(dataFile);
        
        SusiServer.server.start();
        SusiServer.caretaker = new Caretaker();
        SusiServer.caretaker.start();

Now, we will modify the setServetHandler function for registering the Swagger Handler.

There are already 2 handlers so I used a handerCollection object, so that we can give multiple handlers to handerCollection and set the serverHandler as handerCollection.

private static void setServerHandler(File dataFile){
         ServletContextHandler servletHandler = new ServletContextHandler();
       handlerCollection.addHandler(ipaccess);
        ContextHandlerCollection contexts = new ContextHandlerCollection();
        ServletContainer sc = new ServletContainer(resourceConfig);
        ServletHolder holder = new ServletHolder(sc);

        servletHandler.addServlet(holder, "/docs/*");
        handlerCollection.addHandler(contexts);

        SusiServer.server.setHandler(handlerCollection);
}

servletHandler.addServlet(holder, “/docs/*”), this line in the setServerHandler method sets the default swagger.json path to api.susi.ai/docs/swagger.json

This is all the basic setup to initialize swagger and now we need to modify our API endpoints and set annotations for the base URL and parameters.

Now I will discuss how to add swagger annotations to our Servlets.

For the demo, I will use GetAllUsers.class file, which returns the list of all users to Admins.

@Path("/aaa")
@Api(value = "/AAA")
@Produces({"application/json"})
public class GetAllUsers extends AbstractAPIHandler implements APIHandler {

Just before the class starts we will add the Path of the API endpoint and the result it produces. In this case, GetAllUsers returns JSON and is a part of aaa group.

@GET
@Path("/getAllUsers.json")
@ApiOperation(value = "Get All Users Registered on SUSI.AI",
        notes = "This API Endpoint returns the list of all users registered on SUSI Server",
        responseContainer = "Object")
@ApiResponses(value = { @ApiResponse(code = 200, message = "Success"),
@ApiResponse(code = 401, message = "Base user role not sufficient") })

Inside the class, we will declare the path of the API endpoint. A description of this endpoint and the sample response code.

In this file, there are only two possible responses.
1. Response code 200 – when everything goes well.

2. Response code 401 – when base user role is not sufficient, i.e when user is not Admin

When we save the file and browse to 127.0.0.1/docs/swagger.json
We get a response something like –

The Swagger UI is used to parse this JSON and create a UI for the documentation, where we can see the sample responses of the API endpoints, error codes and other things.

Swagger UI presents us the API documentation something like this image below.

Resources

Susi Server:  http://github.com/fossasia/susi_server/

Swagger: https://swagger.io/

Swagger and Jetty Tutorial: http://robferguson.org/2016/12/11/resteasy-embedded-jetty-fat-jars-swagger-and-swagger-ui/

Continue ReadingIntegrating Swagger with SUSI Server

Pie Chart Responses from SUSI Server

Giving out responses in charts and graphs is a very common reply of various assistants. We also have it in SUSI. We can show users the output of stocks, market covers and various percentages output in Pie Charts.  

A pie chart is a circular chart/graph which is divided in some segments like a pie. The segments in a pie chart shows the share of each object or category.

The PieChartServlet  in SUSI Server is a servlet that takes the JSON data of a the pie chart components as input parameters and returns an Image of the rendered Pie Chart..

public class PieChartServlet extends HttpServlet {

This is a simple HttpServlet. It does not require any authentication or base user role. So we extend the HttpServlet here.

    @Override
    protected void doGet(HttpServletRequest request, HttpServletResponse response) throws  IOException {

doGet is a method which is triggered whenever the PieChartServlet receives a GET Query. This will contain all the code that we will need to render the final output.

{"ford":"17.272992","toyota":"27.272992","renault":"47.272992"}

This is the sample JSON that we send to the PieChartServlet. This contains the names of the pie chart components and their respective percentages. After we receive these parameters we parse them and store them in our local variables.
These variables are then further used to plot the pie chart.

To plot these values in pie chart we have used a library JFreeChart.

This is a free and well documented Java chart library. This library supports PNGs and JPEGs as well as vector graphics file formats

JFreeChart chart = getChart(json, legendBit, tooltipBit);

From here we call a function getChart This function accept 3 parameters. The json which we sent as the GET parameter, legendBit and tooltilBit. These are also sent as GET parameters. In this example I will use legendBit as true and tooltipBit as false.

        JFreeChart chart = ChartFactory.createPieChart("SUSI Visualizes -   PieChart", dataset, legend, tooltips, urls);

        chart.setBorderPaint(Color.BLACK);
        chart.setBorderStroke(new BasicStroke(5.0f));
        chart.setBorderVisible(true);

        return chart;

 This is the function getChart. It creates a chart using the ChartFactory method and set sets the SUSI branding on it as “SUSI Visualizes – PieChart”. It accepts the datasets, legends and tool tips. The variable, dataset is nothing but the json keys and their values.

After the ChartFactory returns the chart we set the border of the chart and returns a pie chart back the function where it was called.

ChartUtilities.writeChartAsPNG(outputStream, chart, width, height);

Finally we write the chart as a PNG image and send it to the user.

Output

This can be tested at

https://api.susi.ai/vis/piechart.png?data={%22ford%22:%2217.272992%22,%22toyota%22:%2227.272992%22,%22renault%22:%2247.272992%22}&width=1000&height=1000&legend=true&tooltip=false

References

Continue ReadingPie Chart Responses from SUSI Server

Developing MultiLinePlotter App for Loklak

MultiLinePlotter is a web application which uses Loklak API under the hood to plot multiple tweet aggregations related to different user provided query words in the same graph. The user can give several query words and multiple lines for different queries will be plotted in the same graph. In this way, users will be able to compare tweet distribution for various keywords and visualise the comparison. All the searched queries are shown under the search record section. Clicking on a record causes a dialogue box to pop up where the individual tweets related to the query word is displayed. Users can also remove a series from the plot dynamically by just pressing the Remove button beside the query word in record section. The app is presently hosted on Loklak apps site.

Related issue – https://github.com/fossasia/apps.loklak.org/issues/225

Getting started with the app

Let us delve into the working of the app. The app uses Loklak aggregation API to get the data.

A call to the API looks something like this:

http://api.loklak.org/api/search.json?q=fossasia&source=cache&count=0&fields=created_at

A small snippet of the aggregation returned by the above API request is shown below.

"aggregations": {"created_at": {
    "2017-07-03": 3,
    "2017-07-04": 9,
    "2017-07-05": 12,
    "2017-07-06": 8,
}}

The API provides a nice date v/s number of tweets aggregation. Now we need to plot this. For plotting Morris.js has been used. It is a lightweight javascript library for visualising data.

One of the main features of this app is addition and removal of multiple series from the graph dynamically. How do we achieve that? Well, this can be achieved by manipulating the morris.js data list whenever a new query is made. Let us understand this in steps.

At first, the data is fetched using angular HTTP service.

$http.jsonp('http://api.loklak.org/api/search.json?callback=JSON_CALLBACK',
            {params: {q: $scope.tweet, source: 'cache', count: '0', fields: 'created_at'}})
                .then(function (response) {
                    $scope.getData(response.data.aggregations.created_at);
                    $scope.plotData();
                    $scope.queryRecords.push($scope.tweet);
                });

Once we get the data, getData function is called and the aggregation data is passed to it. The query word is also stored in queryRecords list for future use.

In order to plot a line graph morris.js requires a data object which will contain the required values for a series. Given below is an example of such a data object.

data: [
    { x: '2006', a: 100, b: 90 },
    { x: '2007', a: 75,  b: 65 },
    { x: '2008', a: 50,  b: 40 },
    { x: '2009', a: 75,  b: 65 },
],

For every ‘x’, ‘a’ and ‘b’ will be plotted. Thus two lines will be drawn. Our app will also maintain a data list like the one shown above, however, in our case, the data objects will have a variable number of keys. One key will determine the ‘x’ value and other keys will determine the ordinates (number of tweets).

All the data objects present in the data list needs to be updated whenever a new search is done.

The getData function does this for us.

var value = $scope.tweet;
        for (date in aggregations) {
            var present = false;
            for (var i = 0; i < $scope.data.length; i++) {
                var item = $scope.data[i];
                if (item['day'] === date) {
                    item[[value]] = aggregations[date];
                    $scope.data[i] = item
                    present = true;
                    break;
                }
            }
            if (!present) {
                $scope.data.push({day: date, [value]: aggregations[date]});
            }
        }


The for loop in the above code snippet updates the global data list used by morris.js. It simply iterates over the dates in the aggregation, extracts the object corresponding to a particular date, adds the new query word as a key and, the number of tweets on that date as the value.If a date is not already present in the list, then it inserts a new object corresponding to the date and query word. Once our data list is updated, we are ready to redraw the graph with the updated data. This is done using plotData function. The plotData function simply checks the user selected graph type. If the selected type is aggregations then it calls plotAggregationGraph() to redraw the aggregations plot.

$scope.remove = function(record) {
        $scope.queryRecords = $scope.queryRecords.filter(function(e) {
            return e !== record });

        $scope.data.forEach(function(item) {
            delete item[record];
        });

        $scope.data = $scope.data.filter(function(item) {
            return Object.keys(item).length !== 1;
        });

        $scope.ykeys = $scope.ykeys.filter(function(item) {
            return item !== record;
        });

        $scope.labels = $scope.labels.filter(function(item) {
            return item !== record;
        });

        $scope.plotData();
}

The above function simply scans the data list, filters the objects which contains selected record as a key and removes them using filter method of javascript arrays. It also removes the corresponding labels and entries from labels and ykeys arrays. Finally, it once again calls plotData function to redraw the plot.

Given below is a sample plot generated by this app with the query words – google, android, microsoft, samsung.

 

Conclusion

This blog post explained how multiple series are plotted dynamically in the MultiLinePlotter app. Apart from aggregations plot it also plots tweet statistics like maximum tweets and average tweets containing a query word and visualises them using stacked bar chart. I will be discussing about them in my subsequent blogs.

Important resources

Continue ReadingDeveloping MultiLinePlotter App for Loklak

Managing Related Endpoints in Permission Manager of Open Event API Server

Open Event API Server has its permission manager to manage all permission to different endpoints and some of the left gaps were filled by new helper method has_access. The next challenge for permission manager was to incorporate a feature many related endpoints points to the same resource.
Example:

  • /users-events-roles/<int:users_events_role_id>/user or
  • /event-invoices/<int:event_invoice_id>/user

Both endpoints point to Users API where they are fetching the record of a single user and for this, we apply the permission “is_user_itself”. This permission ensures that the logged in user is the same user whose record is asked through the API and for this we need the “user_id” as the “id” in the permission function, “is_user_itself”
Thus there is need to add the ability in permission manager to fetch this user_id from different models for different endpoints. For example, if we consider above endpoints then we need the ability to get user_id from UsersEventsRole and EventInvoice models and pass it to permission function so that it can use it for the check.

Adding support

To add support for multiple keys, we have to look for two things.

  • fetch_key_url
  • model

These two are key attributes to add this feature, fetch_key_url will take the comma separated list which will be matched with view_kwargs and model receives the array of the Model Classes which will be used to fetch the related records from the model
This snippet provides the main logic for this:

for index, mod in enumerate(model):
   if is_multiple(fetch_key_url):
       f_url = fetch_key_url[index]
   else:
       f_url = fetch_key_url
   try:
       data = mod.query.filter(getattr(mod, fetch_key_model) == view_kwargs[f_url]).one()
   except NoResultFound, e:
       pass
   else:
       found = True

if not found:
   return NotFoundError({'source': ''}, 'Object not found.').respond()

From the above snippet we are:

  • We iterate through the models list
  • Check if fetch_key_url has multiple keys or not
  • Get the key from fetch_key_url on the basis of multiple keys or single key in it.
  • We try to attempt to get object from model for the respective iteration
  • If there is any record/object in the database then it’s our data. Skipping further process
  • Else continue iteration till we get the object or to the end.

To use multiple mode

Instead of providing the single model to the model option of permission manager, provide an array of models. Also, it is optional to provide comma separated values to fetch_key_url
Now there can be scenario where you want to fetch resource from database model using different keys present on your view_kwargs
for example, consider these endpoints

  1. `/notifications/<notification_id>/event`
  2. `/orders/<order_id>/event`

Since they point to same resource and if you want to ensure that logged in user is organizer then you can use these two things as:

  1. fetch_key_url=”notification_id, order_id”
  2. model=[Notification, Order]

Permission manager will always match indexes in both options, the first key of fetch_key_url will be only used for the first key of the model and so on.
Also, fetch_key_url is an optional parameter and even in multiple mode you can provide a single value as well.  But if you provide multiple commas separated values make sure you provide all values such that no of values in fetch_key_url and model must be equal.

Resources

Continue ReadingManaging Related Endpoints in Permission Manager of Open Event API Server

Custom Data Layer in Open Event API Server

Open Event API Server uses flask-rest-jsonapi module to implement JSON API. This module provides a good logical abstraction in the data layer.
The data layer is a CRUD interface between resource manager and data. It is a very flexible system to use any ORM or data storage. The default layer you get in flask-rest-jsonapi is the SQLAlchemy ORM Layer and API Server makes use of default alchemy layer almost everywhere except the case where I worked on email verification part.

To add support for adding user’s email verification in API Server, there was need to create an endpoint for POST /v1/users/<int:user_id>/verify
Clearly here we are working on a single resource i.e, specific user record. This requires us to use ResourceDetail and the only issue was there is no any POST method or view in ResourceDetail class. To solve this I created a custom data layer which enables me to redefine all methods and views by inheriting abstract class. A custom data layer must inherit from flask_rest_jsonapi.data_layers.base.Base.

Creating Custom Layer

To solve email verification process, a custom layer was created at app/api/data_layers/VerifyUserLayer.py

def create_object(self, data, view_kwargs):
   user = safe_query(self, User, 'id', view_kwargs['user_id'], 'user_id')
   s = get_serializer()
   try:
       data = s.loads(data['token'])
   except Exception:
       raise UnprocessableEntity({'source': 'token'}, "Invalid Token")

   if user.email == data[0]:
       user.is_verified = True
       save_to_db(user)
       return user
   else:
       raise UnprocessableEntity({'source': 'token'}, "Invalid Token")

Using custom layer in API

We can easily provide custom layer in API Resource using one of the properties of the Resource Class

data_layer = {
   'class': VerifyUserLayer,
   'session': db.session
}

This is all we have to provide in the custom layer, now all CRUD method will be directed to our custom data layer.

Solution to our issue
Setting up custom layer provides us the ability to create our custom resource methods, i.e, modifying the view for POST request and allowing us to verify the registered users in API Server.
On Setting up the data layer all I need to do is create a ResourceList with using this layer and with permissions

class VerifyUser(ResourceList):

   methods = ['POST', ]
   decorators = (jwt_required,)
   schema = VerifyUserSchema
   data_layer = {
       'class': VerifyUserLayer,
       'session': db.session
   }

This enables me to use the custom layer, VerifyUserLayer for ResourceList resource.

Resources

Continue ReadingCustom Data Layer in Open Event API Server

A guide to use Permission Manager in Open Event API Server

This article provides a simple guide to use permission manager in Open Event API Server. Permission manager is constantly being improved and new features are being added into it. To ensure that all co-developers get to know about it and make use of them, this blog posts describes every part of permission manager.

Bootstrapping

Permission manager as a part of flask-rest-jsonapi works as a decorator for different resources of the API. There are two ways to provide the permission decorator to any view

  • First one is to provide it in the list of decorators
decorators = (api.has_permission('is_coorganizer', fetch="event_id",
                                fetch_as="event_id", model=StripeAuthorization),)
    • Second way is to explicitly provide it as a decorator to any view
@api.has_permission('custom_arg', custom_kwargs='custom_kwargs')
    def get(*args, **kwargs):
        return 'Hello world !'

In the process of booting up, we first need to understand the flow of Resources in API. All resources even before doing any schema check, call the decorators. So this way you will not get any request data in the permission methods. All you will receive is a dict of the URL parameters but again it will not include the filter parameters.
Permission Manager receives five parameters as: 

def permission_manager(view, view_args, view_kwargs, *args, **kwargs):

First three are provided into it implicitly by flask-rest-jsonapi module

  • view: This is the resource’s view method which is called through the API. For example, if I go to /events then the get method of ResourceList will be called.
  • view_args: These are args associated with that view.
  • view_kwargs: These are kwargs associated with that resource view. It includes all your URL parameters as well.
  • args: These are the custom args which are provided when calling the permission manager. Here at permission manager is it expected that the first index of args will be the name of permission to check for.
  • kwargs: This is the custom dict which is provided on calling the permission manager. The main pillar of the permission manager. Described below in usage.

Using Permission Manager

Using permission manager is basically understanding the different options you can send through the kwargs so here is the list of the things you can send to permission manager
These are all described in the order of priority in permission manager

  • method (string): You can provide a string containing the methods where permission needs to be checked as comma separated values of different methods in a string.
    For example: method=”GET,POST”
  • leave_if (lambda): This receives a lambda function which should return boolean values. Based on returned value if is true then it will skip the permission check. The provided lambda function receives only parameter, “view_kwargs”
    Example use case can be the situation where you can leave the permission for any specifically related endpoint to some resource and would like to do a manual check in the method itself.
  • check (lambda): Opposite to leave_if. It receives a lambda function that will return boolean values. Based on returned value, If it is true then only it will go further and check the request for permissions else will throw forbidden error.
  • fetch (string): This is the string containing the name of the key which has to be fetched for the fetch_as key (described below). Permission manager will first look for this value in view_kwargs dict object. If it is not there then it will make the query to get one(described below at model )
  • fetch_as (string): This is the string containing the name of a key. The value of fetch key will be sent to the permission functions by this name.
  • model (string): This is one most interesting concept here. To get the value of the fetch key. Permission manager first looks into view_kwargs and if there no such value then you can still get one through the model. The model attribute here receives the class of the database model which will be used to get the value of the fetch key.
    It makes the query to get the single resource from this model and look for the value of the fetch key and then pass it to the permission functions/methods.
    The interesting part is that by default it uses <id> from view_kwargs to get the resource from the model but in any case if there is no specific ID with name <id> on the view_kwargs. You can use these two options as:
  • fetch_key_url (string): This is the name of the key whose value will be fetched from view_kwargs and will be used to match the records in database model to get the resource.
  • fetch_key_model (string): This is the name of the match column in the database model for the fetch_key_url, The value of it will be matched with the column named as the value of fetch_key_model.
    In case there is no record found in the model then permission manager will throw NotFound 404 Error.

A helper for permissions

The next big thing in permission manager is the addition of new helper function “has_access”

def has_access(access_level, **kwargs):
   if access_level in permissions:
       auth = permissions[access_level](lambda *a, **b: True, (), {}, (), **kwargs)
       if type(auth) is bool and auth is True:
           return True
   return False

This method allows you to check the permission at the mid of any method of any view and of any resource. Just provide the name of permission in the first parameter and then the additional options needed by the permission function as the kwargs values.
This does not throw any exception. Just returns the boolean value so take care of throwing any exception by yourselves.

Anything to improve on?

I will not say this exactly as the improvement but I would really like to make it more meaningful and interesting to add permission. May be something like this below:

permission = "Must be co_organizer OR track_organizer, fetch event_id as event_id, use model Event"

This clearly needs time to make it. But I see this as an interesting way to add permission. Just provide meaningful text and rest leave it to the permission manager.

Continue ReadingA guide to use Permission Manager in Open Event API Server