Adding Unit Test For Local JSON Parsing in Open Event Android App

The Open Event project uses JSON format for transferring event information like tracks, sessions, microlocations and other. The event exported in the zip format from the Open Event server also contains the data in JSON format. The Open Event Android application uses this JSON data. Before we use this data in the app, we have to parse the data to get Java objects that can be used for populating views. There is a chance that the model and the JSON format changes in future. It is necessary that the models are able to parse the JSON data and the change in the model or JSON format don’t break JSON parsing.  In this post I explain how to unit test local JSON parsing so that we can ensure that the models are able to parse the local JSON sample data successfully.

Firstly we need to access assets from the main source set into the unit test. There is no way to directly access assets from main source set. We need to first add assets in test/resources directory. If assets are present in test/resources directory then we can use it using ClassLoader in the unit test. But we can’t just copy assets from the main source set to resources directory. If there is any change in sample JSON then we need to maintain both resources and it may make the sample inconsistent. We need to make assets shared.

Add the following code in the app level build.gradle file.

android {
    ...
    sourceSets.test.resources.srcDirs += ["src/main/assets"]
}

It will add src/main/assets as a source directory for test/resources directory.So after building the project the test will have access to the assets.

Create readFile() method

Now create a method readFile(String name) which takes a filename as a parameter and returns data of the file as a string.

private String readFile(String name) throws IOException {
        String json = "";
        try {
            InputStream inputStream = this.getClass().getClassLoader().getResourceAsStream(name);
            int size = inputStream.available();
            byte[] buffer = new byte[size];
            inputStream.read(buffer);
            inputStream.close();
            json = new String(buffer, "UTF-8");
        } catch (IOException e) {
            e.printStackTrace();
        }
        return json;
}

Here the getResourceAsStream() function is used to open file as a InputStream. Then we are creating byte array object of size same as inputStream data. Using read function we are storing data of file into byte array. After this we are creating a String object using a byte array.

Create ObjectMapper object

Create and initialize an ObjectMapper object in the Test class.

private ObjectMapper objectMapper;

    @Before
    public void setUp() {
        objectMapper = OpenEventApp.getObjectMapper();
        objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, true);
}

Here setting DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES to true is very important. It will fail the test for any unrecognized fields in the sample.

Create doModelDeserialization() method

In the Open Event Android App we are using Jackson for JSON serialization and deserialization. Converting JSON data to java is called deserialization or parsing.

Now create doModelDeserialization() method which takes three parameters,

  • Class<T> type: Model Class type of the data
  • String name: Name of the JSON data file
  • boolean isList: true if JSON string contains the list of object else false

This method returns true if parsing is successful and false if there is any error in parsing.

private <T> boolean doModelDeserialization(Class<T> type, String name, boolean isList) throws IOException {
        if (isList) {
            List<T> items = objectMapper.readValue(readFile(name), objectMapper.getTypeFactory().constructCollectionType(List.class, type));
            if (items == null)
                return false;
        } else {
            T item = objectMapper.readValue(readFile(name), type);
            if (item == null)
                return false;
        }
        return true;
}

Here ObjectMapper is doing the main work of parsing data and returns parsed object using readValue() method.

Add Test

Now all the setup is done we just need to assert value returned by doModelDeserialization() method by passing appropriate parameters.

@Test
public void testLocalJsonDeserialization() throws IOException {
        assertTrue(doModelDeserialization(Event.class, "event", false));
        assertTrue(doModelDeserialization(Microlocation.class, "microlocations", true));
        assertTrue(doModelDeserialization(Sponsor.class, "sponsors", true));
        assertTrue(doModelDeserialization(Track.class, "tracks", true));
        assertTrue(doModelDeserialization(SessionType.class, "session_types", true));
        assertTrue(doModelDeserialization(Session.class, "sessions", true));
        assertTrue(doModelDeserialization(Speaker.class, "speakers", true));
}

Here only event JSON file doesn’t have a list of the objects so passing false as isList parameter for others we are passing true because its data contains a list of objects.

Conclusion:

Running unit tests after every build helps you to quickly catch and fix software regressions introduced by code changes to your app

Continue ReadingAdding Unit Test For Local JSON Parsing in Open Event Android App

Generic Social Links Implementation in Open Event Android App

The Open Event Android App has an About Fragment which displays all the info about the event like name, time, location etc. It also shows social media buttons for the event. The problem was that the implementation of showing the social media buttons was not generic. The implementation was working fine for current FOSSASIA sample. If we generate the app for other events it creates a problem. It shows static buttons without proper mapping from social media button to social media link which creates a problem like on clicking GitHub button it opens Facebook link (issue #1792).

One solution to this problem is to implement recyclerview with social media buttons. In this post I explain how I have made social links implementation generic using RecyclerView.

Add RecyclerView in layout

The first step to do is to add recyclerview in the layout xml file and to create a list item for recyclerview which holds the image for the social link button. Then in the About Fragment find recyclerview element added in the xml file using findFragmentById() method.

1. Add recyclerview in xml file

In the layout xml file of About Fragment add recyclerview element. Define id, width, height, and gravity of recyclerview. Then specify list item for recyclerview using listitem attribute.

<android.support.v7.widget.RecyclerView
            android:id="@+id/list_social_links"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:layout_gravity="center"
            android:overScrollMode="never"
            android:clipToPadding="false"
            tools:listitem="@layout/item_social_link" />

2. Create item_social_link.xml

Now create a item_social_link.xml file and add a FrameLayout element. Inside the FrameLayout add an ImageView with appropriate id, width, height and padding.

<?xml version="1.0" encoding="utf-8"?>
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:id="@+id/layout_social_link_parent"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content">

    <ImageView
        android:id="@+id/img_social_link"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:background="?attr/selectableItemBackgroundBorderless"
        android:contentDescription="@string/social_link"
        android:padding="@dimen/padding_large"
        android:tint="@color/white" />
</FrameLayout>

Add and initialize RecyclerView

After adding recyclerview in xml file we need to add RecyclerView field in the About Fragment java file. Now add and initialize SocialLinksListAdapter which extends RecyclerView.Adapter and will be used for populating the recyclerview with the social links.

@BindView(R.id.list_social_links)
protected RecyclerView socialLinksRecyclerView;

private SocialLinksListAdapter socialLinksListAdapter;
private List<SocialLink> mSocialLinks = new ArrayList<>();

Here mSocialLinks is the list of social links which is fetched from the Event object.

Create ViewHolder for social link button

Now create SocialLinkViewHolder which extends RecyclerView.ViewHolder and holds one social link item defined in item_social_link.xml file. This file is where the magic happens. Add ImageView, FrameLayout, SocialLink and context fields in it.

public class SocialLinkViewHolder extends RecyclerView.ViewHolder {

    @BindView(R.id.img_social_link)
    protected ImageView imageView;

    @BindView(R.id.layout_social_link_parent)
    protected FrameLayout layout;

    private SocialLink socialLink;
    private Context context;

    public SocialLinkViewHolder(View itemView, Context context) {
        super(itemView);
        ButterKnife.bind(this, itemView);
        this.context = context;
    }

    public void bindSocialLink(@NonNull SocialLink socialLink) {...}

    private void setImageDrawable(@NonNull String name,@NonNull String link) {...}

    private Drawable getDrawable(@DrawableRes int id) {...}

    private void showView(boolean show) {...}
}

The bindSocialLink(SocialLink socialLink) method is called in the onBindViewHolder() method of the SocialLinksListAdapter. In this method initialize socialLink field and set drawable for ImageView according to SocialLink name using setImageDrawable() method.

public void bindSocialLink(@NonNull SocialLink socialLink) {
        this.socialLink = socialLink;
        setImageDrawable(socialLink.getName(), socialLink.getLink());
}

The setImageDrawable() method finds and sets appropriate image for social link button using the name of the social link. If the image for the social link is not found then it sets the visibility of the image view to GONE using showView(boolean show).

private void setImageDrawable(@NonNull String name,@NonNull String link) {

        if (Utils.isEmpty(name) || Utils.isEmpty(link) || Utils.getSocialLinkDrawableId(name) == 1) {
            showView(false);
            return;
        }
        imageView.setImageDrawable(getDrawable(Utils.getSocialLinkDrawableId(name)));
}

The showView(boolean show) method handles visibility of ImageView and ImageView’s parent FrameLayout using setVisibility() method.

private void showView(boolean show) {
        if (show) {
            imageView.setVisibility(View.VISIBLE);
            layout.setVisibility(View.VISIBLE);
        } else {
            imageView.setVisibility(View.GONE);
            layout.setVisibility(View.GONE);
        }
}

Set onClickListener to ImageView

Now it’s time to set the onClickListener in the constructor of the SocialLinkViewHolder to define what to do when user clicks on the ImageView.

imageView.setOnClickListener(view -> {
            if (socialLink != null && !Utils.isEmpty(socialLink.getLink())) {
                Utils.setUpCustomTab(context, socialLink.getLink());
            }
        });

Here we are setting custom tab for the social link using the setUpCustomTab() util method which will open the link.

Now run the app on the device or emulator. Here’s how it looks like,

Conclusion

The SocialLink impementation using the RecyclerView gives the great user experience in the application.

Additional resources

Continue ReadingGeneric Social Links Implementation in Open Event Android App

Working With Inter-related Resource Endpoints In Open Event API Server

In this blogpost I will discuss how the resource inter-related endpoints work. These are the endpoints which involve two resource objects which are also related to another same resource object.


The discussion in this post is of the endpoints related to Sessions Model. In the API server, there exists a relationship between event and sessions. Apart from these, session also has relationships with microlocations, tracks, speakers and session-types. Let’s take a look at the endpoints related with the above.

`/v1/tracks/<int:track_id>/sessions` is a list endpoint which can be used to list and create the sessions related to a particular track of an event. To get the list we define the query() method in ResourceList class as such:

The query method is executed for GET requests, so this if clause looks for track_id in view_kwargs dict. When the request is made to `/v1/tracks/<int:track_id>/sessions`, track id will be present as ‘track_id in the view_kwargs. The tracks are filtered based on the id passed here and then joined on the query with all sessions object from database.

For the POST method, we need to add the track_id from view_kwargs to pass into the track_id field of database model. This is achieved by using flask-rest-jsonapi’s before_create_object() method. The implementation for track_id is the following:

When a POST request is made to `/v1/tracks/<int:track_id>/sessions`, the view_kwargs dict will have ‘track_id’ in it. So if track_id is present in the url params, we first ensure that a track with the passed id exists, then only proceed to create a sessions object under the given track. Now the safe_query() method is a generic custom method written to check for such things. The model is passed along with the filter attribute, and a field to include in the error message. This method throws an ObjectNotFound exception if no such object exists for a given id.

We also need to take care of the permissions for these endpoints. As the decorators are called even before schema validation, it was difficult to get the event_id for permissions unless adding highly endpoint-specific code in the permission manager core, leading to loss of generality.  So the leave_if parameter of permission check was used to overcome this issue. Since the permissions manager isn’t fully developed yet, this is to be changed in the improved implementation of the permissions manager.

Similar implementations for micro locations and session types was done. All the same is not explained in this blogpost. For extended code, take a look at the source code of this schema.

For speaker relation, a few things were different. This is because speakers-sessions is a many-to-many relationship. Let’s take a look at this:

As it is a many-to-many relationship, a association_table was used with flask-sqlalchemy. So for the query() method, the same association table is queried after extracting the speaker_id from view_kwargs dict.

For the POST request on `/v1/speakers/<int:speaker_id>/sessions` , flask-rest-jsonapi’s after_create_object() method is used to insert the request in association_table. In this method the parameters are the following: self, obj, data, view_kwargs

Now view_kwargs contains the url parameters, so we make a check for speaker_id in view_wargs. If it is present, then before proceeding to insert data, we ensure that a speaker exists with that id using the safe_query() method as described above. After that, the obj argument of the method is used. This contains the object that was created in  previous method. So now once the sessions object has been created and we are sure that a speaker exists with given speaker_id, this is just to be appended to obj.speakers  so that this relationship tuple is inserted into the association table.

This updates the association table ‘speakers_session’ in this case. The other such endpoints are being worked upon in a similar fashion and will be consolidated as part of a set of robust APIs, along with the improved permissions manager for the Open Event API Server.

Additional Resources

Code, Issues and Pull Request involved

Continue ReadingWorking With Inter-related Resource Endpoints In Open Event API Server

Real time Sensor Data Analysis on PSLab Android

PSLab device has the capacity to connect plug and play sensors through the I2C bus. The sensors are capable of providing data in real time. So, the PSLab Android App and the Desktop app need to have the feature to fetch real time sensor values and display the same in the user interface along with plotting the values on a simple graph.

The UI was made following the guidelines of Google’s Material Design and incorporating some ideas from the Science Journal app. Cards are used for making each section of the UI. There are segregated sections for real time updates and plotting where the real time data can be visualised. A methods for fetching the data are run continuously in the background which receive the data from the sensor and then update the screen.

The following section denotes a small portion of the UI responsible for displaying the data on the screen continuously and are quite simple enough. There are a number of TextViews which are being constantly updated on the screen. Their number depends on the type and volume of data sent by the sensor.

<TextView
       android:layout_width="wrap_content"
       android:layout_height="30dp"
       android:layout_gravity="start"
       android:text="@string/ax"
       android:textAlignment="textStart"
       android:textColor="@color/black"
       android:textSize="@dimen/textsize_edittext"
       android:textStyle="bold" />

<TextView
       android:id="@+id/tv_sensor_mpu6050_ax"
       android:layout_width="wrap_content"
       android:layout_height="30dp"
       android:layout_gravity="start"
       android:textAlignment="textStart"
       android:textColor="@color/black"
       android:textSize="@dimen/textsize_edittext"
       android:textStyle="bold" />

 

The section here represents the portion of the UI responsible for displaying the graph. Like all other parts of the UI of PSLab Android, MPAndroidChart is being used here for plotting the graph.

<LinearLayout
       android:layout_width="match_parent"
       android:layout_height="160dp"
       android:layout_marginTop="40dp">

       <com.github.mikephil.charting.charts.LineChart
               android:id="@+id/chart_sensor_mpu6050"
               android:layout_width="match_parent"
               android:layout_height="match_parent"
               android:background="#000" />
</LinearLayout>

 

Since the updates needs to continuous, a process should be continuously run for updating the display of the data and the graph. There are a variety of options available in Android in this regard like using a Timer on the UI thread and keep updating the data continuously, using ASyncTask to run a process in the background etc.

The issue with the former is that since all the processes i.e. fetching the data and updating the textviews & graph will run on the UI thread, the UI will become laggy. So, the developer team chose to use ASyncTask and make all the processes run in the background so that the UI thread functions smoothly.

A new class SensorDataFetch which extends AsyncTask is defined and its object is created in a runnable and the use of runnable ensures that the thread is run continuously till the time the fragment is used by the user.

scienceLab = ScienceLabCommon.scienceLab;
i2c = scienceLab.i2c;
try {
    MPU6050 = new MPU6050(i2c);
} catch (IOException e) {
    e.printStackTrace();
}
Runnable runnable = new Runnable() {
    @Override
    public void run() {
        while (true) {
            if (scienceLab.isConnected()) {
                try {
                    sensorDataFetch = new SensorDataFetch();
                } catch (IOException e) {
                    e.printStackTrace();
                }
                sensorDataFetch.execute();
            }
        }
    }
};
new Thread(runnable).start();

 

The following is the code for the ASyncTask created. There are two methods defined here doInBackground and onPostExecute which are responsible for fetching the data and updating the display respectively.

The raw data is fetched using the getRaw method of the MPU6050 object and stored in an ArrayList. The data type responsible for storing the data will depend on the return type of the getRaw method of each sensor class and might be different for other sensors. The data returned by getRaw is semi-processed and the data just needs to be split in sections before presenting it for display.

The PSLab Android app’s sensor files can be viewed here and they can give a better idea about how the sensors are calibrated, how the intrinsic nonlinearity is taken care of, how the communication actually works etc.

After the data is stored, the control moves to the onPostExecute method, here the textviews on the display and the chart are updated. The updation is slowed down a bit so that the user can visualize the data received.

private class SensorDataFetch extends AsyncTask<Void, Void, Void> {
   MPU6050 MPU6050 = new MPU6050(i2c);
   ArrayList<Double> dataMPU6050 = new ArrayList<Double>();

   private SensorDataFetch(MPU6050 MPU6050) throws IOException {
   }

   @Override
   protected Void doInBackground(Void... params) {
       try {
           if (MPU6050 != null) {
               dataMPU6050 = MPU6050.getRaw();
           }
       } catch (IOException e) {
           e.printStackTrace();
       }
           return null;
   }

   protected void onPostExecute(Void aVoid) {
       super.onPostExecute(aVoid);
       tvSensorMPU6050ax.setText(String.valueOf(dataMPU6050.get(0)));
       tvSensorMPU6050ay.setText(String.valueOf(dataMPU6050.get(1)));
       tvSensorMPU6050az.setText(String.valueOf(dataMPU6050.get(2)));
       tvSensorMPU6050gx.setText(String.valueOf(dataMPU6050.get(3)));
       tvSensorMPU6050gy.setText(String.valueOf(dataMPU6050.get(4)));
       tvSensorMPU6050gz.setText(String.valueOf(dataMPU6050.get(5)));
       tvSensorMPU6050temp.setText(String.valueOf(dataMPU6050.get(6)));
   }
}

The detailed implementation of the same can be found here.

Additional Resources

  1. Learn more about how real time sensor data analysis can be used in various fields like IOT http://ieeexplore.ieee.org/document/7248401/
  2. Google Fit guide on how to use native built-in sensors on phones, smart watches etc. https://developers.google.com/fit/android/sensors
  3. A simple starter guide to build an app capable of real time sensor data analysis http://developer.telerik.com/products/building-an-android-app-that-displays-live-accelerometer-data/
  4. Learn more about using AsyncTask https://developer.android.com/reference/android/os/AsyncTask.html
Continue ReadingReal time Sensor Data Analysis on PSLab Android

Using Map View to Display Location of Clicked Image in Phimpme

Previously in Phimpme Android app we used to display all the images on the map, based on their location they have taken. In the upcoming version, we are introducing mapview to specify the location when user checks out the details of the image. In this post, I am explaining that how I have implemented Google’s mapView in Phimpme. Note: The prerequisite to display image location in the map view, an image should have geolocation in its meta data.

Let’s get started

Step -1 : First, enable the setting to ‘Show mapview in Image description’ and save it somewhere.

Although, it’s your choice you want to give users choice to view map or not.

First, we need to turn on the movie visibility from map provider in settings. Right now we are adding two maps in our list Google maps and openstreetmap.

Choose Map Provider from the list in Phimpme

Once you choose your preference, it will get stored in sharedPref. As we are displaying the map in the image details so we need to add Image view of map.

Step -2 : Add a ImageView in your XML, in which we will display map.

<ImageView
   android:id="@+id/photo_map"
   android:layout_width="match_parent"
   android:layout_height="150dp"
   android:visibility="gone"
/>

Keep default visibility of mapview is GONE, If a user will enable map view in setting then we will make this image visible.

Step -3 : Load image with map in ImageView:

Good things are that StaticMapProvier gives you the URL of the image in corresponding to particular GeoLocation, which actually is a view of the map.

Now we need to display mapview when the user taps on details of an image.

ImageView imgMap = (ImageView) dialogLayout.findViewById(R.id.photo_map);
final GeoLocation location;
if ((location = f.getGeoLocation()) != null) {
   PreferenceUtil SP = PreferenceUtil.getInstance(activity.getApplicationContext());

   StaticMapProvider staticMapProvider = StaticMapProvider.fromValue(
           SP.getInt(activity.getString(R.string.preference_map_provider), StaticMapProvider.GOOGLE_MAPS.getValue()));

   Glide.with(activity.getApplicationContext())
           .load(staticMapProvider.getUrl(location))
           .asBitmap()
           .centerCrop()
           .animate(R.anim.fade_in)
           .into(imgMap);

To load an image, we used Glide Image library, you can use any library of your choice.

MapView on the top of dialog box in Phimpme

Step 4: Open navigation on the tap of mapView or Image.

Attach a click listener on the imageView and on click of that create a Uri correspond to geolocation and through an Intent to Android to view that link, it will open Google maps with navigation.

 String uri = String.format(Locale.ENGLISH, "geo:%f,%f?z=%d", location.getLatitude(), location.getLongitude(), 17);
            activity.startActivity(new Intent(Intent.ACTION_VIEW, Uri.parse(uri)));

Resources

Continue ReadingUsing Map View to Display Location of Clicked Image in Phimpme

Acceptance Testing of a Feature in Open Event Frontend

In Open Event Frontend, we have integration tests for ember components which are used throughout the project. But even after those tests, user interaction could pose some errors. We perform acceptance tests to alleviate such scenarios.

Acceptance tests interact with application as the user does and ensures proper functionality of a feature or to determine whether or not the software system has met the requirement specifications. They are quite helpful for ensuring that our core features work properly.

Let us write an acceptance test for register feature in Open Event Frontend.

import { test } from 'qunit';
import moduleForAcceptance from 'open-event-frontend/tests/helpers/module-for-acceptance';

moduleForAcceptance('Acceptance | register');

In the first line we import test from ‘ember-qunit’ (default unit testing helper suite for Ember) which contains all the required test functions. For example, here we are using test function to check the rendering of our component. We can use test function multiple times to check multiple components.

Next, we import moduleForAcceptance from ‘open-event-frontend/tests/helpers/module-for-acceptance’ which deals with application setup and teardown.

test('visiting /register', function(assert) {
  visit('/register');

  andThen(function() {
    assert.equal(currentURL(), '/register');
  });
});

Inside our test function, we simulate visiting  /register route and then check for the current route to be /register.

test('visiting /register and registering with existing user', function(assert) {
  visit('/register');
  andThen(function() {
    assert.equal(currentURL(), '/register');
    fillIn('input[name=email]', 'opev_test_user@nada.email');
    fillIn('input[name=password]', 'opev_test_user');
    fillIn('input[name=password_repeat]', 'opev_test_user');
    click('button[type=submit]');
    andThen(function() {
      assert.equal(currentURL(), '/register');
      // const errorMessageDiv = findWithAssert('.ui.negative.message');
      // assert.equal(errorMessageDiv[0].textContent.trim(), 'An unexpected error occurred.');
    });
  });
});

Then we simulate visiting /register route and register a dummy user. For this, we first go to /register route and then check for the current route to be register route. We fill the register form with appropriate data and hit submit.

test('visiting /register after login', function(assert) {
  login(assert);
  andThen(function() {
    visit('/register');
    andThen(function() {
      assert.equal(currentURL(), '/');
    });
  });
});

The third test is to simulate visiting /register route with user logged in and this is very simple. We just visit /register route and then check if we are are at / route or not because a user redirects to / route when he tries to visit /register after login.

And since we checked for all the possible combinations, to run the test we simply use the following command-

ember test --server

But there is a little demerit to acceptance tests. They boot up the whole EmberJS application and start us at the application.index route. We then have to navigate to the page that contains the feature being tested. Writing acceptance tests for each and every feature would be a big waste of time and CPU cycles. For this reason, only core features are tested for acceptance.

Resources

Continue ReadingAcceptance Testing of a Feature in Open Event Frontend

Adding API endpoint to SUSI.AI for Skill Historization

SUSI Skill CMS is an editor to write and edit skill easily. It follows an API centric approach where the Susi server acts as API server. Using Skill CMS we can browse history of a skill, where we get commit ID, commit message  and name the author who made the changes to that skills. In this blogpost we will see how to fetch complete commit history of a skill in the susi skill repository. A skill is a set of intents. One text file represents one skill, it may contain several intents which all belong together. Susi skills are stored in susi_skill_data repository. We can access any skill based on four tuples parameters model, group, language, skill.  For managing version control in skill data repository, the following dependency is added to build.gradle . JGit is a library which implements the Git functionality in Java.

dependencies {
 compile 'org.eclipse.jgit:org.eclipse.jgit:4.6.1.201703071140-r'
}

To implement our servlet we need to extend our servlet to AbstractAPIHandler. In Susi Server, an abstract class AbstractAPIHandler extending HttpServelets and implementing API handler interface is provided.

public class HistorySkillService extends AbstractAPIHandler implements APIHandler {}

The AbstractAPIHandler checks the permissions of the user using the userroles of and comparing it with the value minimum base role of each servlet. Thus to specify the user permission for a servlet we need Override the getMinimalBaseUserRole method.

 @Override
    public BaseUserRole getMinimalBaseUserRole() {
        return BaseUserRole.ANONYMOUS;
    }

UserRoles can be Admin, Privilege, User, Anonymous. In our case it is Anonymous. A User need not to log in to access this endpoint.

  @Override
    public String getAPIPath() {
        return "/cms/getSkillHistory.json";
    }

This methods sets the api endpoint path. One need to send requests at http://api.susi.ai/cms/getSkillHistory.json to get the modification history of skill. Next we will implement The ServiceImpl method where we will be processing the user request and giving back the service response.

@Override
    public ServiceResponse serviceImpl(Query call, HttpServletResponse response, Authorization rights, final JsonObjectWithDefault permissions) {

        String model_name = call.get("model", "general");
        File model = new File(DAO.model_watch_dir, model_name);
        String group_name = call.get("group", "knowledge");
        File group = new File(model, group_name);
        String language_name = call.get("language", "en");
        File language = new File(group, language_name);
        String skill_name = call.get("skill", "wikipedia");
        File skill = new File(language, skill_name + ".txt");
        JSONArray commitsArray;
        commitsArray = new JSONArray();
        String path = skill.getPath().replace(DAO.model_watch_dir.toString(), "models");
        //Add to git
        FileRepositoryBuilder builder = new FileRepositoryBuilder();
        Repository repository = null;
        try {
            repository = builder.setGitDir((DAO.susi_skill_repo))
                    .readEnvironment() // scan environment GIT_* variables
                    .findGitDir() // scan up the file system tree
                    .build();
            try (Git git = new Git(repository)) {
                Iterable<RevCommit> logs;
                logs = git.log().addPath(path).call();
                int i = 0;
                for (RevCommit rev : logs) {
                    commit = new JSONObject();
                    commit.put("commitRev", rev);
                    commit.put("commitName", rev.getName());
                    commit.put("commitID", rev.getId().getName());
                    commit.put("commit_message", rev.getShortMessage());
                    commit.put("author",rev.getAuthorIdent().getName());
                    commitsArray.put(i, commit);
                    i++;
                } success=true;
            } catch (GitAPIException e) {
                e.printStackTrace();
                success=false;
           } if(commitsArray.length()==0){
            success=false;
        }
        JSONObject result = new JSONObject();
        result.put("commits",commitsArray);
        result.put("success",success);
        return new ServiceResponse(result);
    }

To access any skill we need parameters model, group, language. We get this through call.get method where first parameter is the key for which we want to get the value and second parameter is the default value. Based on received model, group and language browse files in that folder we build the susi_skill_data repository path read the git variables and scan up the file system tree using FileRepositoryBuilder build() method. Next we fetch all the logs of the skill file and store them in json commits array and finally pass as a server response with success messages. In case of exceptions, pass service with success flags as false.

We have successfully implemented the servlet. Check the working of endpoint by sending request like http://api.susi.ai/cms/getSkillHistory.json?model=general&group=knowledge&language=en&skill=bitcoin and checking the response.

Susi skill cms uses this endpoint to fetch the skill history, try it out at http://skills.susi.ai/browseHistory

Resources
Continue ReadingAdding API endpoint to SUSI.AI for Skill Historization

Continuous Integration in Yaydoc using GitHub webhook API

In Yaydoc,  Travis is used for pushing the documentation for each and every commit. But this leads us to rely on a third party to push the documentation and also in long run it won’t allow us to implement new features, so we decided to do the continuous documentation pushing on our own. In order to build the documentation for each and every commit we have to know when the user is pushing code. This can be achieved by using GitHub webhook API. Basically we have to register our api to specific GitHub repository, and then GitHub will send a POST request to our API on each and every commit.

“auth/ci” handler is used to get access of the user. Here we request user to give access to Yaydoc such as accessing the public repositories , read organization details and write permission to write webhook to the repository and also I maintaining state by keeping the ci session as true so that I can know that this callback is for gh-pages deploy or ci deployOn

On callback I’m keeping the necessary informations like username, access_token, id and email in session. Then based on ci session state, I’m redirecting to the appropriate handler. In this case I’m redirecting to “ci/register”.After redirecting to the “ci/register”, I’m getting all the public repositories using GitHub API and then I’m asking the users to choose the repository on which users want to integrate Yaydoc CI

After redirecting to the “ci/register”, I’m getting all the public repositories using GitHub API and then I’m asking the users to choose the repository on which users want to integrate Yaydoc CI

router.post('/register', function (req, res, next) {
      request({
        url: `https://api.github.com/repos/${req.session.username}/${repositoryName}/hooks?access_token=${req.session.token}`,
        method: 'POST',
        json: {
          name: "web",
          active: true,
          events: [
            "push"
          ],
          config: {
            url: process.env.HOSTNAME + '/ci/webhook',
            content_type: "json"
          }
        }
      }, function(error, response, body) {
        repositoryModel.newRepository(req.body.repository,
          req.session.username,
          req.session.githubId,
          crypter.encrypt(req.session.token),
          req.session.email)
          .then(function(result) {
            res.render("index", {
              showMessage: true,
              messages: `Thanks for registering with Yaydoc.Hereafter Documentation will be pushed to the GitHub pages on each commit.`
            })
          })
      })
    }
  })

After user choose the repository, they will send a POST request to “ci/register” and then I’m registering the webhook to the repository and I’m saving the repository, user details in the database, so that it can be used when GitHub send request to push the documentation to the GitHub Pages.

router.post('/webhook', function(req, res, next) {
  var event = req.get('X-GitHub-Event')
  if (event == 'Push') {
      repositoryModel.findOneRepository(
        {
          githubId: req.body.repository.owner.id,
          name: req.body.repository.name
        }
      ).
      then(function(result) {
        var data = {
          email: result.email,
          gitUrl: req.body.repository.clone_url,
          docTheme: "",
        }
        generator.executeScript({}, data, function(err, generatedData) {
            deploy.deployPages({}, {
              email: result.email,
              gitURL: req.body.repository.clone_url,
              username: result.username,
              uniqueId: generatedData.uniqueId,
              encryptedToken: result.accessToken
            })
        })
      })
      res.json({
        status: true
      })
   }
})

After you register on webhook, GitHub will send a request to the url which we registered on the repository. In our case “https:/yaydoc.herokuapp.com/ci/auth” is the url. The type of the event can be known by reading ‘X-GitHub-Event’ header. Right now I’m registering only for the push event. So we’ll only be getting the push event. GitHub also gives us the repository details in the request body.

When the user makes a commit to the repository, GitHub will send a POST request to the Yaydoc’s server. Then, we’ll get the repository name and Github’s user ID from the request body. By use of this, I’ll retrieve the access token from the database which we already registered while the user registers the repository to the CI. The documentation will be generated using generate script and pushed to GitHub pages using deploy script.

Now Yaydoc generates documentation on every push when the user commits to the repository and also it will enable us to integrate new features in our own custom environment. We also plan to build a full featured CI platform.

Resources:

Continue ReadingContinuous Integration in Yaydoc using GitHub webhook API

Resource Injection Using ButterKnife in Loklak Wok Android

Loklak Wok Android being a sophisticated Android app uses a lot of views, and of those most are manipulated at runtime. In Android to play with a View or ViewGroup defined in XML at runtime requires developers to add the following line:

(TypeOfView) parentView.findViewById(R.id.id_of_view);

 

This leads to lengthy code. And very often, more than one Views respond to a particular event. For example, hiding Views if a network request fails and showing a message to the user to “Try Again!”. Let’s say you have to hide 4 Views, are you going to do the following:

view1.setVisibility(View.GONE);
view2.setVisibility(View.GONE);
view3.setVisibility(View.GONE);
view4.setVisibility(View.GONE);
textView.setVisibility(View.VISIBLE); // has "Try Again!" message.

// more 5 lines of code when hiding textView and displaying 4 other Views

 

Surely not! And the old fashioned way to get a string value defined as a resource in string.xml

String appName = getActivity().getResources().getString(R.id.app_name);

 

Surely, all this works good, but being a developer while working on a sophisticated app you would like to focus on the logic of the app, rather than scratching your head to debug whether you properly did a findViewById or not, did you typecast it to the proper View, or where did you miss to change the visibility of a view in response to an event.

Well, all of this can be easily handled by using a library which provides you the dependency, here resources. All you need to do is just declare your resources, and that’s it, the library provides the resources to you, yes you don’t need to initialize it using findViewById. So let’s dive in and see how ButterKnife is used in Loklak Wok Android to handle these issues.

Adding ButterKnife to Android Project

In the app/build.gradle:

dependencies {
    compile 'com.jakewharton:butterknife:8.6.0'
    annotationProcessor 'com.jakewharton:butterknife-compiler:8.6.0'
    ...
}

Dealing with Views in Fragments

When views are declared, BindView annotation is used with its parameter as the ID of the view, for example, views in TweetHarvestingFragment :

@BindView(R.id.toolbar)
Toolbar toolbar;
@BindView(R.id.harvested_tweets_count)
TextView harvestedTweetsCountTextView;
@BindView(R.id.harvested_tweets_container)
RecyclerView recyclerView;
@BindView(R.id.network_error)
TextView networkErrorTextView;

NOTE: Views declared can’t be private.

Once Views are declared, then it needs to be injected, it is done using ButterKnife.bind(Object target, View Source). Here in TweetHarvestingFragment the target will be the fragment itself and source i.e. the parent view will be rootView (obtained by inflating the layout file of fragment). All this needs to be done in onCreateView method

View rootView = inflater.inflate(R.layout.fragment_tweet_harvesting, container, false);
ButterKnife.bind(this, rootView);

That’s it, we are done!

The same paradigm can be used to bind views to a ViewHolder of a RecyclerView, as implemented in HarvestTweetViewHolder:

@BindView(R.id.user_fullname)
TextView userFullname;
@BindView(R.id.username)
TextView username;
@BindView(R.id.tweet_date)
TextView tweetDate;
@BindView(R.id.harvested_tweet_text)
TextView harvestedTweetTextView;

public HarvestedTweetViewHolder(View itemView) {
   super(itemView);
   ButterKnife.bind(this, itemView);
}

 

Injecting resources like strings, dimensions, colors, drawables etc. is even easier, only the related annotation and ID needs to be provided. Example the string app_name is used in TweetHarvestingFragment to display the app name i.e. “Loklak Wok” in toolbar

@BindString(R.string.app_name)
String appName;

// directly used inside onCreateView to set the title in toolbar
toolbar.setTitlet(appName);

 

Using ButterKnife onClickListeners can be implemented in separate method, a clean way to define click events rather than polluting onCreate(in Activity) or onCreateView(in Fragment), as implemented in SuggestFragment

@OnClick(R.id.clear_image_button)
public void onClickedClearImageButton(View view) {
   tweetSearchEditText.setText("");
}

 

Multiple Views responding to a single Event

Using @BindViews annotation a list of multiple views can be created, and then a common ButterKnife.Action can be defined to act on the list of views. In TweetHarvestingFragment visibility of some views are changed if a network request fails or succeeds using this feature of ButterKnife:

// declaring List of Views
@BindViews({
       R.id.harvested_tweets_count,
       R.id.harvested__tweet_count_message,
       R.id.harvested_tweets_container})
List<View> networkViews;

// defining action for views
private final ButterKnife.Action<View> VISIBLE = (view, index) -> view.setVisibility(View.VISIBLE);
private final ButterKnife.Action<View> GONE = (view, index) -> view.setVisibility(View.GONE);

// applying action on the views
ButterKnife.apply(networkViews, VISIBLE);
ButterKnife.apply(networkViews, GONE);

Resources

Continue ReadingResource Injection Using ButterKnife in Loklak Wok Android

Configurable Services in Loklak Search

Loklak search being an angular application has a concept of wiring down the code in the special form of classes called Services. These serviced have important characteristics, which make them a powerful feature of angular.

  • Services are shared common object wired together by Dependency Injection.
  • Services are lazily instantiated at the runtime.

 

The DI and the instantiation part of a service are handled by angular itself so we don’t have to bother about it. The parts of the services we are always concerned about is the logical part of the service. As the services are the sharable code at the time of writing a service we have to be 100% sure that this is the part of the code which we want to share with our components, else this can lead to the bad implementation of architecture which makes application harder to debug.

Now, the next question which arises is how services are different from something like redux state? Well, the difference lies in the word itself, services don’t have a persistent state of themselves. They are just a set of methods to separate a common piece of code from all the components into one class. These services have functions which take an input, processes them and spit an output.

Services in Loklak Search

So in loklak search, the main services are the ones which on request, fetch data from the backend API and return the data to the requester. All the services in loklak search have a fixed well-defined task, i.e. to use the API and get the data. This is how all the services must be designed, with a specific set of goals. A service should never try to do what is not necessary, or in other words, each service should have one and only one aim and it should do it nicely.

In loklak search, the services are classified by the API endpoints they hit to retrieve data. These services receive the query to be searched from the requested and they send the AJAX request to correct API endpoint and return the fetched data. This is the common structure of all the Loklak services, they all have a fetchQuery() method which takes a string argument query and requests the API for that query and after completion, it either returns the correct response from the API or throws an error if something goes wrong.

@Injectable()
class SearchService() {
public fetchQuery( query: string ) {  }
private extractData( response ) {  }
private handleError( error ) {  }
}

Problems faced in this structure

This simple structure was good enough for the application in the basic levels, but as the number of features in the application increase, our simple service becomes less and less flexible as the fetchQuery() method takes only a query string as an argument and requests the API for that query, along with some query parameters. These query parameters are the additional information given to the server to process and respond to our request in a particular way, like a number of results to be fetched, aggregations to be carried out, and much more. In the current implementation, the setting up these parameters were solely done by the service itself, so these parameters were fixed inside the service and there was no easy way to modify them. This reduced the flexibility of the service as all the requesters were bound to a fixed set of parameters, thus lacking the usability of the service in other places of the application.

 

Solution – Service Configs

The solution to this problem of service customizability is the Service Config classes. The objects of these classes contain the information about the query parameters which various requesters can configure according to their specific needs, and our services will simply configure the query params accordingly. This idea of having a shared structure for the service configurations plays very nicely with our scenario where we want extra control over the parameters which our service is configuring.

@Injectable()
class SearchService() {
public fetchQuery( query: string, config: SearchServiceConfig ) {  }
private extractData( response ) {  }
private handleError( error ) {  }
}

This small modification to our service structure enables us to have the amount of control which we required. The config class is a fairly simple one.

export class SearchServiceConfig {
private count: number;
private source: Source;
private fields: Set<AggregationFields>;
private aggregationLimit: number;
private maximumRecords: number;
private startRecord: number;
private timezoneOffset: string;
private filters: Set<Filter>;

// Other methods to get/set these attributes
}

Now any requester will instantiate a new object of this class and will set the attributes according to his needs then this object is passed to the fetchQuery() method of our function. Which designs the request to be sent accordingly.

Conclusion

In conclusion, i would like to mention the how these attributes are chosen to be a part of the Config and not as a query string. Our API endpoints accept the query string along with some attributes which filter out the results or run aggregations in various fields. So we should have all these attributes in our config as these all properties may vary according to the requesters need. Therefore, this idea of configurable services makes us not only better reuse the existing models and services in multiple situations but also make us write better predictable code.

Resources and Links

Continue ReadingConfigurable Services in Loklak Search