Implementing Custom Date and Time Picker with 2-way Data Binding Support

The Data binding library is one of the most popular libraries among the android developers. We have been using it in the Open Event Organiser Android app for building interactive UI’s for some time now. The Open Event Organiser Android App is the Event management app for organizers using the Open Event Platform. This blog explains how we implemented our own custom Date and Time picker with 2-way data binding support using the Data binding framework.

Why custom picker ?

One specific requirement in the app is to have a button, clicking on that button should open a DatePicker which would allow the user to select the date. A similar behaviour was required to allow the user to select the time as well. In order to handle this requirement we were using Binding Adapters on Button. For eg. the following Binding Adapter allowed us to define a property date on a button and set an Observable String as it’s value. We implemented a similar Binding Adapter for selecting time as well.

@BindingAdapter("date")
public static void bindDate(Button button, ObservableField<String> date) {
    String format = DateUtils.FORMAT_DATE_COMPLETE;

    bindTemporal(button, date, format, zonedDateTime ->
        new DatePickerDialog(button.getContext(), (picker, year, month, dayOfMonth) ->
                setPickedDate(
                    LocalDateTime.of(LocalDate.of(year, month + 1, dayOfMonth), zonedDateTime.toLocalTime()),
                    button, format, date),
                zonedDateTime.getYear(), zonedDateTime.getMonthValue() - 1, zonedDateTime.getDayOfMonth()));
  }

It calls the bindTemporal method which takes in a function along with the button, date and the format and does two things. First, it sets the value of the date as the text of the button. Secondly, it attaches a click listener to the button and applies the function passed in as the argument when clicked. Below is the bindTemporal method for reference:

private static void bindTemporal(Button button, ObservableField<String> date, String format, Function<ZonedDateTime, AlertDialog> dialogProvider) {
        if (date == null)
            return;

        String isoDate = date.get();
        button.setText(DateUtils.formatDateWithDefault(format, isoDate));

        button.setOnClickListener(view -> {
            ZonedDateTime zonedDateTime = ZonedDateTime.now();
            try {
                zonedDateTime = DateUtils.getDate(isoDate);
            } catch (DateTimeParseException pe) {
                Timber.e(pe);
            }
            dialogProvider.apply(zonedDateTime).show();
        });
    }

It was working pretty well until recently when we started getting deprecation warnings about using Observable fields as a parameter of Binding Adapter. Below is the full warning:

Warning:Use of ObservableField and primitive cousins directly as method parameters is deprecated and support will be removed soon. Use the contents as parameters instead in method org.fossasia.openevent.app.common.app.binding.DateBindings.bindDate

The only possible way that we could think of was to pass in regular String in place of Observable String. Now if we pass in a regular String object then the application won’t be reactive. Hence we decided to implement our own custom view to resolve this problem.

Custom Date and Time Picker

We decided to create an Abstract DateTimePicker class which will hold all the common code of our custom Date and Time pickers. It is highly recommended that you go through this awesome blog post first before reading any further. We won’t be going through the details already explained in the post.

Following are the important features of this Abstract class:

  1. It extends the AppCompatButton class.
  2. It stores an ObservableString named value and an OnDateTimeChangedListener as it’s field. We will discuss the change listener later in the article.
  3. It implements the three mandatory constructors and calls it’s super method. It also calls the init method which sets the current date and time as the default.
  4. It has a bindTemporal method which is the same as we discussed earlier.
  5. It has a setPickedDate method which sets the selected date/time as the text for the button so that users can see the selected date/time on the button itself. Moreover it notifies the change listener about the change in date if attached.
  6. It has an abstract method called setValue. It will be implemented in the sub classes and used to set the date or time value for the field named value.

You can check the full implementation here.

The OnDateTimeChangedListener which we mentioned above is an extremely simple interface. It defines a simple method onDateChanged which takes in the selected date as the argument.

public interface OnDateTimeChangedListener {
    void onDateChanged(ObservableString newDate);
}

Let’s have a look at the implementation of the DatePicker class. The key features of this class are:

  1. It extends the AbstractDateTimePicker class and implements the necessary constructors calling the corresponding super constructor.
  2. It implements the method setValue which sets the date or time passed in to the field value. It also calls the bindTemporal method of the super class.

public class DatePicker extends AbstractDateTimePicker {
    public DatePicker(Context context) {
        super(context);
    }

    public DatePicker(Context context, AttributeSet attrs) {
        super(context, attrs);
    }

    public DatePicker(Context context, AttributeSet attrs, int defStyle) {
        super(context, attrs, defStyle);
    }

  public void setValue(String value) {
        ObservableString observableValue = getValue();
        if (observableValue.get() == null || !TextUtils.equals(observableValue.get(), value)) {
            observableValue.set(value);
            String format = DateUtils.FORMAT_DATE_COMPLETE;

            bindTemporal(value, format, zonedDateTime ->
                new DatePickerDialog(this.getContext(), (picker, year, month, dayOfMonth) ->
                    setPickedDate(
                        LocalDateTime.of(LocalDate.of(year, month + 1, dayOfMonth), zonedDateTime.toLocalTime()), format),
                    zonedDateTime.getYear(), zonedDateTime.getMonthValue() - 1, zonedDateTime.getDayOfMonth()));
        }
    }
}

Next we discuss the BindingAdapter and the InverseBindingAdapter for the custom DatePicker which allows the data binding framework to set the action to be performed when date changes and get the date from the view respectively.

@BindingAdapter(value = "valueAttrChanged")
public static void setDateChangeListener(DatePicker datePicker, final InverseBindingListener listener) {
        if (listener != null) {
            datePicker.setOnDateChangedListener(newDate -> listener.onChange());
        }
    }

@InverseBindingAdapter(attribute = "value")
public static String getRealValue(DatePicker datePicker) {
    return datePicker.getValue().get();
}

Now in order to use our view, we can simply define it in the layout file as shown below:

<org.fossasia.openevent.app.ui.views.DatePicker
                    style="?attr/borderlessButtonStyle"
                    android:layout_width="wrap_content"
                    android:layout_height="wrap_content"
                    android:textColor="@color/purple_500"
                    app:value="@={ date }" />

The key thing to notice is the use of @= instead of @ which denotes two way data binding.    

Conclusion

The Android Data binding framework is extremely powerful and flexible at the same time. We can use it for our custom requirements as shown in this article.

References

 

 

 

 

 

Continue ReadingImplementing Custom Date and Time Picker with 2-way Data Binding Support

Implementing YouTube Search API with WebClient

SUSI.AI is an assistant which enables us to create a lot of skills. These skills offer various functionalities which performs different actions. Therefore SUSI has implemented various action types. Some of these are:

  • Answer
  • Table
  • Maps
  • Rss
  • audio_play
  • video_play

When a user answers a query the server sends response in form of actions type. The client then scans the response JSON object and checks the type of action and performs the desired operations. This action types are verified by

Continue ReadingImplementing YouTube Search API with WebClient

Setting up the Circle.CI config for SUSI Android

SUSI.AI Android app uses CircleCI to check for tests whenever a new PR is made. This is done to ensure that the app is consistent with the new changes and there is no problem adding that particular code change. CircleCI has now launched a v2 version of the .yml file and therefore to continue using CircleCI it is time to upgrade to v2 version of the script.

Circle.CI config version 1

Config file tells the CI on what commands to run to test the build, which will determine if the build is successful or failed like tests, lints etc and hooks commands to run if the tests pass, which environment to run the code in – python, java, android, etc.

Circle.CI published that 31, August 2018 is the last date upto when they will support the config version 1.0, therefore it Circle.CI was updated to version 2.0.

The version 2.0 has an updated syntax of the script and previous script was modified so that it could provide the configuration for the CI builds.

The updated script is shown below :

version: 2
jobs:
build:
  working_directory: ~/code
  docker:
    – image: circleci/android:api-27-alpha
  environment:
    JVM_OPTS: -Xmx3200m
  steps:
    – checkout
    – restore_cache:
        key: jars-{{ checksum “build.gradle” }}-{{ checksum  “app/build.gradle” }}
    – run:
        name: Download Dependencies
        command: ./gradlew androidDependencies
    – save_cache:
        paths:
          – ~/.gradle
        key: jars-{{ checksum “build.gradle” }}-{{ checksum  “app/build.gradle” }}
    – run:
        name: lint check
        command: ./gradlew lint
    – run:
        name: Run Tests
        command: ./gradlew build
    – run:
        command:
          bash exec/prep-key.sh
          bash exec/apk-gen.sh
    – store_artifacts:
        path: app/build/reports
        destination: reports
    – store_artifacts:
        path: app/build/outputs
        destination: outputs
    – store_test_results:
        path: app/build/test-results

Few checks such as the connectedCheck, which were required for the UI testing, are not included in this script and instead an approach towards the increasing number of unit tests is followed. The reason being, implementing UI tests is a hard task for apps which have no constant designs and have constantly changing specifications. Since the architecture used is MVP, so on moving all logic to presenter components there won’t be a need of most UI tests at all. This problem with UI tests will further increase in future. Therefore, moving in the direction of increasing unit tests is better because they are both easy, small and quick to code and run and the degree of dependence on flaky UI tests also decreases.

If the view is implementing small sections of display logic, just verifying that the presenter calls those is enough to guarantee that the UI is going to work.

The command in the previous config such as

./gradlew connectedCheck

was removed and instead ./gradlew build was added to start the unit tests instead, also due to the updated gradle dependencies, changes were made to the apk uploading commands and also changes were performed in the apk-gen.sh file.

bash exec/prep-key.sh
chmod +x exec/apk-gen.sh
./exec/apk-gen.sh

In the above code, lines concerned with apk-gen.sh can be combined and the resultant command was written as :

bash exec/apk-gen.sh

The apk-gen.sh script was configured and the latest build tools were added to make the script work.

REFERENCES

Continuous Integration – Martin Fowler:

https://martinfowler.com/articles/continuousIntegration.html

Circle CI version 2.0 for android :

https://circleci.com/docs/2.0/configuration-reference/#run

Circle CI configuration reference :

https://circleci.com/docs/2.0/configuration-reference/

 

Continue ReadingSetting up the Circle.CI config for SUSI Android

Working with Logic Analyzer in PSLab Android app

This blog demonstrates the working of Logic Analyzer instrument available in PSLab Android app. It also includes a detailed description of the features available in the Logic Analyzer instrument along with a step by step guide on how to work with it which will be beneficial to first-time users of the PSLab application.

The functionality of the Logic Analyzer available in PSLab Android app is same as that in PSLab Desktop App. So, it would be easy for a user of PSLab Desktop Application to get acquainted with this Logic Analyzer. The only difference in this instrument is the changed and attractive UI which makes working with it very easy.

Why use Logic Analyzer?

The Logic Analyzer instrument provides the functionality of capturing and plotting the digital waves on the screen so that it would be easy for a user to determine the time relationship between different waves. So, this instrument would be very useful while working with timing diagrams, protocol decodes, state machines traces, assembly language, or with source-level software.

How to generate different digital pulses in the PSLab app?

Logic Analyzer needs to be provided with some input of digital pulses among whom time relationship is to be found out. Digital pulses generated from different systems can be directly provided as input to the Logic Analyzer for analyzing. But PSLab provides a functionality to generate digital pulses up to some constrained frequency.

Following are the steps to generate different digital waves in PSLab Android application :

  • Open PSLab Android application and click on the Wave Generator tile as shown in figure 1. After opening the instrument, the screen will look as shown in figure 2.

Figure 1. Wave Generator instrument tile available in PSLab Android app

Figure 2. The main screen of the Wave Generator instrument

  • Click on the MODE button to change the mode to PWM. The screen will look as shown in figure 3.

Figure 3. PWM mode in Wave Generator

  • PSLab device provides generation of maximum four digital waves at once. In this example, I will proceed by utilizing only two pins i.e. SQR1and SQR2 (where SQR = Acronym of square wave generator and the number next to it is the pin ID available on the PSLab device) to demonstrate the working of Two Channel Mode in Logic Analyzer. Set the duty cycles and frequency for the selected pins as desired (try to keep all the duty cycles different from each other to understand the process of measurement easily).

NOTE: User can also set phase angle for different waves but I will proceed with defaults.

How to analyze the generated waves in Logic Analyzer?

  • Now go back and select the Logic Analyzer tile as shown in figure 4 from the list of available instruments. A screen as shown in figure 5 should open.

Figure 4. Tile of Logic Analyzer instrument available in the PSLab app

Figure 5. The main screen of the Logic Analyzer instrument

On the right-hand side, you can see a slider whose initial value is SELECT which shows the information on how to use the slider below it. Below the Channel Selection area is the Analyze button used to fetch and plot the data which is generated or provided to the respective Logic Analyzer pins i.e. ID1, ID2, ID3, and ID4.

The blank area on the left is where the graph will be plotted after fetching data points. Below it is the Axis Indicator used to indicate the position of the highlighter so that time measurement can be done easily. To the right of the Axis Indicator is a small light which indicates the status of the device. It turns GREEN if the device is connected else it remains in RED.

  • In this blog, I will demonstrate the Two Channel Mode. But all the other available modes need the same implementation by only varying the number of pins in use. So, slide to 2 in the Carousel View and a screen as shown in figure 6 will pop up.

Figure 6. Two Channel Mode in Logic Analyzer

  • Connect the wires on the PSLab device as shown in Figure 7.

Figure 7. Connecting wires on PSLab

NOTE: The Logic Analyzer pins used in this demonstration are ID1 and ID2. But any IDx pin can be chosen for analysis. But try to maintain the selected choice throughout the implementation.

  • Now from the Channel Selection area, select the channel for the first card to  ID1 (default) and that for the second card to ID2. For Edges Selection, maintain the defaults.

NOTE: There are several options available for plotting the digital waves besides the default selected i.e.

  1. Every Edge – Plot every edge of the signal
  2. Falling Edges – Plot only falling edges of the signal (When a signal comes from 1 to 0 state)
  3. Rising Edges Only – Plot only rising edges of the signal (When a signal comes from 0 to 1 state)
  4. Disabled – Don’t plot the selected wave
  • After Channel Selection, press the Analyze button to plot the data. On pressing the Analyze button, a circular loading sign will appear showing that the data is being fetched and converted to data that can be plotted. As soon as the data is ready to be plotted, the loading sign vanishes and the graph appears as shown in figure 8.

Figure 8. GIF showing the loading and analyzing processes

  • The time relationship between the plotted data can be found out by clicking over the rising/falling edges and noting the time shown in the Axis Indicator as shown in figure 8.
  • An example of Edge Selection is shown in figure 9.

Figure 9. Example of Edge Selection option

Here, EVERY FALLING EDGE option is selected for the ID1 channel and EVERY RISING EDGE option is selected for the ID2 channel.

So in this way, the Logic Analyzer instrument available in PSLab Android application can be used to ease out the process of calculating the time interval between different edges for different/same digital pulse/s.

Resources

  1. PSLab Android Application – https://github.com/fossasia/pslab-android (Link to repo)
  2. PSLab device pins sticker – https://github.com/fossasia/pslab-artwork/blob/master/Sticker/pslabdesign.png
Continue ReadingWorking with Logic Analyzer in PSLab Android app

Implementing the discrete Seekbar for Wave Generator

The Wave Generator instrument in PSLab Android app allows us to produce waveforms having different values of properties like frequency, duty, phase etc.

The range of these properties allowed by PSLab Device are :

Table showing the range of properties that can be set for waves by PSLab device
Wave Property Range
Min Max Step Size
Frequency 10 Hz 5000 Hz 1 Hz
Phase 360°
Duty 10% 100% 10%

We can set these values using the up/down arrow buttons provided by the wave generator but the problem is that the range of values is very high and least counts are small so it is convenient to set the values using only the up and down arrow buttons.

Therefore we need something that could allow us to directly set any value of our choice while keeping the UI interactive.

The solution to this problem – “Discrete Seekbar”. It contains a slider having points at equal intervals and whose length represents the range of the values and a head that slides over the slider and is used to select a specific value from a range of values.

I have included the discrete Seekbar in Wave Generator by using a third-party library if you want to add Seekbar directly you can do that by directly using the default Seekbar widget provided by Android SDK and setting the following attribute in as shown below.

android:theme = “@style/Widget.AppCompat.SeekBar.Discrete”

Refer to this post[2] for implementing Seekbar directly without an external library.

The reason I chose this library is that:-

  • It offers various implementation of different types of Seekbar like discrete and continuous.
  • Implementation of Seekbar is simpler and it offers various customizations like thumb color, track color, tick text etc.  

In following steps I will implement the discrete Seekbar:

Step 1 Adding the dependency

For this project, I will be using an external library “IndicatorSeekbarLibrary” by Warkiz[1], for adding the dependency we need to include the following code in our build.gradle file.

dependencies{
implementation 'com.github.warkiz.widget:indicatorseekbar:2.0.9'
}

Step 2 Including the Seekbar in layout

For this step, we need to add the Seekbar widget using <com.warkiz.widget.IndicatorSeekBar> XML tag in our wave generator layout file to include the Seekbar in our layout as shown in the code below:

<com.warkiz.widget.IndicatorSeekBar
    android:layout_width="match_parent"
    android:layout_height="wrap_content"
    app:isb_max="5000"
    app:isb_min="0"
    app:isb_ticks_count="5"
    app:isb_thumb_color="@color/color_green"
    app:isb_thumb_size="20dp"
    app:isb_track_background_color="@color/color_gray"
    app:isb_track_background_size="2dp"
    app:isb_track_progress_color="@color/color_blue"
    app:isb_track_progress_size="4dp" />

Some important attributes used above:

app:isb_max : defines the max value that can be achieved by the Seekbar.

app:isb_min :  defines the min value that can be achieved by the Seekbar

app:isb_ticks_count: no. of ticks(interval) that has to be shown on the slider

We can see different components of Seekbar like track, indicator, thumb, tick of SeekBar in the following diagram[2].

Figure 1 depicts the different attributes of the slider
(Source – https://github.com/warkiz/IndicatorSeekBar/blob/master/README.md)

Step 3 Attaching the listener to the Seekbar in Java file

In this step we need to attach the listener to the Seekbar to record changes in the Seekbar made by the user, for this we will create a new listener with the help of onSeekBarChangeListener interface and attach it with the Seekbar as shown in following code

IndicatorSeekBar seekbar = (IndicatorSeekBar) findViewbyId(R.id.seekbar);

seekBar.setOnSeekChangeListener(new OnSeekChangeListener() {
            @Override
            public void onSeeking(SeekParams seekParams) {
                /* called when the user is sliding the thumb */
            }

            @Override
            public void onStartTrackingTouch(IndicatorSeekBar seekBar) {
                /* called when the sliding of thumb is started */
            }

            @Override
            public void onStopTrackingTouch(IndicatorSeekBar seekBar) {
                /* called when the sliding of thumb stops */
            }
        });

After following all the above steps, I  implemented the Seekbar shown in Figure 2 below in my wave generator and now it becomes really easy to set different values of properties for without having to continually press the up/down button.

Figure 2 shows the Seekbar included in wave generator beside up/down arrow button

Resources

  1. warkiz/IndicatorSeekBar library  – Github Repo of the Indicator SeekBar library
  2. http://nileshsenta.blogspot.com/2016/10/discrete-seekbar-without-third-party.html – Blog by Nilesh Shenta on how to implement discrete without third party library

 

Continue ReadingImplementing the discrete Seekbar for Wave Generator

Adding product flavors to SUSI Android app

Product flavors are required by production level apps to provide different versions of their app available to people. Each of the flavor has separate functionality depending on the type of flavors chosen. Most common flavors are full and demo flavors, from their name only we can understand that these flavors contain full version and demo version of their apps. Since SUSI.AI android app has to be released on FDroid several libraries that were present in the current app were not  acceptable by them as it only allows apps that completely use only open source software to develop and therefore there was a need to make the app compatible with both the app stores (Play Store and FDroid) product flavors were added in the app.

Proprietary software in the app

There are several libraries which are not acceptable on FDroid but are present in the app, a couple of them being :

  1. Crashlytics using fabric
  2. LinkPreview library

These two libraries were found which are not acceptable in FDroid. Although the link preview library was not proprietary software, but fdroid has guidelines which only allows that recognised and trusted libraries from maven repository is accepted.

Further, youtube api recently integrated in the app is also not acceptable by FDroid as it is under google and categorised as a proprietary software.

Adding product flavors

The flavor names chosen to be made were “fdroid” and “playStore”, fdroid here specifying the build variant for the app that would contain the libraries and code acceptable by fdroid and playStore flavor to contain the sources and libraries for the regular version of the app which can contain proprietary software.

The flavors were added as shown below in the module level build.gradle file :

flavorDimensions “default”

productFlavors {
  fdroid {
      dimension “default”
      applicationIdSuffix “.fdroid”
      versionNameSuffix “-FDroid”
  }

  playStore {
      dimension “default”
      versionNameSuffix “-PlayStore”
  }
}

the flavors are added under the block of code by the name productFlavors, and each flavor needs to have some properties to be set. The applicationId of the app can be changed or can be kept same, in our case for the fdroid flavor the applicationIdSuffix was added which adds “.fdroid” at the end of the applicationId for the fdroid version.

versionNameSuffix property was set in both the flavors which adds “-FDroid” to end of the versionname for the fdroid variant and “-PlayStore” to the end of the versionname for the playstore variant.

Using the product flavors we can generate multiple apks , as in our case after the build flavors were added there were four apks built. These can be seen as  :

Choosing any of these apks we can select the variant that is to be worked upon. The debug apks here are those used for developing purposes.

Conclusion

Adding flavors allows SUSI.AI android app to keep separate code and resources for different variants. Any code that is different to both the variants is added separately after creating the directories named “fdroid” and “playStore” in the src folder.

This was added after switching to the Project view in Android Studio and then moving to the app folder and then inside the src folder make a directories named fdroid and playStore.

References

  1. Configure Build Variants: https://developer.android.com/studio/build/
  2. Android Product Flavors: https://medium.com/@iammert/android-product-flavors-1ef276b2bbc1

Handling multiple java sources and flavors using flavors on Gradle: https://medium.com/@thiagolopessilva/the-handling-multiple-java-source-and-resources-using-flavors-on-gradle-18a4b581285b

Continue ReadingAdding product flavors to SUSI Android app

Adding News Tab in Susper

Most of the current search engines have a News tab where results are displayed which are fetched from different News Organisations. The latest results are displayed on top followed by the older ones. Current market leader, Google even uses AI and Machine Learning to analyze the contents before delivering it as results to the user. In this blog, I will describe how I have implemented a basic News Tab in Susper which show news results from dailymail.co.uk . This News tab can be improved in future to match the market leader.

Implementation of News tab in Susper:

Implementation of News tab in UI:

To implement News tab the current design pattern has been followed in Susper. Here is the code to show News tab on the results page.

<ul type="none" id="search-options">
       <li [class.active_view]="Display('news')" (click)="newsClick()">News</li>

Angular attribute binding is used to bind the active_view property with the “Display(‘news’)” function and the click event is associated with “newsClick()” function.

Here is the code for newsClick() function which filters the result using ‘site:’ parameter of yacy and then dispatches the ‘urldata’ object to ‘QueryServerAction(urldata)’ function in Query Action.

newsClick() {
   let urldata = Object.assign({}, this.searchdata);
   this.getPresentPage(1);
   this.resultDisplay = 'news';
   delete urldata.fq;
   urldata.rows = 10;
   if (urldata.query.substr(urldata.query.length - 25, 25) !== " site:www.dailymail.co.uk") {
     urldata.query += " site:www.dailymail.co.uk";
   } else {
     urldata.query += "";
   }
   urldata.resultDisplay = this.resultDisplay;
   this.store.dispatch(new queryactions.QueryServerAction(urldata));
   }

Here is the code for Query Action in query.ts file. It has two actions QueryAction and QueryServerAction

export const ActionTypes = {
 QUERYCHANGE: type('[Query] Change'),
 QUERYSERVER: type('[Query] Server'),
};
export class QueryAction implements Action {
 type = ActionTypes.QUERYCHANGE;

 constructor(public payload: any) {}
}
export class QueryServerAction implements Action {
 type = ActionTypes.QUERYSERVER;
 constructor(public payload: any) {}
}
export type Actions
 = QueryAction|QueryServerAction ;

Now the reducer takes either of the two query actions and the current state and returns the new state to the store.

Here is the code for the reducer query.ts

export const CHANGE = 'CHANGE';
export interface State {
 query: string;
 wholequery: any;
}
const initialState: State = {
 query: '',
 wholequery: {
   query: '',
   rows: 10,
   start: 0,
   mode: 'text'
 },
};
export function reducer(state: State = initialState, action: query.Actions): State {
 switch (action.type) {
   case query.ActionTypes.QUERYCHANGE: {
     const changeQuery = action.payload;
     return Object.assign({}, state, {
       query: changeQuery,
       wholequery: state.wholequery
     });
   }
   case query.ActionTypes.QUERYSERVER: {
     let serverQuery = Object.assign({}, action.payload);
     let resultCount = 10;
     if (localStorage.getItem('resultscount')) {
       resultCount = JSON.parse(localStorage.getItem('resultscount')).value || 10;
     }
     let instantsearch = JSON.parse(localStorage.getItem('instantsearch'));

     if (instantsearch && instantsearch.value) {
       resultCount = 10;
     }

     serverQuery.rows = resultCount;
     return Object.assign({}, state, {
       wholequery: serverQuery,
       query: state.query
     });
   }
   default: {
     return state;
   }
 }
}
export const getpresentquery = (state: State) => state.query;
export const getpresentwholequery = (state: State) => state.wholequery;

 

From the store the modified query is available to all other components of the appand is used by search service to get the filtered results.

The search results are then stored in observable items$ in results.components.ts and is then used in results.components.html to display results under News tab.

Here is the code to display the results.

 <div class="feed container">
     <div *ngFor="let item of items$|async" class="result">
       <div class="title">
         <a class="title-pointer" href="{{item.link}}">{{item.title}}</a>
       </div>
       <div class="link">
         <p>{{item.link}}</p>
       </div>
     </div>
   </div>

 

Resources

1.YaCy Search Parameters: http://www.yacywebsearch.net/wiki/index.php/En:SearchParameters

2.Redux in Angular: http://blog.ng-book.com/introduction-to-redux-with-typescript-and-angular-2/

3.Corresponding PR: https://github.com/fossasia/susper.com/pull/1023

Continue ReadingAdding News Tab in Susper

Deploying Angular App Susper on GitHub Pages

What are the issues we had with automatic deployment of Susper?

SUSPER project which is built on Angular is kept on master branch of the repository .

Whenever any PR is merged,Travis CI does all the builds, and checks for any error. After successful checking it triggers deployment script in SUSPER (deploy.sh) .

Here is the deployment script:

#!/bin/bash

# SOURCE_BRANCH & TARGET_BRANCH stores the name of different susper.com github branches.
SOURCE_BRANCH="master"
TARGET_BRANCH="gh-pages"

# Pull requests and commits to other branches shouldn't try to deploy.
if [ "$TRAVIS_PULL_REQUEST" != "false" -o "$TRAVIS_BRANCH" != "$SOURCE_BRANCH" ]; then
    echo "Skipping deploy; The request or commit is not on master"
    exit 0
fi

# Store some useful information into variables
REPO=`git config remote.origin.url`
SSH_REPO=${REPO/https:\/\/github.com\//git@github.com:}
SHA=`git rev-parse --verify HEAD`

# Decryption of the `deploy.enc`
openssl aes-256-cbc -k "$super_secret_password" -in deploy.enc -out deploy_key -d

# give `deploy_key`. the permission to read and write
chmod 600 deploy_key
eval `ssh-agent -s`
ssh-add deploy_key

# Cloning the repository to repo/ directory,
# Creating gh-pages branch if it doesn't exists else moving to that branch
git clone $REPO repo
cd repo
git checkout $TARGET_BRANCH || git checkout --orphan $TARGET_BRANCH
cd ..

# Setting up the username and email.
git config user.name "Travis CI"
git config user.email "$COMMIT_AUTHOR_EMAIL"

# Cleaning up the old repo's gh-pages branch except CNAME file and 404.html
find repo/* ! -name "CNAME" ! -name "404.html" -maxdepth 1  -exec rm -rf {} \; 2> /dev/null
cd repo

# commit the changes, move to SOURCE_BRANCH
git add --all
git commit -m "Travis CI Clean Deploy : ${SHA}"

git checkout $SOURCE_BRANCH

# Actual building and setup of current push or PR. Move to `TARGET_BRANCH` and move the `dist` folder
npm install
ng build --prod --aot
mv susper.xml dist/
mv 404.html dist/

git checkout $TARGET_BRANCH
mv dist/* .

# Staging the new build for commit; and then committing the latest build
git add .
git commit --amend --no-edit --allow-empty

# Actual push to gh-pages branch via Travis
git push --force $SSH_REPO $TARGET_BRANCH

This script starts after successfully checking the project (comments shows working of each command). The repository is cleaned,except some files like CNAME and 404.html and a commit is made. Then SUSPER’s build artifacts are generated in dist folder and all the files from dist folder are moved to gh-pages and the changes are amended. GitHub Pages engine then uses the build artifacts to generate static site at susper.com. But we faced a weird problem when we were unable to see changes from latest commits on susper.com. Slowly number of commits increased but still changes were not reflecting on susper.com .

What was the error in deployment of SUSPER?

All the changes were getting committed to the gh-pages branch properly. But the GitHub Pages engine was unable to process the branch to build static site. Due to this the site was not getting updated. The failing builds from gh-engine are notified to the owner via email.

The failing builds from gh-pages can be seen here https://github.com/fossasia/susper.com/commits/gh-pages

Between 21 May to 18 March.

There was a weird error ( Failure: The tag `chartjs` in line 4 on `node_modules/chart.js/docs/charts/bar.md` is not recognized liquid tag) while building the site from build artifacts. There was an issue with `gh-pages` engine as `node_modules/chart.js/docs/charts/bar.md`  was previously also present but then there was no such errors than.

Due to this error all commits which were made to SUSPER repository was not shown on susper.com as the site was not getting updated.

How we solved it?

We tried to search a lot about the above mentioned error but we were unable to find a proper solution. While searching we came across this question https://stackoverflow.com/questions/24713112/my-github-page-wont-update-its-content

on StackOverflow. There was no accepted answer for this question and every answer was different from other. So, we just reproduced the bug in our own repository by just making a similar repository to SUSPER with same branches. Once we successfully reproduced the bug. We tried to correlate all the answers to arrive at a solution to fix this bug. Luckily we found that any commit with no significant change from previous commit removed the error and allowed the gh-pages engine to successfully create the site. So we made a dummy commit in which we created a hidden file with no content and pushed it to gh-pages. This did the trick for us and gh-pages was now able build the site for us.

The whole problem was due to a bug in GitHub pages engine which got fixed by a dummy commit.

Resources

  1. Stackoverflow:https://stackoverflow.com/questions/24713112/my-github-page-wont-update-its-content
  2. Publishing to GitHub Pages:https://help.github.com/articles/configuring-a-publishing-source-for-github-pages/
  3. Commits(gh-pages):https://github.com/fossasia/susper.com/commits/gh-pages
Continue ReadingDeploying Angular App Susper on GitHub Pages

Adding Yarn as new Dependency Manager  along with NPM in Susper

Dependency managers are software modules that coordinate the integration of external libraries or packages into larger application stack. Dependency managers use configuration files like composer.json, package.json, build.gradle or pom.xml to determine: What dependency to get, What version of the dependency in particular and, Which repository to get them from. Currently SUSPER has only NPM as a dependency manager which is used to install all dependencies. In this blog, I will describe how we have added facebook’s Yarn as a new dependency manager in Susper

Lets checkout Yarn in detail:

Yarn is a fast and good alternative to NPM. One of the great advantages of Yarn is that while remaining compatible with the npm registry, it replaces the workflow for npm client or other package managers Yarn was created by Facebook, to solve some particular problems that were faced while using NPM. Yarn was developed to deal with inconsistency in dependency installation while scaling and to increase speed.

What is advantages of using Yarn?

  • Improving Network performance:Queuing up the requests and avoiding requests waterfalls helps to maximize network utilization.
  • Checks Package Integrity:Package integrity is checked after each install to avoid corrupt packages installation.
  • Checks Package Integrity:Package integrity is checked after each install to avoid corrupt packages installation.
  • Caching: Yarn helps to install the dependencies without an internet connection if the dependency has been previously installed on the system. This is done by caching.
  • Lock File: Lock files are used to make sure that the node_modules directory has the exact same structure on all development environments.

Source: https://yarnpkg.com/en/

How Yarn is installed along with NPM in SUSPER?

Installing Yarn is super easy. Here are the steps to setup Yarn along with NPM and begin using it as dependency manager.

On Debian or Ubuntu Linux, we can install Yarn via our Debian package repository. We will first need to configure the repository:

curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee 

/etc/apt/sources.list.d/yarn.list

 

Then simply use:

sudo apt-get update && sudo apt-get install yarn

 

Note: Ubuntu 17.04 comes with cmdtest installed by default. If anyone gets any errors from installing yarn, then remove it by sudo apt remove cmdtest first. Refer to this for more information.

If using nvm you can avoid the node installation by doing:

sudo apt-get install --no-install-recommends yarn

 

Test that Yarn is installed by running:

yarn --version

 

Now delete the node_modules folder so that all dependencies installed by npm is removed.

Now use yarn command in project’s repository.

yarn 

 

Wait while dependencies are installed and then we will be done.

What is happening ?

Yarn has created a lock file  yarn.lock. After each operation the file is updated (installing, updating or removing packages) to keep the track of exact package version. If kept in our Git repository we can see that the exact same result in node_modules is made available to all systems.

Resources

  1. Yarn: https://yarnpkg.com/en/
  2. Announcement of Yarn: https://code.facebook.com/posts/1840075619545360
  3. Yarn Vs NPM: https://stackoverflow.com/questions/40027819/when-to-use-yarn-over-npm-what-are-the-differences
Continue ReadingAdding Yarn as new Dependency Manager  along with NPM in Susper

Make a cumulative API to return skills based on standard metrics in SUSI.AI

In this blog post, we are going to discuss on how to make a cumulative API which returns skills based on different standard metrics. It was implemented to combine various API calls, that were made from various clients, thereby reducing the number of API calls made. It lead to an optimization in the page load time of various clients and also helped to reduce the 503 errors that were received, due to very frequent API hits. The API endpoint for it is https://api.susi.ai/cms/getSkillMetricsData.json.

It accepts 5 optional parameters –

  • model – It represents the model for which the skill are fetched. The default value is set to General.
  • group – It represents the group(category) for which the skill are fetched. The default value is set to Knowledge.
  • language – It represents the language for which the skill are fetched. The default value is set to en.
  • duration – It represents the duration based on which skills are fetched by standard metrics like usage.
  • count – It represents the number of skills to be returned on a particular metric.

The minimalUserRole is set to ANONYMOUS for this API, as the data is required for home-page and doesn’t need the user to be logged-in.

Going through the API development

  • The parameters are first extracted via the call object that is passed to the main function. The  parameters are then stored in variables. If any parameter is absent, then it is set to the default value.
  • There is a check if the count param exists. If exists, then it is checked if it is of valid data-type. If no, an exception is thrown. And if count param doesn’t exist, it is set to default of 10.
  • This code snippet discusses the above two points –
@Override
public ServiceResponse serviceImpl(Query call, HttpServletResponse response, Authorization rights, final JsonObjectWithDefault permissions) throws APIException {

        String model_name = call.get("model", "general");
        File model = new File(DAO.model_watch_dir, model_name);
        String group_name = call.get("group", "All");
        String language_name = call.get("language", "en");
        int duration = call.get("duration", 0);
        JSONArray jsonArray = new JSONArray();
        JSONObject json = new JSONObject(true);
        JSONObject skillObject = new JSONObject();
        String countString = call.get("count", null);
        Integer count = null;

        if(countString != null) {
            if(Integer.parseInt(countString) < 0) {
                throw new APIException(422, "Invalid count value. It should be positive.");
            } else {
                try {
                    count = Integer.parseInt(countString);
                } catch(NumberFormatException ex) {
                    throw new APIException(422, "Invalid count value.");
                }
            }
        } else {
            count = 10;
        }
.
.
.

 

  • Then the skills are fetched based on the group name and then stored in a JSONArray named jsonArray. This array basically contains the metadata objects of each skill.
  • Now, we need to sort and filter these skills based on standard metrics like – Rating, Feedback Count, Usage Count, Latest skills.
  • The implementation for getting the skills based on the creation time is as follows. Rest, all the metrics were also implemented in the same fashion.

JSONObject skillMetrics = new JSONObject();
List<JSONObject> jsonValues = new ArrayList<JSONObject>();

// temporary list to extract objects from skillObject
for (int i = 0; i < jsonArray.length(); i++) {
    jsonValues.add(jsonArray.getJSONObject(i));
}

// Get skills based on creation date - Returns latest skills
Collections.sort(jsonValues, new Comparator<JSONObject>() {
    private static final String KEY_NAME = "creationTime";
    @Override
    public int compare(JSONObject a, JSONObject b) {
        String valA = new String();
        String valB = new String();
        int result = 0;

        try {
            valA = a.get(KEY_NAME).toString();
            valB = b.get(KEY_NAME).toString();
            result = valB.compareToIgnoreCase(valA);
        } catch (JSONException e) {
            e.printStackTrace();
        }
        return result;
    }
});

JSONArray creationDateData = getSlicedArray(jsonValues, count);
skillMetrics.put("latest", creationDateData);

.
.
.

 

  • The above code snippet deals with sorting the skills based on the creation time, that helps us to fetch the latest skills. The latest skills are stored on the skillMetrics object with the key name latest.
  • Similarly, the skills based on metrics like rating, feedback count and usage are stored with key names rating, feedback & usage respectively.

From the above snippet, we can also see a call to the function getSlicedArray. It takes a list of skills and count as input paramters. It returns the first ‘count’ skills from the list depending on the count value. The implementation of it is as follows –

private JSONArray getSlicedArray(List<JSONObject> jsonValues, Integer count)
{
    JSONArray slicedArray = new JSONArray();
    for (int i = 0; i < jsonValues.size(); i++) {
        if(count == 0) {
            break;
        } else {
            count --;
        }
        slicedArray.put(jsonValues.get(i));
    }
    return slicedArray;
}

 

  • The response object is then sent with 6 (six) key values mainly, apart from the session object. They are
    • accepted –  true – It tells that the skills have been fetched.
    • message – “Success: Fetched skill data based on metrics”
    • model –  It is the model that is sent on request params or the default value.
    • group –  It is the group that is sent on request params or the default value.
    • language – It is the language that is sent on request params or the default value.
    • metrics –  It is the main object of relevance for this API, which contains 4 child keys with values as an array of Skill Metadata objects.

{
  "accepted": true,
  "model": "general",
  "group": "All",
  "language": "en",
  "metrics": {
    "feedback": [
    {skill_1}, {skill_2}, ...
    ],
    "usage": [
    {skill_1}, {skill_2}, ...
    ],
    "rating": [
    {skill_1}, {skill_2}, ...
    ],
    "latest": [
    {skill_1}, {skill_2}, ...
    ]
  },
  "message": "Success: Fetched skill data based on metrics",
  "session": {
   ....
  }
}

 

I hope the development of creating the aforesaid API and the purpose of it is clear and proved to be helpful for your understanding.

References

Continue ReadingMake a cumulative API to return skills based on standard metrics in SUSI.AI