Using Flux to embed SUSI’s API Service in a Chat System.

To embed SUSI’s API Service in a chat-like system, I needed a view which could populate the content dynamically and maintain the state of the Application at the same time. Flux follows a unidirectional data flow path and I used this feature to the advantage of the Chat Application to maintain the real time state of the Chat View.

A Flowchart model of Flux looks like

flux flowchart

 

src: https://github.com/facebook/flux

Flux uses a dispatcher service to render its views, thus making the data flow in a unidirectional path. When a user reacts with a React view (here through the TextArea in the chat system), the view propagates an action through the dispatcher service, to the various stores that hold the application’s data and finally update the views that are affected. Here’s another flowchart model from the website which helps one understand the model in a better way.

flux flow

For the current Chat Application, I used a single Message Store which contained all the event listeners to detect any change in the view. For example, when I send a “Hey” to SUSI, an action is called to Dispatch this message to the Message Store with an ActionType  “CREATE_MESSAGE”. This store then renders the message in the Message Section View.

Here is an example snippet from the Actions.js file which performs an action of type CREATE_MESSAGE and dispatches the messages to the MessageStore.js.

export function createMessage(text, currentThreadID) {
let message = ChatMessageUtils.getCreatedMessageData(text, currentThreadID);
ChatAppDispatcher.dispatch({
type: ActionTypes.CREATE_MESSAGE,
message
});
ChatWebAPIUtils.createMessage(message);
};

The response from the message is generated as soon as another ActionType named

“CREATE_SUSI_MESSAGE” is dispatched to the store, thereby rendering the SUSI’s response generated in the view.

The file ChatConstants.js which declares all the ActionTypes.

import keyMirror from 'keymirror';

export default {

  ActionTypes: keyMirror({
    CREATE_MESSAGE: null,
    RECEIVE_RAW_CREATED_MESSAGE: null,
    CREATE_SUSI_MESSAGE: null,
    RECEIVE_SUSI_MESSAGE: null,
    RECEIVE_RAW_MESSAGES: null
  })

};

To get the message up on the view, I have used the following utils to call the API, render the messages to the view and call the different actions. Here’s a code snippet from ChatMessageUtils.js

export function createMessage(message) {
  ChatExampleDataServer.postMessage(message, createdMessage => {
    Actions.receiveCreatedMessage(createdMessage, message.id);
  });
  ChatExampleDataServer.postSUSIMessage(message, createdMessage => {
    Actions.createSUSIMessage(createdMessage, message.threadID);
  });
};

To know more about the project join us on Gitter at gitter.im/fossasia/susi_webchat, or to contribute go to https://github.com/fossasia/chat.susi.ai/.

A demo application can be found running at http://chat.susi.ai.

Resources –

To know more about Flux you can visit the following websites.

Continue ReadingUsing Flux to embed SUSI’s API Service in a Chat System.

Adding Twitter Integration with MVP Architecture in Phimpme Android

The Account manager layout part of Phimpme is set. Now we need to start adding different account to make it functional. We first start with twitter. Twitter functionality is integrated with the help of Twitter Kit provided by Twitter itself. We followed the steps provided on the installation guide.

Note: Before Starting this first go to apps.twitter.com and create new app, add the relevant information such as name, description, URL etc.

How twitter works in Phimpme

A dialog box appear when user selected the add account option in Account manager. Select Twitter option from it.

Twitter guides to use custom TwitterLoginButton for sign in. But as we are using a common dialog box. How to initiate login from there?

Using TwitterAuthClient

Twitter Auth Client invoke the Twitter callback and popup the login window. On authorized the correct user it goes inside the onSuccess method and start a Twitter session which helps us to get the information which we want to store in database such as username, access token.

client.authorize(getActivity(), new Callback<TwitterSession>() {
   @Override
   public void success(Result<TwitterSession> result) {

       // Creating twitter session, after user authenticate
       // in twitter popup
       TwitterSession session = TwitterCore.getInstance()
               .getSessionManager().getActiveSession();
       TwitterAuthToken authToken = session.getAuthToken();


       // Writing values in Realm database
       account.setUsername(session.getUserName());
       account.setToken(String.valueOf(session.getAuthToken()));

   }

Working with MVP architecture to show Twitter data to User in a recyclerView

Finally after successful login from Twitter, we also need to show user that you are successfully logged in Phimpme app and also provide a sign out feature so that user can logout from Twitter anytime.

Account manager has a recyclerView which takes data from the database and show it to the User.

Steps:

class AccountContract {
   internal interface View : MvpView{

       /**
        * Setting up the recyclerView. The layout manager, decorator etc.
        */
       fun setUpRecyclerView()

       /**
        * Account Presenter calls this function after taking data from Database Helper Class
        */
       fun setUpAdapter(accountDetails: RealmResults<AccountDatabase>)

       /**
        * Shows the error log
        */
       fun showError()
   }

   internal interface Presenter {

       /**
        * function to load data from database, using Database Helper class
        */
       fun loadFromDatabase()

       /**
        * setting up the recyclerView adapter from here
        */
       fun handleResults(accountDetails: RealmResults<AccountDatabase>)
   }
}

This class clearly show what functions are in View and what are in Presenter. The View interface extended to MvpView which actually holds some common functions such as onComplete()

  • Implement View interface in AccountActivity

class AccountActivity : ThemedActivity(), AccountContract.View

And perform all the action which are happening on the View such as setting up the RecyclerView

override fun setUpRecyclerView() {
   val layoutManager = LinearLayoutManager(this)
   accountsRecyclerView!!.setLayoutManager(layoutManager)
   accountsRecyclerView!!.setAdapter(accountAdapter)
}
  • Main Business Logic should not be in Activity class

That’s why using MVP we have very less number of lines of code in our Main Activity because it separate the work on different zones which help developer to easily work, maintain and other user to contribute.

So like in our case I need to update the RecyclerView adapter by taking data from database. That work should not be in activity that’s why I create a class AccountPresenter and extend this to our Presenter interface in contract class

class AccountPresenter extends BasePresenter<AccountContract.View>
       implements AccountContract.Presenter

I added the function which take care of loading data from database

@Override
public void loadFromDatabase() {
   handleResults(databaseHelper.fetchAccountDetails());
}
  • Always consider the future and keep an eye for future development

Right now I not need to do alot on Database, I just need to pick the whole data and show it on View. But I need to take case of future development in this part as well. There might be more complex operation on Database in future, then it will create complexity in the codebase, if it is not architectured well.

So, I created a DatabaseHelper class which takes care of all the database operations, the advantage of this is, anyone who have to contribute in Database part or debugging the databse need not to search for every activity and scroll lines of code, the work will be in DatabaseHelper for sure.

Added DatabaseHelper in data package

public class DatabaseHelper {

   private Realm realm;

   public DatabaseHelper(Realm realm) {
       this.realm = realm;
   }

   public RealmResults<AccountDatabase> fetchAccountDetails(){
       return realm.where(AccountDatabase.class).findAll();
   }

   public void deleteSignedOutAccount(String accountName){
       final RealmResults<AccountDatabase> deletionQueryResult =  realm.where(AccountDatabase.class)
               .equalTo("name", accountName).findAll();

       realm.executeTransaction(new Realm.Transaction() {
           @Override
           public void execute(Realm realm) {
               deletionQueryResult.deleteAllFromRealm();
           }
       });
   }
}

Flow Diagram:

Browse the Phimpme GitHub Repository for complete illustration.

Resources

  1. Twitter KIt overview : https://dev.twitter.com/twitterkit/android/overview
  2. Login with Twitter: https://dev.twitter.com/twitterkit/android/log-in-with-twitter
  3. MVP Architecture by Ribot: https://github.com/ribot/android-boilerplate

 

Continue ReadingAdding Twitter Integration with MVP Architecture in Phimpme Android

Adding Image Editor in Phimpme Android app

Image editing is a core feature of our Phimpme Android Application. We are using this already existing components Image Editor. Currently our app flow is Gallery → Open images. Now we add an option menu in the top named “Edit” which redirect to the edit Image Activity with that image path. Perform the edit operations there and apply the changed and finally updated the album with new file.

The task is to first get the image path pass to the EditImageActivity and then save the edited image in different folder after user finish with editing.

Below are the high level architecture of how it is achieved with help of ImageEditor.

Steps to integrate Editor:

  • Get the selected Image URI

Firstly, add the permission in manifest file

<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />

Get the local images in your internal or SD Card using Media Store content provider.

Uri uri = MediaStore.Images.Media.EXTERNAL_CONTENT_URI; 

String[] projection = { MediaStore.Images.Media._ID, MediaStore.Images.Media.BUCKET_ID,MediaStore.Images.Media.BUCKET_DISPLAY_NAME }; 

Cursor cursor = getContentResolver().query(uri, projection, null, null, null);

After that you can iterate the cursor to end and get details.

  • Create a separate folder where edited images will be saved.

Now after editing we have to place the edited image into a seperate folder with a new name. You can create folder like this:

File aviaryFolder = new File(baseDir, FOLDER_NAME);

Generating new file which will be saved as edited image

public static File getEmptyFile(String name) {

  File folder = FileUtils.createFolders();

  if (folder != null) {

     if (folder.exists()) {

        File file = new File(folder, name);

        return file;

     }

  }

  return null;

}
FileUtils.getEmptyFile("edited"

     + System.currentTimeMillis() + ".png");

 

It will save new image as named edited<timeinmillis> so it will be unique.

  • StartActivityforResult with the image URI and edit activity

So firstly, when user open image and click on edit. We create an empty file using the above method. Getting the image uri and pass it into the editImageActivity. Other than Uri, also pass the output file path.

And startActivityForResult with intent and request code.

You can track the editing done by user and finally save if there is some edit done, like in Phimpme we keep a counter which increased on change of Bitmap.

Update the Album by putting the image Media Store

ContentValues values = new ContentValues(2);

String extensionName = getExtensionName(dstPath);

values.put(MediaStore.Images.Media.MIME_TYPE, "image/" + (TextUtils.isEmpty(extensionName) ? "jpeg" : extensionName));

values.put(MediaStore.Images.Media.DATA, dstPath);

context.getContentResolver().insert(MediaStore.Images.Media.EXTERNAL_CONTENT_URI, values);

Please visit the whole code here: github.com/fossasia/phimpme-android

Resources

Continue ReadingAdding Image Editor in Phimpme Android app

Adding Custom Scrollbar to SUSI AI Web Chat

Scrollbar represents the depth of content on your current screen. It appears when the content has overflown the depth of screen and cannot fit it anymore. We see scrollbars everywhere. By default, the scrollbar provided by the browser is not very attractive but efficient in doing its job.

We decided that as our SUSI.AI Web App is improving in both UI and functionality, let’s add a custom scrollbar to it.

Earlier we had a standard  scrollbar in our SUSI.AI webchat:

For adding a custom scrollbar to our web chat we decided to use react-custom-scrollbars npm-package.

Our reasons to choose this package were:

  • Auto Hide feature in the scrollbar, after a specific period of time, which we can modify too.
  • No requirement for extra CSS styles to style our scrollbar.
  • It is well tested and trusted by many developers in open source

To install this npm package:

npm install -S react-custom-scrollbars 

Now comes the usage part, we need to import this into our JavaScript file:

 import { Scrollbars } from 'react-custom-scrollbars';

After importing, wrap it around the data where you want to show a custom scrollbar. In our case it was messageListItems, the code snippet looked like:

<Scrollbars>
 {messageListItems}
</Scrollbars>

This made our scrollbar look much better than the default one:

Now to add Auto Hide functionality to our scrollbar we need to add some attributes to our <Scrollbars>  tag.

    1. autoHide: It allows the auto-hide feature to our scrollbar.
    2. autoHideTimeout: It allows us to set custom time of hiding delay of a scrollbar in milli-seconds.
    3. autoHideDuration: it allows us to set the duration of hiding animation in milliseconds.

After adding the above-mentioned attributes our code changes to:

<Scrollbars
 autoHide
 autoHideTimeout={1000}
 autoHideDuration={200}>
 {messageListItems}
</Scrollbars>

Resources:

A lot more of custom attributes can be found in the documentation of Malte Wessel here.

Testing Link:

Now we had a much better scrollbar for our web chat which can be tested here.

Continue ReadingAdding Custom Scrollbar to SUSI AI Web Chat

Refactor of Dropdown Menu in Susper

The first version of the Susper top menu was providing links to resources and tutorials. In the next version of the menu, we were looking for a menu with more colorful icons, a cleaner UI design and a menu that should appear on the homepage as well. In this blog, I will discuss about refactoring the dropdown menu. This is how earlier dropdown of Susper looks like:

We decided to create a separate component for the menu DropdownComponent.

At first, I created a drop down menu with matching dimensions similar to what Google follows. Then, I gave padding: 28px to create similar UI to market leader. This will make a dropdown menu with clean UI design. I replaced the old icons with colorful icons. In the dropdown we have:

  • Added more projects of FOSSASIA like eventyay, loklak, susi and main website of FOSSASIA. Here how it looks now :

The main problem I faced was aligning the content inside the dropdown and they should not get disturbed when the screen size changes.
I kept the each icon dimensions as 48 x 48 inside drop down menu. I also arranged them in a row. It was easy to use div element to create rows rather than using ul and li tags which were implemented earlier.

To create a horizontal grey line effect, I used the hr element. I made sure, padding remained the same above and below the horizontal line.

At the end of drop down menu, @mariobehling suggested instead of writing ‘more’, it should redirect to projects page of FOSSASIA.

This is how I worked on refactoring drop down menu and added it on the homepage as well.

Resources

Continue ReadingRefactor of Dropdown Menu in Susper

Generate Sine Waves with PSLab Device

Sine wave is type of a waveform with much of a use in frequency related studies in laboratories as well as power electronics to control the level of input to devices. PSLab device  is capable of generating sine waves with a very high accuracy using PSLab-firmware and a set of filters implemented in the PSLab-hardware.

How Sine Wave is generated in PSLab Device

PSLab device uses a PIC micro-controller as its main processor. It has several pins which can generate square pulses at different duty cycles. These are known as PWM pins. PWM waves are a type of a waveform with the shape resembling a set of square pulses. They have an attributed called ‘Duty Cycle’ which varies between 0% to 100%. A PWM wave with 0% duty cycle means simply a zero amplitude block of square pulses repeating at every period. When duty cycle is set to 100%, it is a set of square pulses with the highest amplitude throughout the period repeating in every period. The following figure illustrates how the PWM wave changes according to its duty cycle. Image is extracted from http://static.righto.com/images/pwm1.gif PSLab device is capable of generating this type of pulses with arbitrary duty cycles as per user requirements.

In this context where sine waves are generated, these PWM pins are used to generate a Sinusoidal Pulse Width Modulated (SPWM) waveform as the first step to output a sine wave with high frequency accuracy. The name SPWM is derived from the fact that the duty cycle of the waveform follows an alternatively increasing and decreasing pattern as illustrated in the figure below.

Deriving a set of duty cycles which follows a sinusoidal pattern is a redundant task. Without deriving them mathematically, PSLab firmware has four hard-coded sine_tables which stores different duty cycle values related to a SPWM waveform. These sine_tables in the firmware related to different resolutions set by the PSLab device user. The following code block is extracted from PSLab firmware related to one of the sine_tables. It is used to generate the SPWM wave with 512 data points. Each data point represents a square pulse with a different pulse width. The duty ratio is calculated from dividing an entry by the value 512 and converting it to a percentage.

sineTable1[] = {256, 252, 249, 246, 243, 240, 237, 234, 230, 227, 224, 221, 218, 215, 212, 209, 206, 203, 200, 196, 193, 190, 187, 184, 181, 178, 175, 172, 169, 166, 164, 161, 158, 155, 152, 149, 146, 143, 141, 138, 135, 132, 130, 127, 124, 121, 119, 116, 114, 111, 108, 106, 103, 101, 98, 96, 93, 91, 89, 86, 84, 82, 79, 77, 75, 73, 70, 68, 66, 64, 62, 60, 58, 56, 54, 52, 50, 48, 47, 45, 43, 41, 40, 38, 36, 35, 33, 32, 30, 29, 27, 26, 25, 23, 22, 21, 19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 8, 7, 6, 6, 5, 4, 4, 3, 3, 2, 2, 2, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 4, 4, 5, 6, 6, 7, 8, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 23, 25, 26, 27, 29, 30, 32, 33, 35, 36, 38, 40, 41, 43, 45, 47, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 73, 75, 77, 79, 82, 84, 86, 89, 91, 93, 96, 98, 101, 103, 106, 108, 111, 114, 116, 119, 121, 124, 127, 130, 132, 135, 138, 141, 143, 146, 149, 152, 155, 158, 161, 164, 166, 169, 172, 175, 178, 181, 184, 187, 190, 193, 196, 200, 203, 206, 209, 212, 215, 218, 221, 224, 227, 230, 234, 237, 240, 243, 246, 249, 252, 256, 259, 262, 265, 268, 271, 274, 277, 281, 284, 287, 290, 293, 296, 299, 302, 305, 308, 311, 315, 318, 321, 324, 327, 330, 333, 336, 339, 342, 345, 347, 350, 353, 356, 359, 362, 365, 368, 370, 373, 376, 379, 381, 384, 387, 390, 392, 395, 397, 400, 403, 405, 408, 410, 413, 415, 418, 420, 422, 425, 427, 429, 432, 434, 436, 438, 441, 443, 445, 447, 449, 451, 453, 455, 457, 459, 461, 463, 464, 466, 468, 470, 471, 473, 475, 476, 478, 479, 481, 482, 484, 485, 486, 488, 489, 490, 492, 493, 494, 495, 496, 497, 498, 499, 500, 501, 502, 503, 503, 504, 505, 505, 506, 507, 507, 508, 508, 509, 509, 509, 510, 510, 510, 511, 511, 511, 511, 511, 511, 511, 511, 511, 511, 511, 510, 510, 510, 509, 509, 509, 508, 508, 507, 507, 506, 505, 505, 504, 503, 503, 502, 501, 500, 499, 498, 497, 496, 495, 494, 493, 492, 490, 489, 488, 486, 485, 484, 482, 481, 479, 478, 476, 475, 473, 471, 470, 468, 466, 464, 463, 461, 459, 457, 455, 453, 451, 449, 447, 445, 443, 441, 438, 436, 434, 432, 429, 427, 425, 422, 420, 418, 415, 413, 410, 408, 405, 403, 400, 397, 395, 392, 390, 387, 384, 381, 379, 376, 373, 370, 368, 365, 362, 359, 356, 353, 350, 347, 345, 342, 339, 336, 333, 330, 327, 324, 321, 318, 315, 311, 308, 305, 302, 299, 296, 293, 290, 287, 284, 281, 277, 274, 271, 268, 265, 262, 259};

 

The frequency of the sine wave is achieved using interrupts generated at different time intervals depending on the frequency set by the user. The accuracy of the frequency depends on the number of elements in the sine table array which is known as resolution. As the number of points in the table increases, the accuracy will be increased or resolution will be high. PSLab uses Timer3 and Timer4 counters available in the PIC micro-controller to generate the interrupt time intervals. If the required frequency is f, the interrupt time interval can be derived as

Interrupt time = f/512

The derived SPWM waveform will be then passed through a cascaded setup of Op Amp and RC filter as in Figure 1 to cut off the high frequency components and combine the square pulses in such a manner that a smoother waveform is derived resembling a sine wave.

Filter Circuit in PSLab

Figure 1

This is an inverting filter circuit designed using Op Amps available in PSLab-hardware. The SPWM waveform will be connected to the circuit through R1 resistor. It uses C1 capacitor which creates a short circuit path to high frequency components to ground which will let only the low frequency components to pass through. This will smoothen the square waves reducing the sharp edges forming a simple RC filter circuit.

The C2 capacitor plays an important role in generating the sine wave. It will compensate any voltage drops and absorb excess voltage levels that might occur during transition to let the output waveform follow a path which is similar to a smooth sine wave.

The output waveform can be observed from the SINE1 pin of the PSLab device. As in this schematic, it is the ‘Sine Wave’ pin to the right starting from the Op Amp output.

Resources:

Continue ReadingGenerate Sine Waves with PSLab Device

Using MockBackend to test Susper Angular Front-end Code

In this blog, I’ll be sharing how we are testing services which we are using in Susper development.We’re using Angular 4 in our project as tech stack and we use Jasmine for testing purpose.

Tests are written to avoid issues which occur again and again. For example: Since we have implemented knowledge graph, we faced a lot of issues like:

  • When a user enters a query, search results appear but knowledge graph does not appear.
  • When a fresh query is entered or page is refreshed, knowledge graph does not appear.
  • The API which we have used is not responding.

We overcome this issue by writing test. The data is being taken with the help of an API. So, it will require testing using HTTP. Instead of testing like this, there is a better way by using MockBackend.

Testing with MockBackend is a more sensible approach. This allows us to mock our responses and avoid hitting the actual backend which results in boosting our testing.

To use the MockBackend feature, it requires creating a mock. For knowledge-service it looks like this:

export const MockKnowledgeApi {
  results: {
    uri: ‘http://dbpedia.org/resource/Berlin’,
    label: ‘Berlin’,
  }
  MaxHits: 5
};

To use the MockBackend feature, import MockBackend, MockConnection, BaseRequestOptions and MockKnowledgeApi.

import { MockBackend, MockConnection } from ‘@angular/http/testing’;
import { MockKnowledgeApi } from ‘./shared/mock-backend/knowledge.mock’;
import { BaseRequestOptions } from ‘@angular/http’;

Create a mock setup. In this case, we will create mock setup w.r.t HTTP because data from API is being returned as HTTP. If data, is being returned in JSON format, create a mock setup w.r.t jsonp.

const mockHttp_provider = {
  provide: Http,
  deps: [MockBackend, BaseRequestOptions],
  useFactory: (backend: MockBackend,options: BaseRequestOptions) => {
    return new Http(backend, options);
}
};

Now, describe the test suite. Inside, describe the function, don’t import MockConnection. It will throw error since it is only used to create a fake backend. It should look like this:

providers: [
  KnowledgeapiService,
  MockBackend,
  BaseRequestOptions,
  mockHttp_provider,
]
Define service as KnowledgeService and backend as MockBackend. Inject both the services in beforeEach() function.
Now to actually test the service, create a query.

const searchquery = ‘Berlin’;

The written specs should look like this. I won’t go much in detail here, but I’ll cover up the key points of code.

it(‘should call knowledge service API and return the result’, () => {
backend.connections.subscribe((connection: MockConnection) => {
const options = new ResponseOptions({
body: JSON.stringify(MockKnowledgeApi)
});connection.mockRespond(new Response(options));
expect(connection.request.method).toEqual(RequestMethod.Get);
});
Here, mockRespond will mock our response and it will test whether the service is working or not. Already, we have defined a query.

It should have a link to API and should be equal to searchquery which we have defined already as ‘Berlin’.

expect(connection.request.url).toBe(
`http://lookup.dbpedia.org/api/search/KeywordSearch` +
`?&QueryString=${searchquery}`
);

At last, it will check if it’s working or not. If it’s working, then test case will pass. Please note, it will not hit the actual backend.

service.getsearchresults(searchquery).subscribe((res) => {
expect(res).toEqual(MockKnowledgeApi);
);
In this way, we have written tests for knowledge graph to avoid future issues. We will be adding tests like these for other services as well which are being used in Susper project.

Resources

Continue ReadingUsing MockBackend to test Susper Angular Front-end Code

Filtering out a query in Solr for Susper

In this blog, I would like to share you how- we have implemented tabs feature in Susper.

When a search is attempted, results are fetched in results.component. Since there are many kinds of results being fetched from the server, how should we filter them in separate tabs? For example under the tab ‘All’: All kind of results are being fetched but .png or .mp3 which should be loaded under tabs ‘Images’ and ‘Videos’. The yacy server is written in Java on Solr technology. Going through Solr documents, we found the solution to filter queries using ‘fq’ parameter.

What is fq parameter?

‘fq’ stands for filter query. It returns the document which we want or in simple words returns filtered document. Documents will only be included in the result if they are at the intersection of the document sets resulting from each fq.

How we did we use fq parameter in Susper?

To explain it better, I’m sharing a code snippet here which has been written to filter out queries :

if (query[‘fq’]) {
  if (query[‘fq’].includes(‘png’)) {
    this.resultDisplay = ‘images’;
    urldata.fq = ‘url_file_ext_s:(png+OR+jpeg+OR+jpg+OR+gif)’;
} else if (query[‘fq’].includes(‘avi’)) {
    this.resultDisplay = ‘videos’;
} else {
  this.resultDisplay = ‘all’;
}
}

What is ongoing here is that we have subscribed to a query and used if and else conditions. query[‘fq’] simply filters out the query which has been subscribed. include(‘png’) and include(.avi) is clear that we are filtering out the documents with these tabs. This action happens when the user clicks on a tab.

If the user clicks on images tab: files with .png are displayed. If the user clicks on videos tab: files with .avi are displayed.

url_file_ext_s:() is simply another solr syntax to provide the document format.

The flowchart above explains more clearly, how fq parameter is filtering out documents without affecting the total number of documents which yacy fetches by indexing web pages based on the query.

Resources:

Continue ReadingFiltering out a query in Solr for Susper

How to Make SUSI AI Slack Bot

To make SUSI slack bot we will use real time messaging api of slack which will allow users to receive messages from bot in real time. To make SUSI slack bot you have to follow following steps:

Steps:

  1. First of all you have to create a team on slack in where your bot will be running. To create a team go to https://slack.com/ and create a new team.
  2. After creating sign in to your team and got to apps and integration option by clicking on left corner.
  3. Click manage on top right corner and go to custom integrations to add configuration to Bots.
  4. After adding configuration data,bot username and copying API Token now we have to write code for setting bot in slack. To set up code see below steps: 
  5. Install Node.js from the link below on your computer if you haven’t installed it already. https://nodejs.org/en/
  6. Create a folder with any name and open shell and change your current directory to the new folder you created.
  7. Type npm init in command line and enter details like name, version and entry point.
  8. Create a file with the same name that you wrote in entry point in above given step. i.e index.js and it should be in same folder you created.
  9. Type following commands in command line  npm install –save @slack/client. After slack/client is installed type npm install –save express after express is installed type npm install –save request and then npm install –save http when all the modules are installed check your package.json modules will be included within dependencies portion.
  10. Your package.json file should look like this.
    {
    "name": "slack-bot",
    "version": "1.0.0",
    "description": "SUSI Slack Bot",
    "main": "index.js",
    "dependencies": {
           "express": "^4.15.3",
           "http": "0.0.0",
           "request": "^2.81.0"
    },
    "scripts": {
           "test": "echo \"Error: no test specified\" && exit 1",
           "start": "node index.js"
    }
    }
    
  11. Copy following code into file you created i.e index.js.
    var Slack = require('@slack/client');
    var request = require('request');
    var express = require('express');
    var http = require('http');
    var app = express();
    var RtmClient = Slack.RtmClient; 
    var RTM_EVENTS = Slack.RTM_EVENTS;
    var token = process.env.APIToken;
    
    var rtm = new RtmClient(token, { logLevel: 'info' }); 
    rtm.start();
    
    //to ping heorku app after 20 minutes to keep it active
    
    setInterval(function() {
            http.get(process.env.HerokuUrl);
        }, 1200000);
    
    rtm.on(RTM_EVENTS.MESSAGE, function(message) { 
    var channel = message.channel;
    
    var options = {
           method: 'GET',
           url: 'http://api.asksusi.com/susi/chat.json',
           qs: {
               timezoneOffset: '-330',
               q: message.text
           }
       };
    
    //sending request to SUSI API for response
       request(options, function(error, response, body) {
           if (error) throw new Error(error);
           var ans = (JSON.parse(body)).answers[0].actions[0].expression;
           rtm.sendMessage(ans, channel);
       })
    });
    
    const port = process.env.PORT || 3000;
    app.listen(port, () => {
       console.log(`listening on ${port}`);
    });
     
    
    


  12. Now we have to deploy this code to heroku.
  13. Before deploying we have to make a github repository for chatbot to make github repository follow these steps:

    In command line change current directory to folder we created above and write

    git init
    git add .
    git commit -m”initial”
    git remote add origin <URL for remote repository>
    git remote -v
    git push -u origin master

    You will get URL for remote repository by making repository on your github and copying this link of your repository.

  14. To deploy your bot to heroku you need an account on Heroku and after making an account make an app.
  15. Deploy app using github deployment method.
  16. Select Automatic deployment method.
  17. Add APIToken and HerokuUrl variable to heroku app in settings options.
  18. Your SUSI Slack bot is ready enjoy chatting with it.If you want to learn more about slack API refer to https://api.slack.com
Continue ReadingHow to Make SUSI AI Slack Bot

Improving Harvesting Decision for Kaizen Harvester in loklak server

About Kaizen Harvester

Kaizen is an alternative approach to do harvesting in loklak. It focuses on query and information collecting to generate more queries from collected timelines. It maintains a queue of query that is populated by extracting following information from timelines –

  1. Hashtags in Tweets
  2. User mentions in Tweets
  3. Tweets from areas near to each Tweet in timeline.
  4. Tweets older than oldest Tweet in timeline.

Further, it can also utilise Twitter API to get trending keywords from Twitter and get search suggestions from other loklak peers.

It was introduced by @yukiisbored in pull request loklak/loklak_server#960.

The Problem: Unbiased Harvesting Decision

The Kaizen harvester either searches for queries from the queue, or tries to grab trending queries (using Twitter API or from backend). In the previous version of KaizenHarvester, the decision of “harvesting vs. info-grabbing” was taken based on the value from a random boolean generator –

@Override
public int harvest() {
   if (!queries.isEmpty() && random.nextBoolean())
       return harvestMessages();

   grabSuggestions();

   return 0;
}

[SOURCE]

In sane situations, the Kaizen harvester is configured to use a fixed size queue and drops the queries which are requested to get added once the queue is full. And since the decision doesn’t take into account the amount to which queue is filled, it would often call the grabSuggestions() method.

But since the queue would be full, the grabbed suggestions would simply be lost. This would result in wastage of time and resources in fetching the suggestions (from backend or API). To overcome this, something better was to be done in this part.

The Solution: Making Decision Biased

To solve the problem of dumb harvesting decision, the harvester was triggered based on the following steps –

  1. Calculate the ratio of queue filled (q.size() / q.maxSize()).
  2. Generate a random floating point number between 0 and 1.
  3. If the number is less than the fraction, harvest. Otherwise get harvesting suggestions.

Why would this work?

Initially, when the queue is mostly empty, the ratio would be a small number. So, it would be highly probable that a random number generated between 0 and 1 would be greater than the ratio. And Kaizen would go for grabbing search suggestions.

If this ratio is large (i.e. the queue is almost full), it would be highly likely that the random number generated would be less than it, making it more likely to search for results instead of grabbing suggestions.

Graph?

The following graph shows how the harvester decision would change. It performs 10k iterations for a given queue ratio and plots the number of times harvesting decision was taken.

Change in code

The harvest() method was changed in loklak/loklak_server#1158 to take smart decision of harvesting vs. info-grabbing in following manner –

@Override
public int harvest() {
   float targetProb = random.nextFloat();
   float prob = 0.5F;
   if (QUERIES_LIMIT > 0) {
       prob = queries.size() / (float)QUERIES_LIMIT;
   }
   if (!queries.isEmpty() && targetProb < prob) {
       return harvestMessages();
   }

   grabSuggestions();

   return 0;
}

[SOURCE]

Conclusion

This change brought enhancement in the Kaizen harvester and made it more sensible to how fast its queue if filling. There are no more requests made to backend for suggestions whose queries are not added to the queue.

 

Resources

Continue ReadingImproving Harvesting Decision for Kaizen Harvester in loklak server