Implementing a Collapsible Responsive App Bar of SUSI Web Chat

In the SUSI Web Chat application we wanted to make few static pages such as: Overview, Terms, Support, Docs, Blog, Team, Settings and About. The idea was to show them in app bar. Requirements were  to have the capability to collapse on small viewports and to use Material UI components. In this blog post I’m going to elaborate how we built the responsive app bar Using Material UI components.

First we added usual Material UI app bar component  like this.

             <header className="nav-down" id="headerSection">
             <AppBar
               className="topAppBar"
title={<a href={this.state.baseUrl} ><img src="susi-white.svg" alt="susi-logo"  className="siteTitle"/></a>}
               style={{backgroundColor:'#0084ff'}}
               onLeftIconButtonTouchTap={this.handleDrawer}
               iconElementRight={<TopMenu />}
             />
             </header>

We added SUSI logo instead of the text title using below code snippet and linked it to the home page like this.

title={<a href={this.state.baseUrl} ><img src="susi-white.svg" alt="susi-logo"  className="siteTitle"/></a>}

We have defined “this.state.baseUrl” in constructor and it gets the base url of the web application.

this.state = {
       openDrawer: false, 
baseUrl: window.location.protocol + '//' + window.location.host + '/'
       };

We need to open the right drawer when we click on the button on top left corner. So we have to define two methods to open and close drawer as below.

   handleDrawer = () => this.setState({openDrawer: !this.state.openDrawer});
   handleDrawerClose = () => this.setState({openDrawer: false});

Now we have to add components that we need to show on the right side of the app bar. We connect those elements to the app bar like this. “iconElementRight={}”

We defined “TopMenu” Items like this.

   const TopMenu = (props) => (
   <div>
     <div className="top-menu">
     <FlatButton label="Overview"  href="/overview" style={{color:'#fff'}} className="topMenu-item"/>
     <FlatButton label="Team"  href="/team" style={{color:'#fff'}} className="topMenu-item"/>
     </div>

We added FlatButtons to place links to other static pages. After all we needed a FlatButton that gives IconMenu to show login and signup options.

     <IconMenu {...props} iconButtonElement={
         <IconButton iconStyle={{color:'#fff'}} ><MoreVertIcon /></IconButton>
     }>
     <MenuItem primaryText="Chat" containerElement={<Link to="/logout" />}
                   rightIcon={<Chat/>}/>
     </IconMenu>
   </div>
   );

After adding all these correctly you will see this kind of an app bar in your application.

Now our app bar is ready. But it does not collapse on small viewports.
So we planned to hide flat buttons on small sized screens and show the menu button. For that we used media queries.

@media only screen and (max-width: 800px){
   .topMenu-item{ display: none !important;  }
   .topAppBar button{ display: block  !important; }
}

This is how we built the responsive app bar using Material UI components. You can check the preview from this url. If you are willing to contribute to SUSI Web Chat here is the GitHub repository.

Resources:

  • Material UI Components: http://www.material-ui.com/#/components/
  • Learn More about media queries: https://www.w3schools.com/css/css_rwd_mediaqueries.asp
Continue ReadingImplementing a Collapsible Responsive App Bar of SUSI Web Chat

Displaying a Comments dialogfragment on a Button Click from the Feed Adapter in the Open Event Android App

Developing the live feed of the event page from Facebook for the Open Event Android App, there were questions how best to display the comments in the feed.  A dialog fragment over the feeds on the click of a button was the most suitable solution. Now the problem was, a dialogfragment can only be called from an app component (eg- fragment or an activity). Therefore, the only challenge which remained was to call the dialogfragment from the adapter over the feed fragment with the corresponding comments of the particular post on a button click.

What is a dialogfragment?

A dialogfragment displays a dialog window, floating on top of its activity’s window. This fragment contains a Dialog object, which it displays as appropriate based on the fragment’s state. Control of the dialog (deciding when to show, hide, dismiss it) should be done through the API here, not with direct calls on the dialog (Developer.Android.com).

Solution

The solution which worked on was to define a adapter callback interface with a onMethodCallback method in the feed adapter class itself with the list of comment items fetched at runtime on the button click of a particular post. The interface had to be implemented by the main activity which housed the feed fragment that would be creating the comments dialogfragment with the passed list of comments.

Implementation

Define an interface adapterCallback with the method onMethodCallback parameterized by the list of comment items in your adapter class.

public interface AdapterCallback {
   void onMethodCallback(List<CommentItem> commentItems);
}

 

Create a constructor of the adapter with the adapterCallback as a parameter. Do not forget to surround it with a try/catch.

public FeedAdapter(Context context, AdapterCallback adapterCallback, List<FeedItem> feedItems) {
     this.mAdapterCallback = adapterCallback;
}

 

On the click of the comments button, call onMethodCallback method with the corresponding comment items of a particular feed.

getComments.setOnClickListener(v -> {
   if(commentItems.size()!=0)
       mAdapterCallback.onMethodCallback(commentItems);
});

 

Finally implement the interface in the activity to display the comments dialog fragment populated with the corresponding comments of a feed post. Pass the comments with the help of arraylist through the bundle.

@Override
public void onMethodCallback(List<CommentItem> commentItems) {
   CommentsDialogFragment newFragment = new CommentsDialogFragment();
   Bundle bundle = new Bundle();
   bundle.putParcelableArrayList(ConstantStrings.FACEBOOK_COMMENTS, new ArrayList<>(commentItems));
   newFragment.setArguments(bundle);
   newFragment.show(fragmentManager, "Comments");
}

 

Conclusion

The comments generated with each feed post in the open event android app does complement the feed well. The pagination is something which is an option in the comments and the feed both however that is something for some other time. Until then, keep coding!

Resources

Continue ReadingDisplaying a Comments dialogfragment on a Button Click from the Feed Adapter in the Open Event Android App

Sharing Images on Twitter from Phimpme Android App Using twitter4j

As sharing an image to the social media platform is an important feature in Phimpme android. In my previous blog, I have explained how to authenticate the Android application with Twitter. In this blog, I will discuss how to upload an image directly on Twitter from the application after successfully logging to Twitter.

To check if the application is authenticated to Twitter or not.

When the application is successfully authenticated Twitter issues a Token which tells the application if it is connected to Twitter or not. In LoginActivity.java the function isActive returns a boolean value. True if the Twitter token is successfully issued or else false.  

public static boolean isActive(Context ctx) {
        SharedPreferences sharedPrefs = ctx.getSharedPreferences(AppConstant.SHARED_PREF_NAME, Context.MODE_PRIVATE);
        return sharedPrefs.getString(AppConstant.SHARED_PREF_KEY_TOKEN, null) != null;
    }

We call isActive function from LoginActive class to check if the application is authenticated to Twitter or not. We call it before using the share function in sharingActivity:

if (LoginActivity.isActive(context)) {
                try {
                    // Send Image function
} catch (Exception ex) {
                    Toast.makeText(context, "ERROR", Toast.LENGTH_SHORT).show();
 }

We have saved the image in the internal storage of the device and use saveFilePath to use the path of the saved image. In Phimpme we used HelperMethod class where our share function resides, and while the image is being shared an alert dialog box with spinner pops on the screen.

Sending the image to HelperMethod class

First,

We need to get the image and convert it into Bitmaps. Since, the image captured by the phone camera is usually large to upload and it will take a lot of time we need to compress the Bitmap first. BitmapFactory.decodeFile(specify name of the file) is used to fetch the file and convert it into bitmap.

To send the data we used FileOutStream to the set the path of the file or image in this case. Bitmap.compress method is used to compress the image to desired value and format. In Phimpme we are converting it into PNG.  

Bitmap bmp = BitmapFactory.decodeFile(saveFilePath);
                    String filename = Environment.getExternalStorageDirectory().toString() + File.separator + "1.png";
                    Log.d("BITMAP", filename);
                    FileOutputStream out = new FileOutputStream(saveFilePath);
                    bmp.compress(Bitmap.CompressFormat.PNG, 90, out);

                    HelperMethods.postToTwitterWithImage(context, ((Activity) context), saveFilePath, caption, new HelperMethods.TwitterCallback() {

                        @Override
                        public void onFinsihed(Boolean response) {
                            mAlertBuilder.dismiss();
                            Snackbar.make(parent, R.string.tweet_posted_on_twitter, Snackbar.LENGTH_LONG).show();
                        }

Post image function

To post the image on Twitter we will use ConfigurationBuilder class. We will create a new object of the class and then attach Twitter consumer key, consumer secret key, Twitter access token, and twitter token secret.

  • setOAuthConsumerKey() function is used to set the consumer key which is generated by the Twitter when creating the application in the Twitter development environment.
  • Similarly, setOAuthConsumerSecret() function is used to set the consumer secret key.
  • Specify the token key which generated after successfully connecting to twitter in setOAuthAcessToken() fuction and Token secret in setOAuthAcessTokenSecret() function.  
ConfigurationBuilder configurationBuilder = new ConfigurationBuilder();       configurationBuilder.setOAuthConsumerKey(context.getResources().getString(R.string.twitter_consumer_key));
configurationBuilder.setOAuthConsumerSecret(context.getResources().getString(R.string.twitter_consumer_secret));
configurationBuilder.setOAuthAccessToken(LoginActivity.getAccessToken((context)));
configurationBuilder.setOAuthAccessTokenSecret(LoginActivity.getAccessTokenSecret(context));
        Configuration configuration = configurationBuilder.build();
final Twitter twitter = new TwitterFactory(configuration).getInstance();

Sending Image to twitter:

  • The image is uploaded to twitter using statusUpdate class specified in Twitter4j API.
  • Pass the image file name in status.setMedia(file).
  • Pass the caption in status.updateStatus(caption).
  • updateStatus is used to finally upload the image with the caption.
final File file = new File(imageUrl);

        new Thread(new Runnable() {
            @Override
            public void run() {
                boolean success = true;
                try {
                    if (file.exists()) {
                        StatusUpdate status = new StatusUpdate(message);
                        status.setMedia(file);
                        twitter.updateStatus(status);
                    }else{
                        Log.d(TAG, "----- Invalid File ----------");
                        success = false;
                    }
                } catch (Exception e) {
                    e.printStackTrace();
                    success = false;
                }

 Conclusion:                                                                                                                      Using Twitter4j API allows sharing image on Twitter without leaving the  application and opening any additional view.

Github

Resources

Continue ReadingSharing Images on Twitter from Phimpme Android App Using twitter4j

Integrating Twitter Authenticating using Twitter4j in Phimpme Android Application

We have used Twitter4j API to authenticate Twitter in Phimpme application. Below are the following steps in setting up the Twitter4j API in Phimpme and Login to Twitter from Phimpme android application.

Setting up the environment

Download the Twitter4j package from http://twitter4j.org/en/. For sharing images we will only need twitter4j-core-3.0.5.jar and twitter4j-media-support-3.0.5.jar files. Copy these files and save it in the libs folder of the application.

Go to build.gradle and add the following codes in dependencies:

dependencies {
compile files('libs/twitter4j-core-3.0.5.jar')
compile files('libs/twitter4j-media-support-3.0.5.jar')
}

Adding Phimpme application in Twitter development page

Go to https://dev.twitter.com/->My apps-> Create new apps. Create an application window opens where we have to fill all the necessary details about the application. It is mandatory to fill all the fields. In website field, if you are making an android application then anything can be filled in website field for example www.google.com. But it is necessary to fill this field also.

After filling all the details click on “Create your Twitter application” button.

Adding Twitter Consumer Key and Secret Key

This generates twitter consumer key and twitter secret key. We need to add this in our string.xml folder.

<string name="twitter_consumer_key">ry1PDPXM6rwFVC1KhQ585bJPy</string>
<string name="twitter_consumer_secret">O3qUqqBLinr8qrRvx3GXHWBB1AN10Ax26vXZdNlYlEBF3vzPFt</string> 

Twitter Authentication

Make a new JAVA class say LoginActivity. Where we have to first fetch the twitter consumer key and Twitter secret key.

private static Twitter twitter;
    private static RequestToken requestToken;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_twitter_login);
        twitterConsumerKey = getResources().getString(R.string.twitter_consumer_key);
        twitterConsumerSecret = getResources().getString(R.string.twitter_consumer_secret);  

We are using a web view to interact with the Twitter login page.

twitterLoginWebView = (WebView)findViewById(R.id.twitterLoginWebView);
        twitterLoginWebView.setBackgroundColor(Color.TRANSPARENT);
        twitterLoginWebView.setWebViewClient( new WebViewClient(){
            @Override
            public boolean shouldOverrideUrlLoading(WebView view, String url){

                if( url.contains(AppConstant.TWITTER_CALLBACK_URL)){
                    Uri uri = Uri.parse(url);
                    LoginActivity.this.saveAccessTokenAndFinish(uri);
                    return true;
                }
                return false;
            }             

If the access Token is already saved then the user is already signed in or else it sends the Twitter consumer key and the Twitter secret key to gain access Token. ConfigurationBuilder function is used to set the consumer key and consumer secret key.

ConfigurationBuilder configurationBuilder = new ConfigurationBuilder();
        configurationBuilder.setOAuthConsumerKey(twitterConsumerKey);     configurationBuilder.setOAuthConsumerSecret(twitterConsumerSecret);
        Configuration configuration = configurationBuilder.build();
        twitter = new TwitterFactory(configuration).getInstance();

It is followed by the following Runnable thread to check if the request token is received or not. If authentication fails, an error Toast message pops.

new Thread(new Runnable() {
            @Override
            public void run() {
                try {
                    requestToken = twitter.getOAuthRequestToken(AppConstant.TWITTER_CALLBACK_URL);
                } catch (Exception e) {
                    final String errorString = e.toString();
                    LoginActivity.this.runOnUiThread(new Runnable() {
                        @Override
                        public void run() {
                            mAlertBuilder.cancel();
                            Toast.makeText(LoginActivity.this, errorString, Toast.LENGTH_SHORT).show();
                            finish();
                        }
                    });
                    return;
                }

                LoginActivity.this.runOnUiThread(new Runnable() {
                    @Override
                    public void run() {
                        twitterLoginWebView.loadUrl(requestToken.getAuthenticationURL());
                    }
                });
            }
        }).start();

Conclusion

It offers seamless integration of Twitter in any application. Without leaving actual application, easier to authenticate. Further, it is used to upload the photo to Twitter directly from Phimpme Android application, fetch profile picture and username.

Github

Resources

 

Continue ReadingIntegrating Twitter Authenticating using Twitter4j in Phimpme Android Application

Adding JSONAPI Support in Open Event Android App

The Open Event API Server exposes a well documented JSONAPI compliant REST API that can be used in The Open Even App Generator and Frontend to access and manipulate data. So it is also needed to add support of JSONAPI in external services like The Open Even App Generator and Frontend. In this post I explain how to add JSONAPI support in Android.

There are many client libraries to implement JSONAPI support in Android or Java like moshi-jsonapi, morpheus etc. You can find the list here. The main problem is most of the libraries require to inherit attributes from Resource model but in the Open Event Android App we already inherit from a RealmObject class and in Java we can’t inherit from more than one model or class. So we will be using the jsonapi-converter library which uses annotation processing to add JSONAPI support.

1. Add dependency

In order to use jsonapi-converter in your app add following dependencies in your app module’s build.gradle file.

dependencies {
	compile 'com.github.jasminb:jsonapi-converter:0.7'
}

2.  Write model class

Models will be used to represent requests and responses. To support JSONAPI we need to take care of followings when writing the models.

  • Each model class must be annotated with com.github.jasminb.jsonapi.annotations.Type annotation
  • Each class must contain a String attribute annotated with com.github.jasminb.jsonapi.annotations.Id annotation
  • All relationships must be annotated with com.github.jasminb.jsonapi.annotations.Relationship annotation

In the Open Event Android we have so many models like event, session, track, microlocation, speaker etc. Here I am only defining track model because of its simplicity and less complexity.

@Type("track")
public class Track extends RealmObject {

        	@Id(IntegerIdHandler.class)
        	private int id;
        	private String name;
        	private String description;
        	private String color;
        	private String fontColor;
        	@Relationship("sessions")
        	private RealmList<Session> sessions;

        	//getters and setters
}

Jsonapi-converter uses Jackson for data parsing. To know how to use Jackson for parsing follow my previous blog.

Type annotation is used to instruct the serialization/deserialization library on how to process given model class. Each resource must have the id attribute. Id annotation is used to flag an attribute of a class as an id attribute. In above class the id attribute is int so we need to specify IntegerIdHandler class which is ResourceHandler in the annotation. Relationship annotation is used to designate other resource types as a relationship. The value in the Relationship annotation should be as per JSONAPI specification of the server. In the Open Event Project each track has the sessions so we need to add a Relationship annotation for it.

3.  Setup API service and retrofit

After defining models, define API service interface as you would usually do with standard JSON APIs.

public interface OpenEventAPI {
    @GET("tracks?include=sessions&fields[session]=title")
    Call<List<Track>> getTracks();
}

Now create an ObjectMapper & a retrofit object and initialize them.

ObjectMapper objectMapper = OpenEventApp.getObjectMapper();
Class[] classes = {Track.class, Session.class};

OpenEventAPI openEventAPI = new Retrofit.Builder()
                    .client(okHttpClient)
                    .baseUrl(Urls.BASE_URL)
                    .addConverterFactory(new JSONAPIConverterFactory(objectMapper, classes))
                    .build()
                    .create(OpenEventAPI.class);

 

The classes array instance contains a list of all the model classes which will be supported by this retrofit builder and API service. Here the main task is to add a JSONAPIConverterFactory which will be used to serialize and deserialize data according to JSONAPI specification. The JSONAPIConverterFactory constructor takes two parameters ObjectMapper and list of classes.

4.  Use API service  

Now after setting up all the things according to above steps, you can use the openEventAPI instance to fetch data from the server.

openEventAPI.getTracks();

Conclusion

JSON API is designed to minimize both the number of requests and the amount of data transmitted between clients and servers

Continue ReadingAdding JSONAPI Support in Open Event Android App

Implementing Hiding App Bar of SUSI Web Chat Application

In the SUSI Web Chat application we got a requirement to build a responsive app bar for static pages and there was another requirement to  show and hide the app bar when user scrolls. Basically this is how it should work: The app bar should be hidden after user scrolls down to a certain extent. When user scrolls up, It should appear again.

First we tried readymade node packages to do this task. But these packages are hard to customize. So we planned to make this feature from the sketch. We used Jquery for this. This is how we built this.

First we installed jQuery package using this command.

npm install jquery

Next we imported it on top of the application like this.

import $ from 'jquery'

We have discussed about this app bar and how we made it in previous blog post. Our app bar is like this.

             <header className="nav-down" id="headerSection">
             <AppBar
               className="topAppBar"
               title={<img src="susi-white.svg" alt="susi-logo"
               className="siteTitle"/>}
               style={{backgroundColor:'#0084ff'}}
               onLeftIconButtonTouchTap={this.handleDrawer}
               iconElementRight={<TopMenu />}
             />
             </header>

We have to use these HTML elements to write jQuery code. But we can’t refer HTML elements before it renders. So we have to define it soon after the render method executes. We can do it using “React LifeCycle” method. We have to add our code into the “componentDidMount()” method.
This is how we used jQuery inside the “componentDidMount()” lifeCycle method. Here we assigned the height of the App Bar using “$(‘header’).outerHeight();”

componentDidMount(){
     var didScroll;
     var lastScrollTop = 0;
     var delta = 5;
     var navbarHeight = $('header').outerHeight();

Here we assigned the height of the app bar to “navbarHeight” variable.

     $(window).scroll(function(event){
         didScroll = true;
     });

In this part we checked whether the user has scrolled or not. If user scrolled we set the value of “didScroll” to “true”.
Now we have to define what to do if user has scrolled.

     function hasScrolled() {
         var st = $(window).scrollTop();
         if(Math.abs(lastScrollTop - st) <= delta){

             return;
         }

Here we get the absolute scrolled height. If the height is less than the delta value we defined, it does not do anything. It just returns.

         if (st > lastScrollTop && st > navbarHeight){
             $('header').removeClass('nav-down').addClass('nav-up');
         } else if(st + $(window).height() < $(document).height()) {
             $('header').removeClass('nav-up').addClass('nav-down');
         }
         lastScrollTop = st;
     }

Here we hide the app bar after user scrolled down more than the height of the app bar. If we need to change the height which app bar should disappear, we just need to add a value to the condition like this.

if (st > lastScrollTop && st > navbarHeight + 200){

If the user scrolled down more than that value we change the class name of the element “nav-down” to “nav-up”.
And we change the className “nav-up” to “nav-down” when user is scrolling up.
We defined CSS classes in the stylesheet to do these things and the animations of action.

header {
   background: #f5b335;
   height: 40px;
   position: fixed;
   top: 0;
   transition: top 0.5s ease-in-out;
   width: 100%;
}

.nav-up {
   top: -100px;
}

We have defined the things which we need to do when user scrolls.
Now we have to call this function if user has scrolled

     setInterval(function() {
         if (didScroll) {
             hasScrolled();
             didScroll = false;
         }
     }, 2500);
   }

If the “didcroll” is “true” we execute the “hasScrolled()” function. And set 2500 millisecond time interval. Because of that app bar does not hide right after user scrolls. It triggers the function after 2.5 seconds later.
This is how we built the scroll bar hiding feature using react JS and jQuery.

Resources:

  • Learn more about React LifeCycle Methods http://busypeoples.github.io/post/react-component-lifecycle/
  • Use jQuery in React component: https://medium.com/@shuvohabib/using-jquery-in-react-component-the-refs-way-969de9aa651f
Continue ReadingImplementing Hiding App Bar of SUSI Web Chat Application

Custom UI Implementation for Web Search and RSS actions in SUSI iOS Using Kingfisher for Image Caching

The SUSI Server is an AI powered server which is capable of responding to intelligent answers based on user’s queries. The queries to the susi server are obtained either as a websearch using the application or as an RSS feed. Two of the actions are websearch and RSS. These actions as the name suggests respond to queries based on search results from the web which are rendered in the clients. In order to use use these action types and display them in the SUSI iOS client, we need to first parse the actions looking for these action types and then creating a custom UI for them to display them.

To start with, we need to make send the query to the server to receive an intelligent response from the server. This response is parsed into different action types supported by the server and saved into relevant objects. Here, we check the action types by looping through the answers array containing the actions and based on that, we save the data for that action.

if type == ActionType.rss.rawValue {
   message.actionType = ActionType.rss.rawValue
   message.rssData = RSSAction(data: data, actionObject: action)
} else if type == ActionType.websearch.rawValue {
   message.actionType = ActionType.websearch.rawValue
   message.message = action[Client.ChatKeys.Query] as? String ?? ""
}

Here, we parsed the data response from the server and looked for the rss and websearch action type followed by which we saved the data we received from the server for each of the action types in their own objects.

Next, when a message object is created, we insert it into the dataSource item by appending it and use the `insertItems(at: [IndexPath])` method of collection view to insert them into the views at a particular index. Before adding them, we need to create a Custom UI for them. This UI will consist of a Collection View which is scrollable in the horizontal direction inside a CollectionView Cell. To start with this, we create a new class called `WebsearchCollectionView` which will be a `UIView` consisting of a `UICollectionView`.

We start by adding a collection view into the UIView inside the `init` method by overriding it. Declare a collection view using flow layout and scroll direction set to `horizontal`. Also, hide the scroll indicators and assign the delegate and datasource to `self`.

Now to populate this collection view, we need to specify the number of items that will show up. For this, we make use of the `message` variable declared. We use the `websearchData` in case of websearch action and `rssData` otherwise.

Now to specify the number of cells, we use the below method which returns the number of rss or websearch action objects and defaults to 0 such cells.

func collectionView(_ collectionView: UICollectionView, numberOfItemsInSection section: Int) -> Int {
    if let rssData = message?.rssData {
        return rssData.count
    } else if let webData = message?.websearchData {
        return webData.count
    }
    return 0
}

We display the title, description and image for each object for which we need to create a UI for the cells. Let’s start by creating a Custom Collection View cell with the imageview and 2 labels for title and description.

The imageview is given a contentMode of `aspectFit` and assigned a placeholder image in case the image doesn’t exist. The title and description labels are assigned the same font size, the title being bolder and both are center aligned.

class WebsearchCell: BaseCell {

   var imageView: UIImageView = {
       let iv = UIImageView()
       iv.contentMode = .scaleAspectFit
       iv.image = UIImage(named: "placeholder")
       return iv
   }()

   let titleLabel: UILabel = {
       let label = UILabel()
       label.textColor = .black
       label.font = UIFont.boldSystemFont(ofSize: 14)
       label.textAlignment = .center
       label.numberOfLines = 2
       label.backgroundColor = Color.grey.lighten3
       return label
   }()

   let descriptionLabel: UILabel = {
       let label = UILabel()
       label.font = UIFont.systemFont(ofSize: 14)
       label.textAlignment = .center
       return label
   }()

}

Next, we add constraints for each such view adding a title and description label adding a small margin on both sides of the cell for a cleaner UI.

addSubview(imageView)
addSubview(titleLabel)
addSubview(descriptionLabel)
descriptionLabel.numberOfLines = 5
addConstraintsWithFormat(format: "H:|-4-[v0(\(frame.width * 0.4))]-4-[v1]-4-|", views: imageView, titleLabel)
addConstraintsWithFormat(format: "|-\(frame.width * 0.4 + 8)-[v0]-4-|", views: descriptionLabel)
addConstraintsWithFormat(format: "V:|-4-[v0]-4-|", views: imageView)
addConstraintsWithFormat(format: "V:|-4-[v0(44)]-4-[v1]-4-|", views: titleLabel, descriptionLabel)

Now to use this custom cell, we first need to register it with the collection view and then we can use it easily in the `cellForItemAt` method. Since we are using Kingfisher for image caching, we use the `.kf.setImage` method to download the image from a URL and cache it as soon as it is downloaded.

collectionView.register(WebsearchCell.self, forCellWithReuseIdentifier: cellId)
 func collectionView(_ collectionView: UICollectionView, cellForItemAt indexPath: IndexPath) -> UICollectionViewCell {
       if let cell = collectionView.dequeueReusableCell(withReuseIdentifier: cellId, for: indexPath) as? WebsearchCell {
           cell.backgroundColor = .white

           if message?.actionType == ActionType.rss.rawValue {
               let feed = message?.rssData?.rssFeed[indexPath.item]
               cell.titleLabel.text = feed?.title
               cell.descriptionLabel.text = feed?.desc                cell.imageView.kf.setImage(with: URL(string: feed?.rssData?.image))
           } else if message?.actionType == ActionType.websearch.rawValue {
               let webData = message?.websearchData[indexPath.item]
               cell.titleLabel.text = webData?.title
               cell.descriptionLabel.text = webData?.desc.html2String                cell.imageView.kf.setImage(with: URL(string: feed?.webData?.image))
           }
           return cell
       } else {
           return UICollectionViewCell()
       }
   }

We check the action type and assign data based on that. Also, for the images, if they don’t exist a placeholder is added.

Since we are done with the Custom UI of the cell, we need to add it to the chat cell. For that, we add this `UIView` as a subview. For reasons of reusability, the cell was extracted into a separate one and call the `prepareForReuse()` method for reusability. This is followed by adding a subview and setting constraints for each cell and assigning the message object.

class RSSCell: ChatMessageCell {

   var message: Message? {
       didSet {
           self.addWebsearchView()
       }
   }

   lazy var websearchView: WebsearchCollectionView = {
       let view = WebsearchCollectionView()
       return view
   }()

   override func setupViews() {
       super.setupViews()
       prepareForReuse()
   }

   func addWebsearchView() {
       self.addSubview(websearchView)
       self.addConstraintsWithFormat(format: "H:|[v0]|", views: websearchView)
       self.addConstraintsWithFormat(format: "V:[v0(135)]", views: websearchView)
       websearchView.message = message
   }

}
if message.actionType == ActionType.rss.rawValue || message.actionType == ActionType.websearch.rawValue {
   if let cell = collectionView.dequeueReusableCell(withReuseIdentifier: ControllerConstants.rssCell, for: indexPath) as? RSSCell {
       cell.message = message
       return cell
   } else {
       return UICollectionViewCell()
   }
}

This is all we need to add a custom UI to a chat cell in SUSI iOS, very simple and clean.

Check the screenshot below, the app in action.

Resources:

Continue ReadingCustom UI Implementation for Web Search and RSS actions in SUSI iOS Using Kingfisher for Image Caching

Hotword Recognition in SUSI iOS

Hot word recognition is a feature by which a specific action can be performed each time a specific word is spoken. There is a service called Snowboy which helps us achieve this for various clients (for ex: iOS, Android, Raspberry pi, etc.). It is basically a DNN based hotword recognition toolkit.

In this blog, we will learn how to integrate the snowboy hotword detection wrapper in the SUSI iOS client. This service can be used in any open source project but for using it commercially, a commercial license needs to be obtained.

Following are the files that need to be added to the project which are provided by the service itself: snowboy-detect.h libsnowboy-detect.a and a trained model file which can be created using their online service: snowboy.kitt.ai. For the sake of this blog, we will be using the hotword “Susi”, the model file can be found here.

The way how snowboy works is that speech is recorded for a few seconds and this data is detected with an already trained model by a specific hotword, now if snowboy returns a 1 means word has been successfully detected else wasn’t.

We start with creation of a wrapper class in Objective-C which can be found wrapper and the bridging header in case this needs to be added to a Swift project. The wrapper contains methods for setting sensitivity, audio gain and running the detection using the buffer. It is a wrapper class built on top of the snowboy-detect.h header file.

Let’s initialize the service and run it. Below are the steps followed to enable hotword recognition and print out whether it successfully detected the hotword or not:

  • Create a ViewController class with extensions
    • AVAudioRecorderDelegate
    • AVAudioPlayerDelegate

since we will be recording speech.

  • Import AVFoundation
  • Create a basic layout containing a label which detects whether hotword detected or not and create corresponding `IBOutlet` in the ViewController and a button to trigger the start and stop of recognition.
  • Create the following variables:
    • let WAKE_WORD = “Susi” // hotword used
    • let RESOURCE = Bundle.main.path(forResource: “common”, ofType: “res”)
    • let MODEL = Bundle.main.path(forResource: “susi”, ofType: “umdl”) //path where the model file is stored
    • var wrapper: SnowboyWrapper! = nil // wrapper instance for running detection
    • var audioRecorder: AVAudioRecorder! // audio recorder instance
    • var audioPlayer: AVAudioPlayer!
    • var soundFileURL: URL! //stores the URL of the temp reording file
    • var timer: Timer! //timer to fire a function after an interval
    • var isStarted = false // variable to check if audio recorder already started
  • In `viewDidLoad` initialize the wrapper and set sensitivity and audio gain. Recognition best happens when sensitivity is set to `0.5` and audio gain is set to `1.0` according to the docs.
override func viewDidLoad() {
    super.viewDidLoad()
    wrapper = SnowboyWrapper(resources: RESOURCE, modelStr: MODEL)
    wrapper.setSensitivity("0.5")
    wrapper.setAudioGain(1.0)
}
  • Create an `IBAction` for the button to start recognition. This action will be used to start or stop the recording in which the action toggles based on the `isStarted` variable. When true, recording is stopped and the timer invalidated else a timer is started which calls the `startRecording` method with an interval of 4 seconds.
@IBAction func onClickBtn(_ sender: Any) {
  if (isStarted) {
    stopRecording()
    timer.invalidate()
    btn.setTitle("Start", for: .normal)
    isStarted = false
  } else {
    timer = Timer.scheduledTimer(timeInterval: 4, target: self, 
    selector: #selector(startRecording), userInfo: nil, repeats: true)
    timer.fire()
    btn.setTitle("Stop", for: .normal)
    isStarted = true
  }
}
  • Next, we add the start and stop recording methods.
    • First, a temp file is created which stores the recorded audio output
    • After which, necessary record configurations are made such as setting the sampling rate.
    • The recording is then started and the output stored in the temp file.
func startRecording() {
  do {
    let fileMgr = FileManager.default
    let dirPaths = fileMgr.urls(for: .documentDirectory, in: .userDomainMask)
    soundFileURL = dirPaths[0].appendingPathComponent("temp.wav")
    let recordSettings = [AVEncoderAudioQualityKey: 
    AVAudioQuality.high.rawValue,
    AVEncoderBitRateKey: 128000,
    AVNumberOfChannelsKey: 1,
    AVSampleRateKey: 16000.0] as [String : Any]
    let audioSession = AVAudioSession.sharedInstance()
    try audioSession.setCategory(AVAudioSessionCategoryRecord)
    try audioRecorder = AVAudioRecorder(url: soundFileURL,
settings: recordSettings as [String : AnyObject])
    audioRecorder.delegate = self
    audioRecorder.prepareToRecord()
    audioRecorder.record(forDuration: 2.0)
    instructionLabel.text = "Speak wake word: \(WAKE_WORD)"print("Started recording...")
  } catch let error {
    print("Audio session error: \(error.localizedDescription)")
  }
}
  • The stop recording method, stops the audioRecorder instance and updates the instruction label to show the same.
func stopRecording() {
  if (audioRecorder != nil && audioRecorder.isRecording) {
    audioRecorder.stop()
  }
  instructionLabel.text = "Stop"
  print("Stopped recording...")
}

The final recognition is done in the `audioRecorderDidFinishRecording` delegate method which runs the snowboy detection function which processes the audio recording in the temp file by creating a buffer and storing the audio in it and giving the wrapper this buffer as input which processes the buffer and returning a `1` is hotword was successfully detected.

func runSnowboy() {
  let file = try! AVAudioFile(forReading: soundFileURL)
  let format = AVAudioFormat(commonFormat: .pcmFormatFloat32, 
  sampleRate: 16000.0, channels: 1, interleaved: false)
  let buffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: AVAudioFrameCount(file.length))
  try! file.read(into: buffer)
  let array = Array(UnsafeBufferPointer(start: 
  buffer.floatChannelData![0], count:Int(buffer.frameLength)))
  // print output
  let result = wrapper.runDetection(array, length: Int32(buffer.frameLength))
  print("Result: \(result)")
}

To test this out, click the start button and speak different words and you will notice that once the Hot Word is spoken, log with `result: 1` is printed out.

The snowboy hotword recognition also offers to train the personalized model with the help of Rest Apis for which the docs can be found here. The complete project implementation can be found here.

Sources:

Continue ReadingHotword Recognition in SUSI iOS

Developing MultiLinePlotter App for Loklak

MultiLinePlotter is a web application which uses Loklak API under the hood to plot multiple tweet aggregations related to different user provided query words in the same graph. The user can give several query words and multiple lines for different queries will be plotted in the same graph. In this way, users will be able to compare tweet distribution for various keywords and visualise the comparison. All the searched queries are shown under the search record section. Clicking on a record causes a dialogue box to pop up where the individual tweets related to the query word is displayed. Users can also remove a series from the plot dynamically by just pressing the Remove button beside the query word in record section. The app is presently hosted on Loklak apps site.

Related issue – https://github.com/fossasia/apps.loklak.org/issues/225

Getting started with the app

Let us delve into the working of the app. The app uses Loklak aggregation API to get the data.

A call to the API looks something like this:

http://api.loklak.org/api/search.json?q=fossasia&source=cache&count=0&fields=created_at

A small snippet of the aggregation returned by the above API request is shown below.

"aggregations": {"created_at": {
    "2017-07-03": 3,
    "2017-07-04": 9,
    "2017-07-05": 12,
    "2017-07-06": 8,
}}

The API provides a nice date v/s number of tweets aggregation. Now we need to plot this. For plotting Morris.js has been used. It is a lightweight javascript library for visualising data.

One of the main features of this app is addition and removal of multiple series from the graph dynamically. How do we achieve that? Well, this can be achieved by manipulating the morris.js data list whenever a new query is made. Let us understand this in steps.

At first, the data is fetched using angular HTTP service.

$http.jsonp('http://api.loklak.org/api/search.json?callback=JSON_CALLBACK',
            {params: {q: $scope.tweet, source: 'cache', count: '0', fields: 'created_at'}})
                .then(function (response) {
                    $scope.getData(response.data.aggregations.created_at);
                    $scope.plotData();
                    $scope.queryRecords.push($scope.tweet);
                });

Once we get the data, getData function is called and the aggregation data is passed to it. The query word is also stored in queryRecords list for future use.

In order to plot a line graph morris.js requires a data object which will contain the required values for a series. Given below is an example of such a data object.

data: [
    { x: '2006', a: 100, b: 90 },
    { x: '2007', a: 75,  b: 65 },
    { x: '2008', a: 50,  b: 40 },
    { x: '2009', a: 75,  b: 65 },
],

For every ‘x’, ‘a’ and ‘b’ will be plotted. Thus two lines will be drawn. Our app will also maintain a data list like the one shown above, however, in our case, the data objects will have a variable number of keys. One key will determine the ‘x’ value and other keys will determine the ordinates (number of tweets).

All the data objects present in the data list needs to be updated whenever a new search is done.

The getData function does this for us.

var value = $scope.tweet;
        for (date in aggregations) {
            var present = false;
            for (var i = 0; i < $scope.data.length; i++) {
                var item = $scope.data[i];
                if (item['day'] === date) {
                    item[[value]] = aggregations[date];
                    $scope.data[i] = item
                    present = true;
                    break;
                }
            }
            if (!present) {
                $scope.data.push({day: date, [value]: aggregations[date]});
            }
        }


The for loop in the above code snippet updates the global data list used by morris.js. It simply iterates over the dates in the aggregation, extracts the object corresponding to a particular date, adds the new query word as a key and, the number of tweets on that date as the value.If a date is not already present in the list, then it inserts a new object corresponding to the date and query word. Once our data list is updated, we are ready to redraw the graph with the updated data. This is done using plotData function. The plotData function simply checks the user selected graph type. If the selected type is aggregations then it calls plotAggregationGraph() to redraw the aggregations plot.

$scope.remove = function(record) {
        $scope.queryRecords = $scope.queryRecords.filter(function(e) {
            return e !== record });

        $scope.data.forEach(function(item) {
            delete item[record];
        });

        $scope.data = $scope.data.filter(function(item) {
            return Object.keys(item).length !== 1;
        });

        $scope.ykeys = $scope.ykeys.filter(function(item) {
            return item !== record;
        });

        $scope.labels = $scope.labels.filter(function(item) {
            return item !== record;
        });

        $scope.plotData();
}

The above function simply scans the data list, filters the objects which contains selected record as a key and removes them using filter method of javascript arrays. It also removes the corresponding labels and entries from labels and ykeys arrays. Finally, it once again calls plotData function to redraw the plot.

Given below is a sample plot generated by this app with the query words – google, android, microsoft, samsung.

 

Conclusion

This blog post explained how multiple series are plotted dynamically in the MultiLinePlotter app. Apart from aggregations plot it also plots tweet statistics like maximum tweets and average tweets containing a query word and visualises them using stacked bar chart. I will be discussing about them in my subsequent blogs.

Important resources

Continue ReadingDeveloping MultiLinePlotter App for Loklak

Download SUSI.AI Setting Files from Data Folder

In this blog, I will discuss how the DownloadDataSettings servlet hosted on SUSI server functions. This post also covers a step by step demonstration on how to use this feature if you have hosted your own custom SUSI server and have admin rights to it. Given below is the endpoint where the request to download a particular file has to be made.

/data/settings

For systematic functionality and workflow, Users with admin login, are given a special access. This allows them to download the settings files and go through them easily when needed. There are various files which have email ids of registered users (accounting.json), user roles associated to them (authorization.json), groups they are a part of (groups.json) etc. To list all the files in the folder, use the given below end point:

/aaa/listSettings.json

How does the above servlet works? Prior to that, let us see how to to get admin rights on your custom SUSI.AI server.
For admin login, it is required that you have access to files and folders on server. Signup with an account and browse to

/data/settings/authorization.json

Find the email id with which you signed up for admin login and change userRole to “admin”. For example,

{
	"email:test@test.com": {
		"permissions": {},
		"userRole": "user"
	}
}

If you have signed up with an email id “test@test.com” and want to give admin access to it, modify the userRole to “admin”. See below.

{
	"email:test@test.com": {
		"permissions": {},
		"userRole": "admin"
	}
}

Till now, server did not have any email id with admin login or user role equal to admin. Hence, this exercise is required only for the first admin. Later admins can use changeUserRole application and give/change/modify user roles for any of the users registered. By now you must have admin login session. Let’s see now how the download and file listing servlets work.
First, the server creates a path by locally referencing settings folder with the help of DAO.data_dir.getPath(). This will give a string path to the data directory containing all the data-settings files. Now the server just has to make a JSONArray and has to pass a String array to JSONArray’s constructor, which will eventually be containing the name of all the data/settings files. If the process is not successfull ,then, “accepted” = false will be sent as an error to the user. The base user role to access the servlet is ADMIN as only admins are allowed to download data/setting files,
The file name which you have to download has to be sent in a HTTP request as a get parameter. For example, if an admin has to download accounting.json to get the list of all the registered users, the request is to be made in the following way:

BASE_URL+/data/settings?file=file_name

*BASE_URL is the URL where the server is hosted. For standard server, use BASE_URL = http://api.susi.ai.

In the initial steps, Server generates a path to data/settings folder and finds the file, name of which it receives in the request. If no filename is specified in the API call, by default, the server sends accounting.json file.

File settings = new File(DAO.data_dir.getPath()+"/settings");
String filePath = settings.getPath(); 
String fileName = post.get("file","accounting"); 
filePath += "/"+fileName+".json";

Next, the server will extract the file and using ServletOutputStream class, it will generate properties for it and set appropriate context for it. This context will, in turn, fetch the mime type for the file generated. If the mime type is returned as null, by default, mime type for the file will be set to application/octet-stream. For more information on mime type, please look at the following link. A complete list of mime types is compiled and documented here.

response.setContentType(mimetype);
response.setContentLength((int)file.length());

In the above code snippet, mime type and length of the file being downloaded is set. Next, we set the headers for the download response and use filename for that.

response.setHeader("Content-Disposition", "attachment; filename=" + fileName +".json");

All the manual work is done by now. The only thing left is to open a buffer stream, size of which has been defined as a class variable.
Here we use a byte array of size 4096 elements and write the file to client’s default download storage.

private static final int BUFSIZE = 4096;
byte[] byteBuffer = new byte[BUFSIZE];
             DataInputStream in = new DataInputStream(new FileInputStream(file));
            while ((in != null) && ((length = in.read(byteBuffer)) != -1))
            {
                outStream.write(byteBuffer,0,length);
            }

            in.close();
            outStream.close();

All the above-mentioned steps are enclosed in a try-catch block, which catches an exception if any ,and logs it in the log file. This message is also sent to the client for appropriate user information along with the success or failure indication through a boolean flag. Do not forget to close the input and output buffers as it may lead to memory leaks and someone with proper knowledge of network and buffer stream would be able to steal any essential or secured data.

Additional Resources

Continue ReadingDownload SUSI.AI Setting Files from Data Folder