Implementing Stickers on an Image in Phimpme

One feature we implemented in the Phimpme photo editing application is to enable users to add stickers on top of the image. In this blog, I will explain how stickers are implemented in Phimpme.

Features of Stickers

  • Users can resize the stickers.
  • Users can place the stickers wherever in the canvas.

Step.1-Storing the Stickers in assets folder

In Phimpme I stored the stickers in the assets folder. To distribute the stickers in different categories I made different folders according to the categories namely type1, type2, type3, type4 and so on.  

Displaying stickers

We used onBindViewHolder to Display the stickers in different categories like:

  • Facial
  • Express
  • Objects
  • Comments
  • Wishes
  • Emojis
  • Hashtags

String path will get the position of the particular type of stickers collection. This type is then loaded to the ImageLoader with the specific icon associating with the types.   

@Override
public void onBindViewHolder(mRecyclerAdapter.mViewHolder holder, final int position) {

   String path = pathList.get(position);
       ImageLoader.getInstance().displayImage("assets://" + path,holder.icon, imageOption);
       holder.itemView.setTag(path);
       holder.title.setText("");

   int size = (int) getActivity().getResources().getDimension(R.dimen.icon_item_image_size_filter_preview);
   LinearLayout.LayoutParams layoutParams = new LinearLayout.LayoutParams(size,size);
   holder.icon.setLayoutParams(layoutParams);

   holder.itemView.setOnClickListener(new View.OnClickListener() {
       @Override
       public void onClick(View v) {
           String data = (String) v.getTag();
           selectedStickerItem(data);
       }
   });
}

Step.2- Applying a sticker on the image

When a particular sticker is selected selectedStickerItem() function is called.This function calls StickerView class to add the Bitmap on the image. It sends the path of the sticker as a parameter.  

public void selectedStickerItem(String path) {
   mStickerView.addBitImage(getImageFromAssetsFile(path));
}

In StickerView class the image of the sticker is then converted into a Bitmap. It creates an object(item) of StickerItem class. This object calls the init function, which handles the size of the sticker and the placement of the sticker on the image.

public void addBitImage(final Bitmap addBit) {
   StickerItem item = new StickerItem(this.getContext());
   item.init(addBit, this);
   if (currentItem != null) {
       currentItem.isDrawHelpTool = false;
   }
   bank.put(++imageCount, item);
   this.invalidate();
}

Step.3-Resizing the Sticker in the canvas

A bitmap or any image has two axes namely x and y. We can resize the image using matrix calculation.

float c_x = dstRect.centerX();
float c_y = dstRect.centerY();

float x = this.detectRotateRect.centerX();
float y = this.detectRotateRect.centerY();

We then calculate the source length and the current length:

float srcLen = (float) Math.sqrt(xa * xa + ya * ya);
float curLen = (float) Math.sqrt(xb * xb + yb * yb);

Then we calculate the scale. This is required to calculate the zoom ratio.

float scale = curLen / srcLen;

We need to rescale the bitmap. That is if the user rotates the sticker or zoom in or zoom out the sticker. A helpbox surrounds the stickers showing the actual size of the sticker. This helpbox which is rectangular shape helps in resizing the sticker.

RectUtil.scaleRect(this.dstRect, scale);// Zoom destination rectangle

// Recalculate the Toolbox coordinates
helpBox.set(dstRect);
updateHelpBoxRect();// Recalculate
rotateRect.offsetTo(helpBox.right - BUTTON_WIDTH, helpBox.bottom
       - BUTTON_WIDTH);
deleteRect.offsetTo(helpBox.left - BUTTON_WIDTH, helpBox.top
       - BUTTON_WIDTH);

detectRotateRect.offsetTo(helpBox.right - BUTTON_WIDTH, helpBox.bottom
       - BUTTON_WIDTH);
detectDeleteRect.offsetTo(helpBox.left - BUTTON_WIDTH, helpBox.top
       - BUTTON_WIDTH);

Conclusion

In Phimpme a user can now place the sticker on top of the image. Resize the sticker, that is Zoom in the image or zoom out of the image. Move the image around the canvas. This will give users the flexibility to add multiple stickers on the image.

Github

Resources

Continue ReadingImplementing Stickers on an Image in Phimpme

Share Images on Pinterest from Phimpme Android Application

After successfully establishing Pinterest authentication in Phimpme our next goal was to share the image on the Pinterest website directly from Phimpme, without using any native Android application.

Adding Pinterest Sharing option in Sharing Activity in Phimpme

To add various sharing options in Sharing Activity in the Phimpme project, I have applied a ScrollView for the list of the different sharing options which include: Facebook, Twitter, Pinterest, Imgur, Flickr and Instagram. All the App icons with the name are arranged in a TableLayout in the activity_share.xml file. Table rows consist of two columns. In this way, it is easier to add more app icons for future development.

<ScrollView
   android:layout_width="wrap_content"
   android:layout_height="@dimen/scroll_view_height"
   android:layout_above="@+id/share_done"
   android:id="@+id/bottom_view">
   <LinearLayout
       android:layout_width="wrap_content"
       android:layout_height="wrap_content"
       android:orientation="vertical">
       <TableLayout

Adding Pinterest app icon on the icons_drawable array. This array is then used to inflate the icon on the list view.

private int[] icons_drawables = {R.drawable.ic_facebook_black, R.drawable.ic_twitter_black,R.drawable.ic_instagram_black, R.drawable.ic_wordpress_black, R.drawable.ic_pinterest_black);

Adding Pinterest text on the titles_text array. This array is then used to inflate the names of the various sharing activity.

private int[] titles_text = {R.string.facebook, R.string.twitter, R.string.instagram,
       R.string.wordpress, R.string.pinterest);

Prerequisites to share Image on Pinterest

To share an Image on Pinterest a user has to add a caption and Board ID. Our first milestone was to get the input of the Board ID  by the user. I have achieved this by taking the input in a Dialog Box. When the user clicks on the Pinterest option, a dialog box pops and then the user can add their Board ID.

private void openPinterestDialogBox() {
   AlertDialog.Builder captionDialogBuilder = new AlertDialog.Builder(SharingActivity.this, getDialogStyle());
   final EditText captionEditText = getCaptionDialog(this, captionDialogBuilder);

   captionEditText.setHint(R.string.hint_boardID);

   captionDialogBuilder.setNegativeButton(getString(R.string.cancel).toUpperCase(), null);
   captionDialogBuilder.setPositiveButton(getString(R.string.post_action).toUpperCase(), new DialogInterface.OnClickListener() {
       @Override
       public void onClick(DialogInterface dialog, int which) {
           //This should be empty it will be overwrite later
           //to avoid dismiss of the dialog on the wrong password
       }
   });

   final AlertDialog passwordDialog = captionDialogBuilder.create();
   passwordDialog.show();

   passwordDialog.getButton(AlertDialog.BUTTON_POSITIVE).setOnClickListener(new View.OnClickListener() {
       @Override
       public void onClick(View v) {
           String captionText = captionEditText.getText().toString();
           boardID =captionText;
           shareToPinterest(boardID);
           passwordDialog.dismiss();
       }
   });
}

A user can fetch the Board ID by following the steps:

Board ID is necessary because it specifies where the image needs to be posted.

Creating custom post function for Phimpme

The image is posted using a function in PDKClient class. PDKClient is found in the PDK module which we get after importing Pinterest SDK. Every image is posted on Pinterest is called a Pin. So we will call createPin function. I have made my custom createPin function so that it also accepts Bitmaps as a parameter. In the Pinterest SDK it only accepts image URL to share, The image should already be on the internet to be shared. For this reason, we to add a custom create Pin function to accept Bitmaps as an option.

public void createPin(String note, String boardId, Bitmap image, String link, PDKCallback callback) {
   if (Utils.isEmpty(note) || Utils.isEmpty(boardId) || image == null) {
       if (callback != null) callback.onFailure(new PDKException("Board Id, note, Image cannot be empty"));
       return;
   }

   HashMap<String, String> params = new HashMap<String, String>();
   params.put("board", boardId);
   params.put("note", note);
   params.put("image_base64", Utils.base64String(image));
   if (!Utils.isEmpty(link)) params.put("link", link);
   postPath(PINS, params, callback);
}

Compressing Bitmaps

Since Pinterest SDK cannot accept Bitmap I have added a function to compress the Bitmap and encode it to strings.

public static String base64String(Bitmap bitmap) {
   ByteArrayOutputStream baos = new ByteArrayOutputStream();
   bitmap.compress(Bitmap.CompressFormat.JPEG, 100, baos);
   String b64Str = Base64.encodeToString(baos.toByteArray(), Base64.NO_WRAP);
   return b64Str;
}

Calling createPin function from the sharingActivity

From the sharingActivity we will call createPin activity. We will pass caption of the image, Board ID, Image bitmap and link(which is optional) as parameters.

PDKClient
       .getInstance().createPin(caption, boardID, image, null, new PDKCallback() {

If the image is posted successfully then onSuccess function is called which pops a snackbar and shows the success message. Otherwise, onFailure function is called which displays failure message on a snackbar.

@Override
public void onSuccess(PDKResponse response) {
   Log.d(getClass().getName(), response.getData().toString());
   Snackbar.make(parent, R.string.pinterest_post, Snackbar.LENGTH_LONG).show();
   //Toast.makeText(SharingActivity.this,message,Toast.LENGTH_SHORT).show();

}

@Override
public void onFailure(PDKException exception) {
   Log.e(getClass().getName(), exception.getDetailMessage());
   Snackbar.make(parent, R.string.Pinterest_fail, Snackbar.LENGTH_LONG).show();
   //Toast.makeText(SharingActivity.this, boardID + caption, Toast.LENGTH_SHORT).show();

}

Conclusion

In Phimpme user can now send Image on Pinterest directly from the application. This is done without the use of the native Pinterest application.

Github

Resources

Continue ReadingShare Images on Pinterest from Phimpme Android Application

Adding Sentry Integration in Open Event Orga Android App

Sentry is a service that allows you to track events, issues and crashes in your apps and provide deep insights with context about them. This blog post will discuss how we implemented it in Open Event Orga App (Github Repo).

Configuration

First, we need to include the gradle dependency in build.gradle
compile ‘io.sentry:sentry-android:1.3.0’
Now, our project uses proguard for release builds which obfuscates the code and removes unnecessary class to shrink the app. For the crash events to make sense in Sentry dashboard, we need proguard mappings to be uploaded every time release build is generated. Thankfully, this is automatically handled by sentry through its gradle plugin, so to include it, we add this in our project level build.gradle in dependencies block

classpath 'io.sentry:sentry-android-gradle-plugin:1.3.0'

 

And then apply the plugin by writing this at top of our app/build.gradle

apply plugin: 'io.sentry.android.gradle'

 

And then configure the options for automatic proguard configuration and mappings upload

sentry {
   // Disables or enables the automatic configuration of proguard
   // for Sentry.  This injects a default config for proguard so
   // you don't need to do it manually.
   autoProguardConfig true

   // Enables or disables the automatic upload of mapping files
   // during a build.  If you disable this you'll need to manually
   // upload the mapping files with sentry-cli when you do a release.
   autoUpload false
}

 

We have set the autoUpload to false as we wanted Sentry to be an optional dependency to the project. If we turn it on, the build will crash if sentry can’t find the configuration, which we don’t want to happen.

Now, as we want Sentry to configurable, we need to set Sentry DSN as one of the configuration options. The easiest way to externalize configuration is to use environment variables. There are other methods to do it given in the official documentation for config https://docs.sentry.io/clients/java/config/

Lastly, for proguard configuration, we also need 3 other config options, namely:

defaults.project=your-project
defaults.org=your-organisation
auth.token=your-auth-token

 

For getting the auth token, you need to go to https://sentry.io/api/

Now, the configuration is complete and we’ll move to the code

Implementation

First, we need to initialise the sentry instance for all further actions to be valid. This is to be done when the app starts, so we add it in onCreate method Application class of our project by calling this method

// Sentry DSN must be defined as environment variable
// https://docs.sentry.io/clients/java/config/#setting-the-dsn-data-source-name
Sentry.init(new AndroidSentryClientFactory(getApplicationContext()));

 

Now, we’re all set to send crash reports and other events to our Sentry server. This would have required a lot of refactoring if we didn’t use Timber for logging. We are using default debug tree for debug build and a custom Timber tree for release builds.

if (BuildConfig.DEBUG)
   Timber.plant(new Timber.DebugTree());
else
   Timber.plant(new ReleaseLogTree());

 

The ReleaseLogTree extends Timber.Tree which is an abstract class requiring you to override this function:

@Override
protected void log(int priority, String tag, String message, Throwable throwable) {

 }

 

This function is called whenever there is a log event through Timber and this is where we send reports through Sentry. First, we return from the function if the event priority is debug or verbose

if(priority == Log.DEBUG || priority == Log.VERBOSE)
   return;

 

If the event if if info priority, we attach it to sentry bread crumb

if (priority == Log.INFO) {
    Sentry.getContext().recordBreadcrumb(new BreadcrumbBuilder()
          .setMessage(message)
          .build());
}

 

Breadcrumbs are stored and only send with an event. What event comprises for us is the crash event or something we want to be logged to dashboard whenever the user does it. But since info events are just user interactions throughout the app, we don’t want to crowd the issue dashboard with them. However, we want to understand what user was doing before the crash happened, and that is why we use bread crumbs to store the events and only send them attached to a crash event. Also, only the last 100 bread crumbs are stored, making it easier to parse through them.

Now, if there is an error event, we want to capture and send it to the server

if (priority == Log.ERROR) {
   if (throwable == null)
       Sentry.capture(message);
   else
       Sentry.capture(throwable);
}

 

Lastly, we want to set Sentry context to be user specific so that we can easily track and filter through issues based on the user. For that, we create a new class ContextManager with two methods:

  • setOrganiser: to be called at login
  • clearOrganiser: to be called at logout

public void setOrganiser(User user) {
   Map<String, Object> userData = new HashMap<>();
   userData.put("details", user.getUserDetail());
   userData.put("last_access_time", user.getLastAccessTime());
   userData.put("sign_up_time", user.getSignupTime());

   Timber.i("User logged in - %s", user);
   Sentry.getContext().setUser(
       new UserBuilder()
       .setEmail(user.getEmail())
       .setId(String.valueOf(user.getId()))
       .setData(userData)
       .build()
   );
}

 

In this method, we put all the information about the user in the context so that every action from here on is attached to this user.

public void clearOrganiser() {
   Sentry.clearContext();
}

 

And here, we just clear the sentry context.

This concludes the implementation of our sentry client. Now all Timber log events will through sentry and appropriate events will appear on the sentry dashboard. To read more about sentry features and Timber, visit these links:

Sentry Java Documentation (check Android section)

https://docs.sentry.io/clients/java/

Timber Library

https://github.com/JakeWharton/timber

Continue ReadingAdding Sentry Integration in Open Event Orga Android App

Implementing Attendee Detail BottomSheet UI in Open Event Orga App

In Open Event Orga App (Github Repo), we allow the option to check the attendee details before checking him/her in or out. Originally, a dialog was shown showing the attendee details, which did not contain much information about the attendee, ticket or the order. The disadvantage of such design was also that it was tied to only one view. We couldn’t show the check in dialog elsewhere in the app, like during QR scanning. So we had to switch back to the attendee view for showing the check in dialog. We decided to create a usable detached component in the form of a bottom sheet containing all required information. This blog will outline the procedure we employed to design the bottom sheet UI.

The attendee check in dialog looked like this:

So, first we decide what we need to show on the check in bottom sheet:

  • Attendee Name
  • Attendee Email
  • Attendee Check In Status
  • Order Status ( Completed, Pending, etc )
  • TIcket Type ( Free, Paid, Donation )
  • Ticket Price
  • Order Date
  • Invoice Number
  • Order ‘Paid Via’

As we are using Android Data Binding in our layout, we’ll start by including the variables required in the layout. Besides the obvious attendee variable, we need presenter instance to handle the check in and check out of the attendee and DateUtils class to parse the order date. Additionally, to handle the visibility of views, we need to include the View class too

<data>
   <import type="org.fossasia.openevent.app.utils.DateUtils" />
   <import type="android.view.View" />

   <variable
       name="presenter"
       type="org.fossasia.openevent.app.event.checkin.contract.IAttendeeCheckInPresenter" />

   <variable
       name="checkinAttendee"
       type="org.fossasia.openevent.app.data.models.Attendee" />
</data>

 

Then, we make the root layout to be CoordinatorLayout and add a NestedScrollView inside it, which contains a vertical linear layout in it. This vertical linear layout will contain our fields.

Note: For brevity, I’ll skip most of the layout attributes from the blog and only show the ones that correspond to the text

Firstly, we show the attendee name:

<TextView
   style="@style/TextAppearance.AppCompat.Headline"
   android:text='@{attendee.firstName + " " + attendee.lastName }'
   tools:text="Name" />

 

The perks of using data binding can be seen here, as we are using string concatenation in layout itself. Furthermore, data binding also handles null checks for us if we add a question mark at the end of the variable name ( attendee.firstName? ).

But our server ensures that both these fields are not null, so we skip that part.

Next up, we display the attendee email

<TextView
   android:text="@{ checkinAttendee.email }"
   tools:text="xyz@example.com" />

 

And then the check in status of the attendee

<TextView
   android:text="@{ checkinAttendee.checkedIn ? @string/checked_in : @string/checked_out }"
   android:textColor="@{ checkinAttendee.checkedIn ? @color/light_green_500 : @color/red_500 }"
   tools:text="CHECKED IN" />

 

Notice that we dynamically change the color and text based on the check in status of the attendee

Now we begin showing the fields with icons to their left. You can use Compound Drawable to achieve this effect, but we use vector drawables which are incompatible with compound drawables on older versions of Android, so we use a horizontal LinearLayout instead.

The first field is the order status denoting if the order is completed or in transient state

<LinearLayout android:orientation="horizontal">

   <ImageView app:srcCompat="@drawable/ic_transfer" />
   <TextView android:text="@{ checkinAttendee.order.status }" />
</LinearLayout>

 

Now, again for keeping the snippets relevant, I’ll skip the icon portion and only show the text binding from now on.

Next, we include the type of ticket attendee has. There are 3 types of ticket supported in Open Event API – free, paid, donation

<TextView
   android:text="@{ checkinAttendee.ticket.type }"  />

 

Next, we want to show the price of the ticket, but only when the ticket is of paid type.

I’ll include the previously omitted LinearLayout part in this snippet because it is the view we control to hide or show the field

<LinearLayout
   android:visibility='@{ checkinAttendee.ticket.type.equalsIgnoreCase("paid") ? View.VISIBLE : View.GONE }'>

   <ImageView app:srcCompat="@drawable/ic_coin" />
   <TextView
       android:text='@{ "$" + checkinAttendee.ticket.price }'
       tools:text="3.78" />
</LinearLayout>

 

As you can see, we are showing this layout only if the ticket type equals paid

The next part is about showing the date on which the order took place

<TextView
   android:text="@{ DateUtils.formatDateWithDefault(DateUtils.FORMAT_DAY_COMPLETE, checkinAttendee.order.completedAt) }" />

 

Here we are using internal DateUtils method to format the date into complete date time from the ISO 8601 standard date present in the order object

Now, we show the invoice number of the order

<TextView
   android:text="@{ checkinAttendee.order.invoiceNumber }" />

 

Lastly, we want to show how the ticket was paid for via

<LinearLayout
   android:visibility='@{ checkinAttendee.order.paidVia.equalsIgnoreCase("free") ? View.GONE : View.VISIBLE }'>

   <ImageView app:srcCompat="@drawable/ic_ray" />
   <TextView  android:text="@{ checkinAttendee.order.paidVia }" />
</LinearLayout>

 

Notice that here too we are controlling the visibility of the layout container and only showing it if the ticket type is paid

This ends our vertical linear layout showing the fields about attendee detail. Now, we add a floating action button to toggle the check in status of attendee

<FrameLayout
   android:layout_gravity="top|end">

   <android.support.design.widget.FloatingActionButton
       android:layout_gravity="center"
       android:onClick="@{() -> presenter.toggleCheckIn() }"
       app:backgroundTint="@{ checkinAttendee.checkedIn ? @color/red_500 : @color/light_green_500 }"
       app:srcCompat="@{ checkinAttendee.checkedIn ? @drawable/ic_checkout : @drawable/ic_checkin }"
       app:tint="@android:color/white" />

   <ProgressBar
       android:layout_gravity="center" />

</FrameLayout>

 

We have used a FrameLayout to wrap a FAB and progress bar together in top end of the bottom sheet. The progress bar shows the indeterminate progress of the toggling of attendee status. And you can see the click binder on FAB triggering the presenter method toggleCheckIn() and how the background color and icon change according to the check in status of the attendee.

This wraps up our layout design. Now we just have to create a BottomSheetDialogFragment, inflate this layout in it and bind the attendee variable and we are all set. The result with all fields visible looks like this:

To learn more about bottom sheet and android data binding, please refer to these links:

Continue ReadingImplementing Attendee Detail BottomSheet UI in Open Event Orga App

Invalidating user login using JWT in Open Event Orga App

User authentication is an essential part of Open Event Orga App (Github Repo), which allows an organizer to log in and perform actions on the event he/she organizes. Backend for the application, Open Event Orga Server sends an authentication token on successful login, and all subsequent privileged API requests must include this token. The token is a JWT (Javascript Web Token) which includes certain information about the user, such as identifier and information about from when will the token be valid, when will it expire and a signature to verify if it was tampered.

Parsing the Token

Our job was to parse the token to find two fields:

  • Identifier of user
  • Expiry time of the token

We stored the token in our shared preference file and loaded it from there for any subsequent requests. But, the token expires after 24 hours and we needed our login model to clear it once it has expired and shown the login activity instead.

To do this, we needed to parse the JWT and compare the timestamp stored in the exp field with the current timestamp and determine if the token is expired. The first step in the process was to parse the token, which is essentially a Base 64 encoded JSON string with sections separated by periods. The sections are as follows:

  • Header ( Contains information about algorithm used to encode JWT, etc )
  • Payload ( The data in JWT – exp. Iar, nbf, identity, etc )
  • Signature ( Verification signature of JWT )

We were interested in payload and for getting the JSON string from the token, we could have used Android’s Base64 class to decode the token, but we wanted to unit test all the util functions and that is why we opted for a custom Base64 class for only decoding our token.

So, first we split the token by the period and decoded each part and stored it in a SparseArrayCompat

public static SparseArrayCompat<String> decode(String token) {
   SparseArrayCompat<String> decoded = new SparseArrayCompat<>(2);

   String[] split = token.split("\\.");
   decoded.append(0, getJson(split[0]));
   decoded.append(1, getJson(split[1]));

   return decoded;
}

 

The getJson function is primarily decoding the Base64 string

private static String getJson(String strEncoded) {
   byte[] decodedBytes = Base64Utils.decode(strEncoded);
   return new String(decodedBytes);
}

The decoded information was stored in this way

0={"alg":"HS256","typ":"JWT"},  1={"nbf":1495745400,"iat":1495745400,"exp":1495745800,"identity":344}

Extracting Information

Next, we create a function to get the expiry timestamp from the token. We could use GSON or Jackson for the task, but we did not want to map fields into any object. So we simply used JSONObject class which Android provides. It took 5 ms on average to parse the JSON instead of 150 ms by GSON

public static long getExpiry(String token) throws JSONException {
   SparseArrayCompat<String> decoded = decode(token);

   // We are using JSONObject instead of GSON as it takes about 5 ms instead of 150 ms taken by GSON
   return Long.parseLong(new JSONObject(decoded.get(1)).get("exp").toString());
}

 

Next, we wanted to get the ID of user from token to determine if a new user is logging in or an old one, so that we can clear the database for new user.

public static int getIdentity(String token) throws JSONException {
   SparseArrayCompat<String> decoded = decode(token);

   return Integer.parseInt(new JSONObject(decoded.get(1)).get("identity").toString());
}

Validating the token

After this, we needed to create a function that tells if a stored token is expired or not. With all the right functions in place, it was just a matter of comparing current time with the stored timestamp

public static boolean isExpired(String token) {
   long expiry;

   try {
       expiry = getExpiry(token);
   } catch (JSONException jse) {
       return true;
   }

   return System.currentTimeMillis()/1000 >= expiry;
}

 

Since the token provides timestamp from epoch in terms of seconds, we needed to divide the current time in milliseconds by 1000 and the function returned true if current timestamp was greater than the expiry time of token.

After writing a few unit tests for both functions, we just needed to plug them in our login model at the time of authentication.

At the time of starting of the application, we use this function to check if a user is logged in or not:

public boolean isLoggedIn() {
   String token = utilModel.getToken();

   return token != null && !JWTUtils.isExpired(token);
}

 

So, if there is no token or the token is expired, we do not automatically login the user and show the login screen.

Implementing login

The next task were

  • Sequest the server to login
  • Store the acquired token
  • Delete database if it is a new user

Before implementing the above logic, we needed to implement a function to determine if the person logging in is previous user, or new one. For doing so, we first loaded the saved user from our database, if the query is empty, surely it is a new user logging in. So we return false, and if there is a user in the database, we match its ID with the logged in user’s ID:

public Single<Boolean> isPreviousUser(String token) {
   return databaseRepository.getAllItems(User.class)
       .first(EMPTY)
       .map(user -> !user.equals(EMPTY) && user.getId() == JWTUtils.getIdentity(token));
}

 

We have added a default user EMPTY in the first operator so that RxJava returns it if there are no users in the database and then we simply map the user to a boolean denoting if they are same or different using the EMPTY user and getIdentity method from JWTUtils

Finally, we use all this information to implement our self contained login request:

eventService
   .login(new Login(username, password))
   .flatMapSingle(loginResponse -> {
       String token = loginResponse.getAccessToken();
       utilModel.saveToken(token);

       return isPreviousUser(token);
   })
   .flatMapCompletable(isPrevious -> {
       if (!isPrevious)
           return utilModel.deleteDatabase();

       return Completable.complete();
   });

 

Let’s see what is happening here. A request using username and password is made to the server which returns a login response containing a JWT, which we store for future use. Next, we flatMapSingle to the Single returned by the isPreviousUser method. And we finally clear the database if it is not a previous user.

Creating these self contained models help reduce complexity in presenter or view layer and all data is handled in one layer making presenter layer model agnostic.

To learn more about JWT and some of the Rx operators I mentioned here, please visit these links:

Continue ReadingInvalidating user login using JWT in Open Event Orga App

Implementing Text-to-Speech (TTS) in SUSI Android

Mobile assistants are designed to perform tasks that the user “commands” through by chat UI or speech. The Android OS already provides Text to speech (TTS) and Speech to text (STT) features. This feature is available from Android version 1.6 onward. In this blog post I will show how tts is implemented in SUSI Android and how I fix the issue ‘delay in speech response’.

TextToSpeech class controls the tts engine. To use TextToSpeech class import it in the activity where you want to use text to speech feature.

import android.speech.tts.TextToSpeech;

After you import TextToSpeech class now we need to initialize TextToSpeech

TextToSpeech tts = new TextToSpeech(this,this);

Here first parameter is the Context and the other one is the listener. The listener is  use  to  inform our app that the engine is ready to use. In order to be notified we have to  implement  TextToSpeech.OnInitListener.

TextToSpeech.OnInitListener listener = new  TextToSpeech.OnInitListener {
@Override
public void onInit(int status) {
if (status == TextToSpeech.SUCCESS)
tts.setLanguage(Locale.UK/* set the default language*/);
}
}

Hence the engine can be initialized asIf status is success then, it means that TTS is initialized successfully and now we can use it. Otherwise, we can’t use it. setLanguage method is used to set language in which we want reply.

TextToSpeech tts = new TextToSpeech(getApplicationContext,listener)

When you use TTS one thing you have to remember that TTS run  on main thread so sometimes it may cause delays in text to speech conversion or it may block UI for a while. It is better to wrap it like below code.

new Handler().post(new Runnable() {
      @Override
      public void run() {
         tts = new TextToSpeech(getApplicationContext(), listener);
        }
    });

Now our engine is ready to speak, we need simply pass the string we want to read.

tts.speak(text to read,TextToSpeech.QUEUE_FLUSH, null, null);

But before tts.speak, it is important to check for the audio focus change request. It is important because only one audio source can have focus at a time. You can check it using below code.

private AudioManager.OnAudioFocusChangeListener afChangeListener =
           new AudioManager.OnAudioFocusChangeListener() {
                 public void onAudioFocusChange(int focusChange) {
                                                        //check for focus
                                                   }
                                           };

OnAudioFocusChangeListener is called when audio focus of the system is changed and according to value of focusChange either we stop TTS or keep using it.

AudioManager audiofocus = (AudioManager)                                    getSystemService(Context.AUDIO_SERVICE);

audiofocus is instance of AudioManager class. We need it to call requestAudioFocus method of AudioManager class. requestAudioFocus method returns the status of request for audio focus change. This method requires three parameter  instance of AudioManager.OnAudioFocusChangeListener, stream type and duration hint. If request is granted only then we can we can use tts.speak .

int result = audiofocus.requestAudioFocus(afChangeListener,AudioManager.STREAM_MUSIC, AudioManager.AUDIOFOCUS_GAIN);

if (result == AudioManager.AUDIOFOCUS_REQUEST_GRANTED) {

tts.speak(text to read,TextToSpeech.QUEUE_FLUSH, null, null);

}

We were continuously facing issue ‘delay in speech response’ because voiceReply method implementation was wrong. We were initializing TextToSpeech on each call of voiceReply method and since onInit method runs on main thread causing delay in voice response. So I removed it and instead of initializing tts each time I used the tts instance already initialized when activity create.

 String spoken = reply;

textToSpeech.speak(spoken, TextToSpeech.QUEUE_FLUSHnull);

You can also control how the engine read text. Like we can modify pitch and speech rate.

tts.setPitch((float)pitch);

tts.setSpeechRate((float)speed);

Resource

Continue ReadingImplementing Text-to-Speech (TTS) in SUSI Android

Integration of SUSI AI in Twitter

We will be making a Susi messenger bot on Twitter. The messenger bot will tweet back to your tweets and reply instantly when you chat with it. Feel free to tweet to the already made SUSI AI account (mentioning @SusiAI1 in it). Follow it, to have a personal chat.

Make a new account, which you want to use as the bot account. You can make one from sign up option from https://www.twitter.com.

Prerequisites

To create your account on -:
1. Twitter
2. Github
3. Heroku
4. Node js

Setup your own Messenger Bot

1. Make a new app here, to know the access token and other properties for our application. These properties will help us communicate with Twitter.

Click “modify the app permissions” link, as shown here:

Select the Read, Write and Access direct messages option:

Don’t forget to click the update settings button at the bottom.

Click the Generate My Access Token and Token Secret button.

3. Create a new heroku app here.

This app will accept the requests from Twitter and Susi api.

4. Create a config variable by switching to settings page of your app.
  
  The name of your first config variable should be HEROKU_URL and its value is the url address of the heroku app created by you.
 

The other config variables that need to be created will be these:
 

The corresponding names of these variables in the same order are:
  i) Access token
  ii) Access token secret
  iii) Consumer key
  iv) Consumer secret
  
We need to visit our app from here, the keys and access tokens tab will help us with the values of these variables.

  1. Let’s start with the code part of the integration of SUSI AI to Twitter. We will be using Node js to achieve this integration.

First we need to require some packages:

Now using the Twit module, we need to authenticate our requests, by using our environment variables as set up in step 4:

Now let’s make a user stream:

var stream = T.stream('user');

We will be using the capabilities of this stream, to catch events of getting tweeted or receiving a direct message by using:

stream.on('tweet', functionToBeCalledWhenTweeted);
stream.on('follow', functionToBeCalledWhenFollowed);
stream.on('direct_message', functionToBeCalledWhenDirectMessaged);

So, when a person tweets to our account like this:

We can catch it with ‘tweet’ event and execute a set of instructions:

stream.on('tweet', tweetEvent);

    function tweetEvent(eventMsg) {
        var replyto = eventMsg.in_reply_to_screen_name;     

       // to store the message tweeted excluding '@SusiAI1' substring
        var text = eventMsg.text.substring(9);

        // to store the name of the tweeter
        var from = eventMsg.user.screen_name;
        
        if (replyto === 'SusiAI1') {
            var queryUrl = 'http://api.asksusi.com/susi/chat.json?q=' + encodeURI(text);
            var message = '';
            request({
                url: queryUrl,
                json: true
            }, function (err, response, data) {
                if (!err && response.statusCode === 200) {
                    // fetching the answer from the data object returned
                                        message = data.answers[0].actions[0].expression + data;


                }
                else {
                    message = 'Oops, Looks like Susi is taking a break';    
                    console.log(err);
                }
                console.log(message);
                // If the message length is more than tweet limit
                if(message.length > 140){
                    tweetIt('@' + from + ' Sorry due to tweet word limit, I have sent you a personal message. Check inbox'+date);
                    sendMessage(from, message);
                }
                else{
                    tweetIt('@' + from + ' ' + message + date);
                }
            });
        }
    }
  • When we a person follows this SUSI AI account, we can thank him/her by making use of the “follow” event. Also, we need to follow him/her back, to enable personal chat between Susi and that person (according to the rules of twitter):
stream.on('follow',followed);

function followed(eventMsg) {
        console.log('Follow event !');
        var name = eventMsg.source.name;
        var screenName = eventMsg.source.screen_name;
        var user_id1 = eventMsg.source.id_str;

        // To follow back the person.
        T.post('friendships/create', {user_id : user_id1},  function(err, tweets, response){
            if (err) {
                console.log("Couldn't follow back!");
            } 
            else {    
tweetIt('@' + screenName + ' Thank you for following me! I followed you back, you can also direct message me now! ');
                console.log("Followed back!");
            } 
        }); 
    }

When we personally message this SUSI AI account

This can be handled through the ‘direct_message’ event:

stream.on('direct_message', reply);
    function reply(directMsg) {
        console.log('You receive a message!');
        // If its our own bot messaging, ignore it, as this can lead to an infinite loop when we answer a user.
        if (directMsg.direct_message.sender_screen_name === 'SusiAI1') {
            return;
        }

        // code to fetch the reply from SUSI API
        
        // reply the user with the SUSI API's message
        sendMessage(directMsg.direct_message.sender_screen_name, message);
        });
    }
  • The tweetIt and sendMessage function code can be seen from the repo code.

6. Connect the heroku app to the forked repository.

Connect the app to Github by selecting the name of this forked repository.

7. Deploy on development branch. If you intend to contribute, it is recommended to Enable Automatic Deploys.

Branch Deployment.

Successful Deployment.

  1. Visit your own personal account and tweet to this new bot account with your query and enjoy a tweet back from the bot account! Also, you can enjoy personal chatting with Susi.

    Feel free to play around with the already made SUSI AI account on twitter here. Follow it, to have a personal chat with it.

    Resources
Continue ReadingIntegration of SUSI AI in Twitter

Integration of Susi AI to Gitter

This blog post discusses the development of Susi Messenger bot on Gitter. It replies instantly to the messages sent to it, using the Susi API. The Streaming API notifies us when a user messages to the SUSI chat room. The REST API helps to message back with a reply from SUSI API, to the SUSI chat room.

Feel free to message to the already made SUSI AI account on Gitter and have a chat with it.

Prerequisites

  1. Basic knowledge about calling API’s and fetching data or posting data to the API.
  2. Node.js language.
  3. Github
  4. Heroku

Figure – Architecture for running SUSI AI on different messaging services.

This blog post will walk you through each of the steps required to integrate SUSI AI to Gitter:

Setup SUSI AI Bot on Gitter

  1. Create a Github or twitter account with a username having ‘Susi’ as its substring because this is the name that will be shown with the reply string we will get from Susi AI.
  2. Now you need to sign in to Gitter with a twitter or Github account from here.
  3. Create your community by visiting this page. After writing your community name press next, invite the people you want to be in this room and press next. You will be redirected to your communities lobby. This lobby is the chat room to which we will deploy our SUSI AI.
  4. Now visit the Gitter developer page, press sign in on the top right. You will be redirected to your apps page. Copy the personal access token written there as shown in this image: (The area colored black will have your access token).

5. On a new tab, in your browser visit   https://api.gitter.im/v1/rooms?access_token=YOUR_ACCESS_TOKEN, with YOUR_ACCESS_TOKEN replaced by the token we just copied.

A JSON object will be shown on our browser screen. You will see the value of ‘name’ key as YOUR_COMMUNITY_NAME/Lobby. Copy the id of this chat room, as we will need it later. You can refer to the image below, you will have your chat room id in the area colored black.

  1. Create a new heroku app here.

This app will accept the requests from Gitter and Susi api.

  1. Set the config variables for this heroku app in the setting tab of your account. Set ROOM_ID to the id of the chat room and TOKEN to the personal access token, we copied in steps 4 and 5.

These were the formalities to be done to have our chat bot account on Gitter.

  1. Let’s jump to the code part of how this integration will be done:

To use the two config variables set in Heroku, we need these two lines in our Node js code:

var roomId = process.env.ROOM_ID;
var token = process.env.TOKEN;

We need to set up an options variable with our access token and room id in it:

// Setting the options variable to use it in the https request block
var options = {
    hostname: 'stream.gitter.im',
    port:     443,
    path:     '/v1/rooms/' + roomId + '/chatMessages',
    method:   'GET',
    headers:  {'Authorization': 'Bearer ' + token}
};

We will send this options variable when making a request so that Gitter lets our request through and notifies us when a client messages to our SUSI chat room.

The res.on(‘data’) accepts a function which is called when a client messages to our SUSI chat room:

// making a request to gitter stream API
var req = https.request(options, function(res) {
 res.on('data', function(chunk) {
    // do stuff
 }
}

req.on('error', function(e) {
 console.log('Hey something went wrong: ' + e.message);
});

req.end();

According to the docs of REST API in Gitter, the JSON data that we receive when a client sends a message to a chat room is like this:

To get this response set in a variable, we can use this code snippet:

res.on('data', function(chunk) {
   var msg = chunk.toString();
   if(msg != " \n"){              // If message is not an empty message
     var jsonMsg = JSON.parse(msg);

Now we have this json response as shown in the above figure in the jsonMsg variable. To extract the client’s message from this json object:

var clientMsg = jsonMsg.text;

As we now have the user query in clientMsg variable. Let’s call Susi API and fetch an answer from it for a query.

As an example, let’s first take the query as ‘hi’ and visit http://api.asksusi.com/susi/chat.json?q=hi from the browser. We will get a JSON object as follows:

{
        "query": "hi",
        "count": 1,
        "client_id": "aG9zdF8xMDMuMjkuMjIyLjE4MA==",
        "query_date": "2017-07-17T02:29:44.171Z",
        "answers": [{
            "data": [{
                "0": "hi",
                "token_original": "hi",
                "token_canonical": "hi",
                "token_categorized": "hi",
                "timezoneOffset": "-330",
                "language": "en"
            }],
            "metadata": {"count": 1},
            "actions": [{
                    "type": "answer",
                    "language": "de",
                    "expression": "Hallo!"
            }],
   "skills": ["/susi_skill_data/models/general/smalltalk/de/German-Standalone-aiml2susi.txt"]
        }],
        "answer_date": "2017-07-17T02:29:44.179Z",
        "answer_time": 8,
        "language": "en",
        "session": {"identity": {
            "type": "host",
            "name": "103.29.222.180",
            "anonymous": true
        }}
    }

The answer can be found as the value of the key named expression. In this case, it is “Hallo!”.

To fetch the answer through coding for our client message, we can use this code snippet in Node js:

// including request module
var request = require(‘request’);

// setting options to make a successful call to Susi API.
var susiOptions = {
            method: 'GET',
            url: 'http://api.asksusi.com/susi/chat.json',
            qs:
            {
                timezoneOffset: '-330',
                q: clientMsg  //the client message sent to SUSI chat room.
            }
        };

// A request to the Susi bot
request(susiOptions, function (error, response, body) {
    if (error)
        throw new Error(error);
    // answer fetched from susi
    ans = (JSON.parse(body)).answers[0].actions[0].expression;
}

The properties required for the call are set up through a JSON object (i.e. susiOptions). Pass the susiOptions object to our request function as its 1st parameter. The response from the API will be stored in ‘body’ variable. We need to parse this body, to be able to access the properties of that body object. Hence, fetching the answer from Susi API.

As we now have the answer, let’s call the API of Gitter to show our answer back to the user. Let’s code the request for that:

// To send a reply by Susi AI to client's message back to Gitter
         var gitterOptions = {
                               method: 'POST',
                               url: 'https://api.gitter.im/v1/rooms/'+roomId+'/chatMessages',
                               headers:
                               {
                                 'authorization': 'Bearer '+ token ,
                                 'content-type': 'application/json',
                                 'accept': 'application/json'
                               },
                               body:
                               {
                                 text: ans
                               },
                               json: true
                             };

         // making the request to Gitter API
         request(gitterOptions, function (error, response, body) {
           if(error)
             throw new Error(error);
           console.log(body);
         });

Hence, we have made the basic chat work!

The streaming API of Gitter notifies us for every message sent to our chat room. We will also be notified about the reply message sent by ourselves. To not fall into an infinite loop of answers and questions by SUSI itself, we must include this line in our code:

res.on('data', function(chunk) {
   var msg = chunk.toString();
   if(msg != ” \n”){              // If message is not an empty message
     var jsonMsg = JSON.parse(msg);
     if(jsonMsg.fromUser.displayName != 'SusiAI'){ // If it’s not our own answer
        // do stuff 
     }
   }
}

req.on('error', function(e) {
 console.log('Hey something went wrong: ' + e.message);
});

req.end();

The display name in my case is ‘SusiAI’, but it may be different in your case according to the Github or Twitter id made by you.

  1. Upload this code to Github.
  2. Connect the Heroku app to the Github repository, which has your code.

  1. Deploy on the development branch. If you intend to contribute, it is recommended to Enable Automatic Deploys.

Branch Deployment.

Successful Deployment.

  1. Go to your Gitter room created and enjoy chatting with Susi.Resources
Continue ReadingIntegration of Susi AI to Gitter

Managing States in SUSI MagicMirror Module

SUSI MagicMirror Module is a module for MagicMirror project by which you can use SUSI directly on MagicMirror. While developing the module, a problem I faced was that we need to manage the flow between the various stages of processing of voice input by the user and displaying SUSI output to the user. This was solved by making state management flow between various states of SUSI MagicMirror Module namely,

  • Idle State: When SUSI MagicMirror Module is actively listening for a hotword.
  • Listening State: In this state, the user’s speech input from the microphone is recorded to a file.
  • Busy State: The user has finished speaking or timed out. Now, we need to transcribe the audio spoken by the user, send the response to SUSI server and speak out the SUSI response.

The flow between these states can be explained by the following diagram:

As clear from the above diagram, transitions are not possible from a state to all other states. Only some transitions are allowed. Thus, we need a mechanism to guarantee only allowed transitions and ensure it triggers on the right time.

For achieving this, we first implement an abstract class State with common properties of a state. We store the information whether a state can transition into some other state in a map allowedTransitions which maps state names “idle”, “listening” and “busy” to their corresponding states. The transition method to transition from one state to another is implemented in the following way.

protected transition(state: State): void {
   if (!this.canTransition(state)) {
       console.error(`Invalid transition to state: ${state}`);
       return;
   }

   this.onExit();
   state.onEnter();
}

private canTransition(state: State): boolean {
   return this.allowedStateTransitions.has(state.name);
}

Here we first check if a transition is valid. Then we exit one state and enter into the supplied state.  We also define a state machine that initializes the default state of the Mirror and define valid transitions for each state. Here is the constructor for state machine.

constructor(components: IStateMachineComponents) {
        this.idleState = new IdleState(components);
        this.listeningState = new ListeningState(components);
        this.busyState = new BusyState(components);

        this.idleState.AllowedStateTransitions = new Map<StateName, State>([["listening", this.listeningState]]);
        this.listeningState.AllowedStateTransitions = new Map<StateName, State>([["busy", this.busyState], ["idle", this.idleState]]);
        this.busyState.AllowedStateTransitions = new Map<StateName, State>([["idle", this.idleState]]);

        this.currentState = this.idleState;
        this.currentState.onEnter();
}

Now, the question arises that how do we detect when we need to transition from one state to another. For that we subscribe on the Snowboy Detector Observable. We are using Snowboy library for Hotword Detection. Snowboy detects whether an audio stream is silent, has some sound or whether hotword was spoken. We bind all this information to an observable using the ReactiveX Observable pattern. This gives us a stream of events to which we can subscribe and get the results. It can be understood in the following code snippet.

detector.on("silence", () => {
   this.subject.next(DETECTOR.Silence);
});

detector.on("sound", () => {});

detector.on("error", (error) => {
   console.error(error);
});

detector.on("hotword", (index, hotword) => {
   this.subject.next(DETECTOR.Hotword);
});
public get Observable(): Observable<DETECTOR> {
   return this.subject.asObservable();
}

Now, in the idle state, we subscribe to the values emitted by the observable of the detector to know when a hotword is detected to transition to the listening state. Here is the code snippet for the same.

this.detectorSubscription = this.components.detector.Observable.subscribe(
   (value) => {
   switch (value) {
       case DETECTOR.Hotword:
           this.transition(this.allowedStateTransitions.get("listening"));
           break;
   }
});

In the listening state, we subscribe to the states emitted by the detector observable to find when silence is detected so that we can stop recording the audio stream for processing and move to busy state.

this.detectorSubscription = this.components.detector.Observable.subscribe(
   (value) => {
   switch (value) {
       case DETECTOR.Silence:
           record.stop();
           this.transition(this.allowedStateTransitions.get("busy"));
           break;
   }
});

The task of speaking the audio and displaying results on the screen is done by a renderer. The communication to renderer is done via a RendererCommunicator object using a notification system. We also bind its events to an observable so that we know when SUSI has finished speaking the result. To transition from busy state to idle state, we subscribe to renderer observable in the following manner.

this.rendererSubscription = this.components.rendererCommunicator.Observable.subscribe((type) => {
   if (type === "finishedSpeaking") {
       this.transition(this.allowedStateTransitions.get("idle"));
   }
});

In this way, we transition between various states of MagicMirror Module for SUSI in an efficient manner.

Resources

Continue ReadingManaging States in SUSI MagicMirror Module