Search Functionalities in SUSI Android App Using Android SearchView Widget

Searching is a common feature that is required in most applications. But the problem in implementing searching functionality is that there is no common way to do that. People fight over whose way is best to implement search functionality. In this blog post we’ll be looking at how search functionality works in SUSI Android App and how is it implemented. We have used Android’s SearchView widget to do that. There are many other ways to do so but this one is best suited for our requirements. Let’s see how it works.

UI Components used for Searching

1. Search icon (magnifying glass icon)

In the action bar, you can see a small icon. Clicking on the icon initiates search.

2. Edit text

An Obvious requirement is an edit test to enter search query.

3. Up and Down arrow keys

Required to search through the whole app. Simply use the up and down arrow keys to navigate through the app and find out each occurrence of the word you want to search.

 

 

 

 

 

 

 

4. Cross Button

Last but not the least, a close or cross button to close the search action.

Implementation

We have used Android’s inbuilt Widget SearchView. According to official android documentation

A widget that provides a user interface for the user to enter a search query and submit a request to a search provider. Shows a list of query suggestions or results, if available, and allows the user to pick a suggestion or result to launch into.

This widget makes searching a lot easier. It provides all methods and listeners which are actually required for searching. Let’s cover them one by one.

  1. Starting the search: searchView.setOnSearchClickListener Listener simply activates when a user clicks on search icon in the toolbar. Do all your work which needs to be done at the starting of the search like, hiding some other UI elements of doing an animation inside the listener
searchView.setOnSearchClickListener({
   chatPresenter.startSearch()
})
  1. Stop the Search: searchView.setOnCloseListener Listener gets activated when a user clicks on the cross icon to close the search. Add all the code snippet you want which is needed to be executed when the search is closed inside this like maybe notify the adapter about data set changes or closing the database etc.
searchView.setOnCloseListener({
   chatPresenter.stopSearch()
   false
})
  1.  Searching a query:  searchView.setOnQueryTextListener Listener overrides 2 methods:

3.1 onQueryTextSubmit: As the name suggests, this method is called when the query to be searched is submitted.

3.2 onQueryTextChange: This method is called when query you are writing changes.

We, basically wanted same thing to happen if user has submitted the query or if he is still typing and that is to take the query at that particular moment, find it in database and highlight it. So, chatPresenter.onSearchQuerySearched(query) this method is called in both onQueryTextSubmit and onQueryTextSubmit  to do that.

 searchView.setOnQueryTextListener(object : SearchView.OnQueryTextListener {
 
      override fun onQueryTextSubmit(query: String): Boolean {
           //Handle Search Query
           chatPresenter.onSearchQuerySearched(query)
           recyclerAdapter.query = query
           return false
       }

       override fun onQueryTextChange(newText: String): Boolean {
           if (TextUtils.isEmpty(newText)) {
               modifyMenu(false)
               recyclerAdapter.highlightMessagePosition = -1
               recyclerAdapter.notifyDataSetChanged()
               if (!editText.isFocused) {
                   editText.requestFocus()
               }
           } else {
               chatPresenter.onSearchQuerySearched(newText)
               recyclerAdapter.query = newText
           }
           return false
       }
   })
   return true
}
  1. Finding query in database: Now we have a query to be searched, we can just use a database operation to do that. The below code snippet finds all the messages which has the query present in it and work on it. If the query is not found, it simply displays a toast saying “Not found”
override fun onSearchQuerySearched(query: String) {
   chatView?.displaySearchElements(true)
   results = databaseRepository.getSearchResults(query)
   offset = 1
   if (results.size > 0) {
       chatView?.modifyMenu(true)
       chatView?.searchMovement(results[results.size - offset].id.toInt())
   } else {
       chatView?.showToast(utilModel.getString(R.string.not_found))
   }
}

This is the database operation.

override fun getSearchResults(query: String): RealmResults<ChatMessage> {
   return realm.where(ChatMessage::class.java).contains(Constant.CONTENT,
           query, Case.INSENSITIVE).findAll()
}

  1. Highlighting the part of message: Now, we know which message has the query, we just want to highlight it with a bright color to display the result. For that, we used SpannableString to highlight a part of complete string.
String text = chatTextView.getText().toString();
SpannableString modify = new SpannableString(text);
Pattern pattern = Pattern.compile(query, Pattern.CASE_INSENSITIVE);
Matcher matcher = pattern.matcher(modify);
while (matcher.find()) {
   int startIndex = matcher.start();
   int endIndex = matcher.end();
   modify.setSpan(new BackgroundColorSpan(Color.parseColor("#ffff00")), startIndex, endIndex, Spannable.SPAN_EXCLUSIVE_EXCLUSIVE);
}
chatTextView.setText(modify);

Summary

The whole point of this blog post was to educate about SearchView widget of android and how it makes it easy to search queries. All the methods you need are already implemented. You just need to call them and add database operation.

Resources

  1. The link to official android documentation explaining about different methods in SearchView Class https://developer.android.com/reference/android/widget/SearchView.html
  2. Another tutorial about SearchView http://www.journaldev.com/12478/android-searchview-example-tutorial

Sorting Photos in Phimpme Android

The Phimpme Android application features a fully fledged gallery interface with an option to switch to all photos mode, albums mode and to sort photos according to various sort actions. Sorting photos via various options helps the user to get to the desired photo immediately without having to scroll down till the end in case it is the last photo in the list generated automatically by scanning the phone for images using the Android’s mediaStore class. In this tutorial, I will be discussing how we achieved the sorting option in the Phimpme application with the help of some code snippets.

To sort the all photos list, first of all we need a list of all the photos by scanning the phone using the media scanner class via the code snippet provided below:

uri = android.provider.MediaStore.Images.Media.EXTERNAL_CONTENT_URI;
      String[] projection = {MediaStore.MediaColumns.DATA};
      cursor = activity.getContentResolver().query(uri, projection, null, null, null);

In the above code we are using a cursor to point to each photos and then we are extracting the path of the images and storing it in a list using a while loop. After we generate the list of path of all the images, we have to convert the into a list of media using the file path using the code below:

for (String path : listOfAllImages) {
          list.add(new Media(new File(path)));
      }
      return list;
  }

After generating the list of all images we can sort the photos using the Android’s collection class. In Phimpme Android we provide the option to sort photos in different categories namely:

  1. Name Sort action
  2. Date Sort action
  3. Size Sort action
  4. Numeric Sort action

As sorting is somewhat heavy task so doing this in the main thread will result in freezing UI of the application so we have to put this into an AsyncTask with a progress dialog to sort the photos. After putting the above four options in the menu options. We can define an Asynctask to load the images and in the onPreExecute method of the AsyncTask, we are displaying the progress dialog to the user to depict that the sorting process is going on as depicted in the code snippet below

AlertDialog.Builder progressDialog = new AlertDialog.Builder(LFMainActivity.this, getDialogStyle());
dialog = AlertDialogsHelper.getProgressDialog(LFMainActivity.this, progressDialog,
      getString(R.string.loading_numeric), all_photos ? getString(R.string.loading_numeric_all) : getAlbum().getName());
dialog.show();

In the doInBackgroundMethod of the AsyncTask, we are sorting the list of all photos using the Android’s collection class and using the static sort method defined in that class which takes the list of all the media files as a parameter and the MediaComparator which takes the sorting mode as the first parameter and the sorting order as the second. The sorting order decides whether to arrange the list in ascending or in descending order.

getAlbum().setDefaultSortingMode(getApplicationContext(), NUMERIC);
Collections.sort(listAll, MediaComparators.getComparator(getAlbum().settings.getSortingMode(), getAlbum().settings.getSortingOrder()));

After sorting, we have to update the data set to reflect the changes of the list in the UI. This we are doing in the onPostExecute method of the AsyncTask after dismissing the progress Dialog to avoid the window leaks in the application. You can read more about the window leaks in Android due to progressdialog here.

dialog.dismiss();
mediaAdapter.swapDataSet(listAll);

To get the full source code, you can refer the Phimpme Android repository listed in the resources below.

Resources

  1. Android developer guide to mediastore class: https://developer.android.com/reference/android/provider/MediaStore.html
  2. GitHub LeafPic repository: https://github.com/HoraApps/LeafPic
  3. Stackoverflow – Preventing window leaks in Android: https://stackoverflow.com/questions/6614692/progressdialog-how-to-prevent-leaked-window
  4. Blog – Sorting lists in Android: http://www.worldbestlearningcenter.com/tips/Android-sort-ListView.htm

Integrating Stock Sensors with PSLab Android App

A sensor is a digital device (almost all the time an integrated circuit) which can receive data from outer environment and produce an electric signal proportional to that. This signal will be then processed by a microcontroller or a processor to provide useful functionalities. A mobile device running Android operating system usually has a few sensors built into it. The main purpose of these sensors is to provide user with better experience such as rotating the screen as he moves the device or turn off the screen when he is making a call to prevent unwanted screen touch events. PSLab Android application is capable of processing inputs received by different sensors plugged into it using the PSLab device and produce useful results. Developers are currently planning on integrating the stock sensors with the PSLab device so that the application can be used without the PSLab device.

This blog is about how to initiate a stock sensor available in the Android device and get readings from it. Sensor API provided by Google developers is really helpful in achieving this task. The process is consist of several steps. It is also important to note the fact that there are devices that support only a few sensors while some devices will support a lot of sensors. There are few basic sensors that are available in every device such as

  • “Accelerometer” – Measures acceleration along X, Y and Z axis
  • “Gyroscope” – Measures device rotation along X, Y and Z axis
  • “Light Sensor” – Measures illumination in Lux
  • “Proximity Sensor” – Measures distance to an obstacle from sensor

The implementing steps are as follows;

  1. Check availability of sensors

First step is to invoke the SensorManager from Android system services. This class has a method to list all the available sensors in the device.

SensorManager sensorManager;
sensorManager = (SensorManager) getSystemService(Context.SENSOR_SERVICE);
List<Sensor> sensors = sensorManager.getSensorList(Sensor.TYPE_ALL);

Once the list is populated, we can iterate through this to find out if the required sensors are available and obstruct displaying activities related to sensors that are not supported by the device.

for (Sensor sensor : sensors) {
   switch (sensor.getType()) {
       case Sensor.TYPE_ACCELEROMETER:
           break;
       case Sensor.TYPE_GYROSCOPE:
           break;
       ...
   }
}

  1. Read data from sensors

To read data sent from the sensor, one should implement the SensorEventListener interface. Under this interface, there are two method needs to be overridden.

public class StockSensors extends AppCompatActivity implements SensorEventListener {

    @Override
    public void onSensorChanged(SensorEvent sensorEvent) {

    }

    @Override
    public void onAccuracyChanged(Sensor sensor, int i) {

    }
}

Out of these two methods, onSensorChanged() method should be addressed. This method provides a parameter SensorEvent which supports a method call getType() which returns an integer value representing the type of sensor produced the event.

@Override
public void onSensorChanged(SensorEvent sensorEvent) {
   switch (sensorEvent.sensor.getType()) {
       case Sensor.TYPE_ACCELEROMETER:
           break;
       case Sensor.TYPE_GYROSCOPE:
           break;
       ...
   }
}

Each available sensor should be registered under the SensorEventListener to make them available in onSensorChanged() method. The following code block illustrates how to modify the previous code to register each sensor easily with the listener.

for (Sensor sensor : sensors) {
   switch (sensor.getType()) {
       case Sensor.TYPE_ACCELEROMETER:
           sensorManager.registerListener(this, sensorManager.getDefaultSensor(Sensor.TYPE_ACCELEROMETER), SensorManager.SENSOR_DELAY_UI);
           break;
       case Sensor.TYPE_GYROSCOPE:
           sensorManager.registerListener(this, sensorManager.getDefaultSensor(Sensor.TYPE_GYROSCOPE), SensorManager.SENSOR_DELAY_UI);
           break;
   }
}

Depending on the readings we can provide user with numerical data or graphical data using graphs plotted using MPAndroidChart in PSLab Android application.

The following images illustrate how a similar implementation is available in Science Journal application developed by Google.

Resources

Controlling Camera Actions Using Voice Interaction in Phimpme Android

In this blog, I will explain how I implemented Google voice actions to control camera features on the Phimpme Android project. I will cover the following features I have implemented on the Phimpme project:

  • Opening the application using Google Voice command.
  • Switching between the cameras.
  • Clicking a Picture and saving it through voice command.

Opening application when the user gives a command to Google Now.                       When the user gives command “Take a selfie” or “Click a picture” to Google Now it directly opens Phimpme camera activity.

 First                                                                                                                                        We need to add an intent filter to the manifest file so that Google Now can  detect Phimpme camera activity

<activity
   android:name=".opencamera.Camera.CameraActivity"
   android:screenOrientation="portrait"
   android:theme="@style/Theme.AppCompat.NoActionBar">
   <intent-filter>
       <action android:name="android.media.action.IMAGE_CAPTURE"/>

       <category android:name="android.intent.category.DEFAULT"/>
       <category android:name="android.intent.category.VOICE"/>
   </intent-filter>
</activity>

category android:name=”android.intent.category.VOICE” is added to the IMAGE_CAPTURE intent filter for the Google Now to detect the camera activity. For the Google Now assistance to accept the command in the camera activity we need to add the following in the STILL_IMAGE_CAMERA intent filter in the camera activity.

<intent-filter>
   <action android:name="android.media.action.STILL_IMAGE_CAMERA"/>

   <category android:name="android.intent.category.DEFAULT"/>
   <category android:name="android.intent.category.VOICE"/>
</intent-filter>

So, now when the user says “OK Google” and “Take a Picture” the camera activity on Phimpme opens.

Integrating Google Voice assistance in Camera Activity

Second,                                                                                                                               After opening the camera application the Google Assistance should ask a question.

The cameraActivity in Phimpme can be opened in two ways, such as:

  • When opened from a different application.
  • When given the command to Goole Now assistance.

We need to check whether the camera activity is prompted from Google assistance or not to activate voice command. We will check it in onResume function.

@Override
public void onResume() {
if (CameraActivity.this.isVoiceInteraction()) {
     startVoiceTrigger();
  }
} 

If is.VoiceInteraction gives “true” then voice Assistance prompts.             Assistance to ask which camera to use

Third,                                                                                                                                 After the camera activity opens the Google assistance should ask which camera to use either front or back.

To take any voice input from the user, we to store the expected commands in VoiceInteractor.PickoptionRequest. This function listens to the command by the user. We need to add synonyms for the same command.

To choose the rear camera

VoiceInteractor.PickOptionRequest.Option rear = new VoiceInteractor.PickOptionRequest.Option(getResources().getString(R.string.camera_rear), 0);
rear.addSynonym(getResources().getString(R.string.rear));
rear.addSynonym(getResources().getString(R.string.back));
rear.addSynonym(getResources().getString(R.string.normal)); 

I added synonyms like the rear, normal and back.

To choose front camera

VoiceInteractor.PickOptionRequest.Option front = new VoiceInteractor.PickOptionRequest.Option(getResources().getString(R.string.camera_front), 1);
front.addSynonym(getResources().getString(R.string.front));
front.addSynonym(getResources().getString(R.string.selfie_camera));
front.addSynonym(getResources().getString(R.string.forward));

I added synonyms like the front, selfie camera and forward. 

For assistance to ask any question such as “Which camera to use we” I have used getVoiceinteractor and inflating VoiceInteractor.PickOptionRequest.option[] array with options front and rear.

CameraActivity.this.getVoiceInteractor()
     .submitRequest(new VoiceInteractor.PickOptionRequest(
           new VoiceInteractor.Prompt(getResources().getString(“Which camera would you like to use?”),
           new VoiceInteractor.PickOptionRequest.Option[]{front, rear},
           null) {

The google assistance waits for a response from the user for only a few seconds and it goes inactive. If the user gives any unexpected command the assistance will ask the question one more time.

Check if the user gives an expected command or not.

We will override OnOptionResult(boolean finished, Options[] selection, Bundle result) function.if  (finished && selections.length == 1) if the speech length matches with any of the options provided it checks which option was used.

Check the command given by the user to switch between the cameras.

Two array objects are passed 0 and 1.  If the command given was “rear” then selection[0].getindex() = 0 and camera activity switches to the rear camera and if the the command given by the user is rear then selection[0].getIndex = 1 and camera activity switches to front camera.

@Override
public void onPickOptionResult(boolean finished, Option[] selections, Bundle result) {
  if (finished && selections.length == 1) {
     Message message = Message.obtain();
     message.obj = result;
     if (selections[0].getIndex() == 0)
     {  rearCamera();
        asktakePicture();
     }
     if (selections[0].getIndex() == 1)
     {
        asktakePicture();
     }
  }else{

       getActivity().finish();
  }

Click Picture when the user says “Cheese

After switching the camera the assistant prompts the message”Say cheese”. We need to add voiceInteractor.prompt(“Say cheese”).

We need to store the synonyms in VoiceInteractor.PickOption.Options options. I have added synonyms like ready, go, take it, OK, and Cheese to click a picture. If the user gives an unexpected output the assistance checks selection.length ==1 or not and prompts the message “Say cheese” again.

private void asktakePicture() {
  VoiceInteractor.PickOptionRequest.Option option = new VoiceInteractor.PickOptionRequest.Option(getResources().getString(R.string.cheese), 2);
  option.addSynonym(getResources().getString(R.string.ready));
  option.addSynonym(getResources().getString(R.string.go));
  option.addSynonym(getResources().getString(R.string.take));
  option.addSynonym(getResources().getString(R.string.ok));
getVoiceInteractor()
        .submitRequest(new VoiceInteractor.PickOptionRequest(
              new VoiceInteractor.Prompt(getResources().getString(R.string.say_cheese)),
              new VoiceInteractor.PickOptionRequest.Option[]{option},
              null) {
           @Override
           public void onPickOptionResult(boolean finished, Option[] selections, Bundle result) {
              if (finished && selections.length == 1) {
                 Message message = Message.obtain();
                 message.obj = result;
                 takePicture();
              } else {
                 getActivity().finish();
              }
           }
           @Override
           public void onCancel() {
              getActivity().finish();
           }
        });                                                                                                                                     

Conclusion

Now, Users can start camera activity on Phimpme through voice command “Take a Selfie”. They can switch between the cameras through voice command “Selfie camera” or “back camera”, “back” or “front” and finally click a picture by giving voice command “Cheese”, “Click it” and related synonyms.

Github

Resources

Implementing Stickers on an Image in Phimpme

One feature we implemented in the Phimpme photo editing application is to enable users to add stickers on top of the image. In this blog, I will explain how stickers are implemented in Phimpme.

Features of Stickers

  • Users can resize the stickers.
  • Users can place the stickers wherever in the canvas.

Step.1-Storing the Stickers in assets folder

In Phimpme I stored the stickers in the assets folder. To distribute the stickers in different categories I made different folders according to the categories namely type1, type2, type3, type4 and so on.  

Displaying stickers

We used onBindViewHolder to Display the stickers in different categories like:

  • Facial
  • Express
  • Objects
  • Comments
  • Wishes
  • Emojis
  • Hashtags

String path will get the position of the particular type of stickers collection. This type is then loaded to the ImageLoader with the specific icon associating with the types.   

@Override
public void onBindViewHolder(mRecyclerAdapter.mViewHolder holder, final int position) {

   String path = pathList.get(position);
       ImageLoader.getInstance().displayImage("assets://" + path,holder.icon, imageOption);
       holder.itemView.setTag(path);
       holder.title.setText("");

   int size = (int) getActivity().getResources().getDimension(R.dimen.icon_item_image_size_filter_preview);
   LinearLayout.LayoutParams layoutParams = new LinearLayout.LayoutParams(size,size);
   holder.icon.setLayoutParams(layoutParams);

   holder.itemView.setOnClickListener(new View.OnClickListener() {
       @Override
       public void onClick(View v) {
           String data = (String) v.getTag();
           selectedStickerItem(data);
       }
   });
}

Step.2- Applying a sticker on the image

When a particular sticker is selected selectedStickerItem() function is called.This function calls StickerView class to add the Bitmap on the image. It sends the path of the sticker as a parameter.  

public void selectedStickerItem(String path) {
   mStickerView.addBitImage(getImageFromAssetsFile(path));
}

In StickerView class the image of the sticker is then converted into a Bitmap. It creates an object(item) of StickerItem class. This object calls the init function, which handles the size of the sticker and the placement of the sticker on the image.

public void addBitImage(final Bitmap addBit) {
   StickerItem item = new StickerItem(this.getContext());
   item.init(addBit, this);
   if (currentItem != null) {
       currentItem.isDrawHelpTool = false;
   }
   bank.put(++imageCount, item);
   this.invalidate();
}

Step.3-Resizing the Sticker in the canvas

A bitmap or any image has two axes namely x and y. We can resize the image using matrix calculation.

float c_x = dstRect.centerX();
float c_y = dstRect.centerY();

float x = this.detectRotateRect.centerX();
float y = this.detectRotateRect.centerY();

We then calculate the source length and the current length:

float srcLen = (float) Math.sqrt(xa * xa + ya * ya);
float curLen = (float) Math.sqrt(xb * xb + yb * yb);

Then we calculate the scale. This is required to calculate the zoom ratio.

float scale = curLen / srcLen;

We need to rescale the bitmap. That is if the user rotates the sticker or zoom in or zoom out the sticker. A helpbox surrounds the stickers showing the actual size of the sticker. This helpbox which is rectangular shape helps in resizing the sticker.

RectUtil.scaleRect(this.dstRect, scale);// Zoom destination rectangle

// Recalculate the Toolbox coordinates
helpBox.set(dstRect);
updateHelpBoxRect();// Recalculate
rotateRect.offsetTo(helpBox.right - BUTTON_WIDTH, helpBox.bottom
       - BUTTON_WIDTH);
deleteRect.offsetTo(helpBox.left - BUTTON_WIDTH, helpBox.top
       - BUTTON_WIDTH);

detectRotateRect.offsetTo(helpBox.right - BUTTON_WIDTH, helpBox.bottom
       - BUTTON_WIDTH);
detectDeleteRect.offsetTo(helpBox.left - BUTTON_WIDTH, helpBox.top
       - BUTTON_WIDTH);

Conclusion

In Phimpme a user can now place the sticker on top of the image. Resize the sticker, that is Zoom in the image or zoom out of the image. Move the image around the canvas. This will give users the flexibility to add multiple stickers on the image.

Github

Resources

Share Images on Pinterest from Phimpme Android Application

After successfully establishing Pinterest authentication in Phimpme our next goal was to share the image on the Pinterest website directly from Phimpme, without using any native Android application.

Adding Pinterest Sharing option in Sharing Activity in Phimpme

To add various sharing options in Sharing Activity in the Phimpme project, I have applied a ScrollView for the list of the different sharing options which include: Facebook, Twitter, Pinterest, Imgur, Flickr and Instagram. All the App icons with the name are arranged in a TableLayout in the activity_share.xml file. Table rows consist of two columns. In this way, it is easier to add more app icons for future development.

<ScrollView
   android:layout_width="wrap_content"
   android:layout_height="@dimen/scroll_view_height"
   android:layout_above="@+id/share_done"
   android:id="@+id/bottom_view">
   <LinearLayout
       android:layout_width="wrap_content"
       android:layout_height="wrap_content"
       android:orientation="vertical">
       <TableLayout

Adding Pinterest app icon on the icons_drawable array. This array is then used to inflate the icon on the list view.

private int[] icons_drawables = {R.drawable.ic_facebook_black, R.drawable.ic_twitter_black,R.drawable.ic_instagram_black, R.drawable.ic_wordpress_black, R.drawable.ic_pinterest_black);

Adding Pinterest text on the titles_text array. This array is then used to inflate the names of the various sharing activity.

private int[] titles_text = {R.string.facebook, R.string.twitter, R.string.instagram,
       R.string.wordpress, R.string.pinterest);

Prerequisites to share Image on Pinterest

To share an Image on Pinterest a user has to add a caption and Board ID. Our first milestone was to get the input of the Board ID  by the user. I have achieved this by taking the input in a Dialog Box. When the user clicks on the Pinterest option, a dialog box pops and then the user can add their Board ID.

private void openPinterestDialogBox() {
   AlertDialog.Builder captionDialogBuilder = new AlertDialog.Builder(SharingActivity.this, getDialogStyle());
   final EditText captionEditText = getCaptionDialog(this, captionDialogBuilder);

   captionEditText.setHint(R.string.hint_boardID);

   captionDialogBuilder.setNegativeButton(getString(R.string.cancel).toUpperCase(), null);
   captionDialogBuilder.setPositiveButton(getString(R.string.post_action).toUpperCase(), new DialogInterface.OnClickListener() {
       @Override
       public void onClick(DialogInterface dialog, int which) {
           //This should be empty it will be overwrite later
           //to avoid dismiss of the dialog on the wrong password
       }
   });

   final AlertDialog passwordDialog = captionDialogBuilder.create();
   passwordDialog.show();

   passwordDialog.getButton(AlertDialog.BUTTON_POSITIVE).setOnClickListener(new View.OnClickListener() {
       @Override
       public void onClick(View v) {
           String captionText = captionEditText.getText().toString();
           boardID =captionText;
           shareToPinterest(boardID);
           passwordDialog.dismiss();
       }
   });
}

A user can fetch the Board ID by following the steps:

Board ID is necessary because it specifies where the image needs to be posted.

Creating custom post function for Phimpme

The image is posted using a function in PDKClient class. PDKClient is found in the PDK module which we get after importing Pinterest SDK. Every image is posted on Pinterest is called a Pin. So we will call createPin function. I have made my custom createPin function so that it also accepts Bitmaps as a parameter. In the Pinterest SDK it only accepts image URL to share, The image should already be on the internet to be shared. For this reason, we to add a custom create Pin function to accept Bitmaps as an option.

public void createPin(String note, String boardId, Bitmap image, String link, PDKCallback callback) {
   if (Utils.isEmpty(note) || Utils.isEmpty(boardId) || image == null) {
       if (callback != null) callback.onFailure(new PDKException("Board Id, note, Image cannot be empty"));
       return;
   }

   HashMap<String, String> params = new HashMap<String, String>();
   params.put("board", boardId);
   params.put("note", note);
   params.put("image_base64", Utils.base64String(image));
   if (!Utils.isEmpty(link)) params.put("link", link);
   postPath(PINS, params, callback);
}

Compressing Bitmaps

Since Pinterest SDK cannot accept Bitmap I have added a function to compress the Bitmap and encode it to strings.

public static String base64String(Bitmap bitmap) {
   ByteArrayOutputStream baos = new ByteArrayOutputStream();
   bitmap.compress(Bitmap.CompressFormat.JPEG, 100, baos);
   String b64Str = Base64.encodeToString(baos.toByteArray(), Base64.NO_WRAP);
   return b64Str;
}

Calling createPin function from the sharingActivity

From the sharingActivity we will call createPin activity. We will pass caption of the image, Board ID, Image bitmap and link(which is optional) as parameters.

PDKClient
       .getInstance().createPin(caption, boardID, image, null, new PDKCallback() {

If the image is posted successfully then onSuccess function is called which pops a snackbar and shows the success message. Otherwise, onFailure function is called which displays failure message on a snackbar.

@Override
public void onSuccess(PDKResponse response) {
   Log.d(getClass().getName(), response.getData().toString());
   Snackbar.make(parent, R.string.pinterest_post, Snackbar.LENGTH_LONG).show();
   //Toast.makeText(SharingActivity.this,message,Toast.LENGTH_SHORT).show();

}

@Override
public void onFailure(PDKException exception) {
   Log.e(getClass().getName(), exception.getDetailMessage());
   Snackbar.make(parent, R.string.Pinterest_fail, Snackbar.LENGTH_LONG).show();
   //Toast.makeText(SharingActivity.this, boardID + caption, Toast.LENGTH_SHORT).show();

}

Conclusion

In Phimpme user can now send Image on Pinterest directly from the application. This is done without the use of the native Pinterest application.

Github

Resources

Using OpenCV for Native Image Processing in Phimpme Android

OpenCV is very widely used open-source image processing library. After the integration of OpenCV Android SDK in the Phimpme Android application, the image processing functions can be written in Java part or native part. Taking runtime of the functions into consideration we used native functions for image processing in the Phimpme application.

We didn’t have the whole application written in native code, we just called the native functions on the Java OpenCV Mat object. Mat is short for the matrix in OpenCV. The image on which we perform image processing operations in the Phimpme Android application is stored as Mat object in OpenCV.

Creating a Java OpenCV Mat object

Mat object of OpenCV is same whether we use it in Java or C++. We have common OpenCV object in Phimpme for accessing from both Java part and native part of the application. We have a Java bitmap object on which we have to perform image processing operations using OpenCV. For doing that we need to create a Java Mat object and pass its address to native. Mat object of OpenCV can be created using the bitmap2Mat() function present in the OpenCV library. The implementation is shown below.

Mat inputMat = new Mat(bitmap.getWidth(), bitmap.getHeight(), CvType.CV_8UC3);
Utils.bitmapToMat(bitmap, inputMat);

“bitmap” is the Java bitmap object which has the image to be processed. The third argument in the Mat function indicates that the Mat should be of type 8UC3 i.e. three color channels with 8-bit depth. With the second line above, the bitmap gets saved as the OpenCV Mat object.

Passing Mat Object to Native

We have the OpenCV Mat object in the memory. If we pass the whole object again to native, the same object gets copied from one memory location to another. In Phimpme application, instead of doing all that we can just get the memory location of the current OpenCV Mat object and pass it to native. As we have the address of the Mat, we can access it directly from native functions. Implementation of this is shown below.

Native Function Definition:

private static native void nativeApplyFilter(long inpAddr);

Native Function call:

nativeApplyFilter(inputMat.getNativeObjAddr());

Getting Native Mat Object to Java

We can follow the similar steps for getting the Mat from the native part after processing. In the Java part of Phimpme, we created an OpenCV Mat object before we pass the inputMat OpenCV Mat object to native for processing. So we have inputMat and outputMat in the memory before we send them to native. We get the memory locations of both the Mat objects and pass those addresses to native part. After the processing is done, the data gets written to the same memory location and can be accessed in Java. The above functions can be modified and rewritten for this purpose as shown below

Native Function Definition:

private static native void nativeApplyFilter(long inpAddr, long outAddr );

Native Function call:

nativeApplyFilter(inputMat.getNativeObjAddr(),outputMat.getNativeObjAddr());
inputMat.release();

if (outputMat !=null){
   Bitmap outbit = Bitmap.createBitmap(bitmap.getWidth(),bitmap.getHeight(),bitmap.getConfig());
   Utils.matToBitmap(outputMat,outbit);
   outputMat.release();
   return outbit;
}

Native operations on Mat using OpenCV

The JNI function in the native part of Phimpme application receives the memory locations of both the OpenCV Mat objects. As we have the addresses, we can create Mat object pointing that memory location and can be passed to processing functions for performing native operations just like all OpenCV functions. This implementation is shown below.

#include <jni.h>
#include <opencv2/core/core.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include "enhance.h"
using namespace std;
using namespace cv;

JNIEXPORT void JNICALL
Java_org_fossasia_phimpme_editor_editimage_filter_PhotoProcessing_nativeApplyFilter(JNIEnv *env, jclass type, jlong inpAddr,jlong outAddr) {
       Mat &src = *(Mat*)inpAddr;
       Mat &dst = *(Mat*)outAddr;
       applyFilter(src, dst);
}

applyFilter() function can have any image processing operation. The implementation of edge detection function using OpenCV in the Phimpme Android is shown below. We were able to do this in very few lines which otherwise would have needed an extremely large code.  

Mat grey,detected_edges;
cvtColor( src, grey, CV_BGR2GRAY );
blur( grey, detected_edges, Size(3,3) );
dst.create( grey.size(), grey.type() );
Canny( detected_edges, detected_edges, 70, 200, 3 );
dst = Scalar::all(0);
detected_edges.copyTo( dst, detected_edges);
 

  

The general structure of an OpenCV function which is necessary for implementing custom image processing operations can be understood by referring this below-mentioned brightness adjustment function.  

int x,y,bright;
cvtColor(src,src,CV_BGRA2BGR);
dst = Mat::zeros( src.size(), src.type() );
for (y = 0; y < src.rows; y++) {
   for (x = 0; x < src.cols; x++) {
       dst.at<Vec3b>(y, x)[0] =
                  saturate_cast<uchar>((src.at<Vec3b>(y, x)[0]) + bright);
       dst.at<Vec3b>(y, x)[1] =
               saturate_cast<uchar>((src.at<Vec3b>(y, x)[1]) + bright);
       dst.at<Vec3b>(y, x)[2] =
               saturate_cast<uchar>((src.at<Vec3b>(y, x)[2]) + bright);
   }
}

    

Resources:

Adding Sentry Integration in Open Event Orga Android App

Sentry is a service that allows you to track events, issues and crashes in your apps and provide deep insights with context about them. This blog post will discuss how we implemented it in Open Event Orga App (Github Repo).

Configuration

First, we need to include the gradle dependency in build.gradle
compile ‘io.sentry:sentry-android:1.3.0’
Now, our project uses proguard for release builds which obfuscates the code and removes unnecessary class to shrink the app. For the crash events to make sense in Sentry dashboard, we need proguard mappings to be uploaded every time release build is generated. Thankfully, this is automatically handled by sentry through its gradle plugin, so to include it, we add this in our project level build.gradle in dependencies block

classpath 'io.sentry:sentry-android-gradle-plugin:1.3.0'

 

And then apply the plugin by writing this at top of our app/build.gradle

apply plugin: 'io.sentry.android.gradle'

 

And then configure the options for automatic proguard configuration and mappings upload

sentry {
   // Disables or enables the automatic configuration of proguard
   // for Sentry.  This injects a default config for proguard so
   // you don't need to do it manually.
   autoProguardConfig true

   // Enables or disables the automatic upload of mapping files
   // during a build.  If you disable this you'll need to manually
   // upload the mapping files with sentry-cli when you do a release.
   autoUpload false
}

 

We have set the autoUpload to false as we wanted Sentry to be an optional dependency to the project. If we turn it on, the build will crash if sentry can’t find the configuration, which we don’t want to happen.

Now, as we want Sentry to configurable, we need to set Sentry DSN as one of the configuration options. The easiest way to externalize configuration is to use environment variables. There are other methods to do it given in the official documentation for config https://docs.sentry.io/clients/java/config/

Lastly, for proguard configuration, we also need 3 other config options, namely:

defaults.project=your-project
defaults.org=your-organisation
auth.token=your-auth-token

 

For getting the auth token, you need to go to https://sentry.io/api/

Now, the configuration is complete and we’ll move to the code

Implementation

First, we need to initialise the sentry instance for all further actions to be valid. This is to be done when the app starts, so we add it in onCreate method Application class of our project by calling this method

// Sentry DSN must be defined as environment variable
// https://docs.sentry.io/clients/java/config/#setting-the-dsn-data-source-name
Sentry.init(new AndroidSentryClientFactory(getApplicationContext()));

 

Now, we’re all set to send crash reports and other events to our Sentry server. This would have required a lot of refactoring if we didn’t use Timber for logging. We are using default debug tree for debug build and a custom Timber tree for release builds.

if (BuildConfig.DEBUG)
   Timber.plant(new Timber.DebugTree());
else
   Timber.plant(new ReleaseLogTree());

 

The ReleaseLogTree extends Timber.Tree which is an abstract class requiring you to override this function:

@Override
protected void log(int priority, String tag, String message, Throwable throwable) {

 }

 

This function is called whenever there is a log event through Timber and this is where we send reports through Sentry. First, we return from the function if the event priority is debug or verbose

if(priority == Log.DEBUG || priority == Log.VERBOSE)
   return;

 

If the event if if info priority, we attach it to sentry bread crumb

if (priority == Log.INFO) {
    Sentry.getContext().recordBreadcrumb(new BreadcrumbBuilder()
          .setMessage(message)
          .build());
}

 

Breadcrumbs are stored and only send with an event. What event comprises for us is the crash event or something we want to be logged to dashboard whenever the user does it. But since info events are just user interactions throughout the app, we don’t want to crowd the issue dashboard with them. However, we want to understand what user was doing before the crash happened, and that is why we use bread crumbs to store the events and only send them attached to a crash event. Also, only the last 100 bread crumbs are stored, making it easier to parse through them.

Now, if there is an error event, we want to capture and send it to the server

if (priority == Log.ERROR) {
   if (throwable == null)
       Sentry.capture(message);
   else
       Sentry.capture(throwable);
}

 

Lastly, we want to set Sentry context to be user specific so that we can easily track and filter through issues based on the user. For that, we create a new class ContextManager with two methods:

  • setOrganiser: to be called at login
  • clearOrganiser: to be called at logout

public void setOrganiser(User user) {
   Map<String, Object> userData = new HashMap<>();
   userData.put("details", user.getUserDetail());
   userData.put("last_access_time", user.getLastAccessTime());
   userData.put("sign_up_time", user.getSignupTime());

   Timber.i("User logged in - %s", user);
   Sentry.getContext().setUser(
       new UserBuilder()
       .setEmail(user.getEmail())
       .setId(String.valueOf(user.getId()))
       .setData(userData)
       .build()
   );
}

 

In this method, we put all the information about the user in the context so that every action from here on is attached to this user.

public void clearOrganiser() {
   Sentry.clearContext();
}

 

And here, we just clear the sentry context.

This concludes the implementation of our sentry client. Now all Timber log events will through sentry and appropriate events will appear on the sentry dashboard. To read more about sentry features and Timber, visit these links:

Sentry Java Documentation (check Android section)

https://docs.sentry.io/clients/java/

Timber Library

https://github.com/JakeWharton/timber

Invalidating user login using JWT in Open Event Orga App

User authentication is an essential part of Open Event Orga App (Github Repo), which allows an organizer to log in and perform actions on the event he/she organizes. Backend for the application, Open Event Orga Server sends an authentication token on successful login, and all subsequent privileged API requests must include this token. The token is a JWT (Javascript Web Token) which includes certain information about the user, such as identifier and information about from when will the token be valid, when will it expire and a signature to verify if it was tampered.

Parsing the Token

Our job was to parse the token to find two fields:

  • Identifier of user
  • Expiry time of the token

We stored the token in our shared preference file and loaded it from there for any subsequent requests. But, the token expires after 24 hours and we needed our login model to clear it once it has expired and shown the login activity instead.

To do this, we needed to parse the JWT and compare the timestamp stored in the exp field with the current timestamp and determine if the token is expired. The first step in the process was to parse the token, which is essentially a Base 64 encoded JSON string with sections separated by periods. The sections are as follows:

  • Header ( Contains information about algorithm used to encode JWT, etc )
  • Payload ( The data in JWT – exp. Iar, nbf, identity, etc )
  • Signature ( Verification signature of JWT )

We were interested in payload and for getting the JSON string from the token, we could have used Android’s Base64 class to decode the token, but we wanted to unit test all the util functions and that is why we opted for a custom Base64 class for only decoding our token.

So, first we split the token by the period and decoded each part and stored it in a SparseArrayCompat

public static SparseArrayCompat<String> decode(String token) {
   SparseArrayCompat<String> decoded = new SparseArrayCompat<>(2);

   String[] split = token.split("\\.");
   decoded.append(0, getJson(split[0]));
   decoded.append(1, getJson(split[1]));

   return decoded;
}

 

The getJson function is primarily decoding the Base64 string

private static String getJson(String strEncoded) {
   byte[] decodedBytes = Base64Utils.decode(strEncoded);
   return new String(decodedBytes);
}

The decoded information was stored in this way

0={"alg":"HS256","typ":"JWT"},  1={"nbf":1495745400,"iat":1495745400,"exp":1495745800,"identity":344}

Extracting Information

Next, we create a function to get the expiry timestamp from the token. We could use GSON or Jackson for the task, but we did not want to map fields into any object. So we simply used JSONObject class which Android provides. It took 5 ms on average to parse the JSON instead of 150 ms by GSON

public static long getExpiry(String token) throws JSONException {
   SparseArrayCompat<String> decoded = decode(token);

   // We are using JSONObject instead of GSON as it takes about 5 ms instead of 150 ms taken by GSON
   return Long.parseLong(new JSONObject(decoded.get(1)).get("exp").toString());
}

 

Next, we wanted to get the ID of user from token to determine if a new user is logging in or an old one, so that we can clear the database for new user.

public static int getIdentity(String token) throws JSONException {
   SparseArrayCompat<String> decoded = decode(token);

   return Integer.parseInt(new JSONObject(decoded.get(1)).get("identity").toString());
}

Validating the token

After this, we needed to create a function that tells if a stored token is expired or not. With all the right functions in place, it was just a matter of comparing current time with the stored timestamp

public static boolean isExpired(String token) {
   long expiry;

   try {
       expiry = getExpiry(token);
   } catch (JSONException jse) {
       return true;
   }

   return System.currentTimeMillis()/1000 >= expiry;
}

 

Since the token provides timestamp from epoch in terms of seconds, we needed to divide the current time in milliseconds by 1000 and the function returned true if current timestamp was greater than the expiry time of token.

After writing a few unit tests for both functions, we just needed to plug them in our login model at the time of authentication.

At the time of starting of the application, we use this function to check if a user is logged in or not:

public boolean isLoggedIn() {
   String token = utilModel.getToken();

   return token != null && !JWTUtils.isExpired(token);
}

 

So, if there is no token or the token is expired, we do not automatically login the user and show the login screen.

Implementing login

The next task were

  • Sequest the server to login
  • Store the acquired token
  • Delete database if it is a new user

Before implementing the above logic, we needed to implement a function to determine if the person logging in is previous user, or new one. For doing so, we first loaded the saved user from our database, if the query is empty, surely it is a new user logging in. So we return false, and if there is a user in the database, we match its ID with the logged in user’s ID:

public Single<Boolean> isPreviousUser(String token) {
   return databaseRepository.getAllItems(User.class)
       .first(EMPTY)
       .map(user -> !user.equals(EMPTY) && user.getId() == JWTUtils.getIdentity(token));
}

 

We have added a default user EMPTY in the first operator so that RxJava returns it if there are no users in the database and then we simply map the user to a boolean denoting if they are same or different using the EMPTY user and getIdentity method from JWTUtils

Finally, we use all this information to implement our self contained login request:

eventService
   .login(new Login(username, password))
   .flatMapSingle(loginResponse -> {
       String token = loginResponse.getAccessToken();
       utilModel.saveToken(token);

       return isPreviousUser(token);
   })
   .flatMapCompletable(isPrevious -> {
       if (!isPrevious)
           return utilModel.deleteDatabase();

       return Completable.complete();
   });

 

Let’s see what is happening here. A request using username and password is made to the server which returns a login response containing a JWT, which we store for future use. Next, we flatMapSingle to the Single returned by the isPreviousUser method. And we finally clear the database if it is not a previous user.

Creating these self contained models help reduce complexity in presenter or view layer and all data is handled in one layer making presenter layer model agnostic.

To learn more about JWT and some of the Rx operators I mentioned here, please visit these links: