Adding Face Recognition based Authentication to SUSI MagicMirror Module

SUSI MagicMirror Module is a module designed for MagicMirror that helps you get SUSI Intelligence right on your Mirror. You may then ask it questions in the way the Queen in the tale “Snow White and the Seven Dwarfs” asked. One key feature that was missing in it was that the user could be recognized and queries he asked are answered in a personalized manner. This could be achieved if SUSI uses the account dedicated to that person to answer his/her queries. Thus, we need an authentication support.

The authentication on MagicMirror is not as trivial as on Web, Android and iOS client apps for SUSI. Key difference here is that user, while using the MagicMirror, does not have access to a keyboard and mouse. Therefore, we cannot simply ask him to input email and password. Furthermore, a MagicMirror installed in your home may be used by several members of your family. Thus, we need a mechanism to tell each user apart.

This was done with the help of MMM-Facial-Recognition module which brings face recognition support to MagicMirror.

MMM-Facial-Recognition module provides support for recognizing multiple faces and setting the modules on the mirror screen based on the user facing the mirror using OpenCV. Other modules can also take advantage of knowing about the person with the help of module notifications sent by MMM-Facial-Recognition Module.

To add Face based Authentication support to SUSI with MMM-Facial-Recognition, we first need to add the latter to MagicMirror. It can be added easily by first cloning the repository to modules directory of MagicMirror.

$ git clone https://github.com/paviro/MMM-Facial-Recognition

Go inside the directory and install dependencies

$ npm run install

Now, we need to train a model for the users who are going to use the MagicMirror. This can be done by the MMM-Facial-Recognition-Tools. This tool captures photos from the camera and trains a model for Face Recognition. The guide to use the tool is very well written on the Github page so I am not including it here. After training for faces of the users, you will get a training.xml file. This file contains the information about the facial features of every person so that it can tell users apart. You need to copy this file to the Module directory for MMM-Facial-Recognition module i.e. MagicMirror/module/MMM-Facial-Recognition.

After this we can add the module to MagicMirror, by modifying the config file. Add the following lines in the config file (config.js). Copy and paster username array from the training script in the asked position.

{
    module: 'MMM-Facial-Recognition',
    config: {
        // 1=LBPH | 2=Fisher | 3=Eigen
        recognitionAlgorithm: 1,
        lbphThreshold: 50,
        fisherThreshold: 250,
        eigenThreshold: 3000,
        useUSBCam: true,
        trainingFile: 'modules/MMM-Facial-Recognition/training.xml',
        interval: 2,
        logoutDelay: 15,
        // Array with usernames (copy and paste from training script)
        users: [],
        defaultClass: "default",
        everyoneClass: "everyone",
        welcomeMessage: true
    }
}

You may configure the show and hide behavior of modules based on the person. Find more information about it in the official guide on the repository. After setting up it recognizes and shows welcome message to each user like this.

 

Now, we need to integrate this module to SUSI for Authentication. To do this first of all we make config for SUSI MagicMirror Module to add user authentication along with their name registered on Facial Recognition Module. It can be done by adding SUSI MagicMirror module config file (config.js) like below.

{
       module: "MMM-SUSI-AI",
       position: "top_center",
       config: {
            hotword: "Susi",
            users: [{
                face_recognition_username: "Pranjal Paliwal",
                email: "[email protected]",
                password: "PASSWORD_HERE"
            }, {
                face_recognition_username: "Chashmeet Singh",
                email: "[email protected]",
                password: "PASSWORD_HERE"
            }],
        },
        classes: 'default everyone'
},

Now, we need to know that which user is facing the mirror at that time. MMM-Facial-Recognition sends a module notification when a user is detected. The format of the notification is

sender : MMM-Facial-Recognition
type: CURRENT_USER
payload: Name of the User / None 

If the user is recognized we get the name of the User as payload. If no face could be identified, we get None as payload.

We need to find out user based on the user’s name registered in the module. We already have that parameter in the user object in users array in config for SUSI MagicMirror Module (MMM-SUSI-AI). We can iterate over users array to find out the user facing the mirror on receiving the notification. In SUSI Chat API, users are identified with the help of an access token. On identifying a user, we perform login with the help of SignInService to obtain token for him. The implementation of the above task can be understood via the following snippet.

public receivedNotification(type: NotificationType, payload: any): void {
   if (type === "CURRENT_USER") {
       console.log("Current User", payload);
       if (payload === "None") {
           this.configService.Config.accessToken = null;
       } else {
           console.log(this.config.users);
           for (const user of this.config.users) {
               if (user.face_recognition_username === payload) {
                   if (isUndefined(this.signInService)) {
                       this.signInService = new SignInService(user);
                   }
                   this.signInService.updateUser(user).then((token) => {
                       console.log("updating token for " + user);
                       this.configService.Config.accessToken = token;
                   });
                   return;
               }
           }
           this.configService.Config.accessToken = null;
       }
   }
}

Explanation: In the receivedNotification method of the Main Component of SUSI MagicMirror module, we check if notification is of type CURRENT_USER. If the payload is None, we set access-token to null. If a user is identified, we check if it is contained in the users array. If present, we perform Sign In to SUSI Server for that user and store the access token obtained in the Config.

Now, every time a recognized my Facial Recognition module, the access token is updated in the config. We use the accessToken field in Config to send the message to SUSI Chat API. The implementation of it can be referred below.

public async askSusi(query: string): Promise<any> {

   const accessToken = this.configService.Config.accessToken;

   const requestString: string = (!isUndefined(accessToken) && accessToken != null) ?
       `http://api.susi.ai/susi/chat.json?q=${query}&access_token=${accessToken}` :
       `http://api.susi.ai/susi/chat.json?q=${query}`;

   const response = await WebRequest.get(requestString);
   return JSON.parse(response.content);
}

By using the above approach, the request sent to SUSI Server are identified according to the person facing the mirror. SUSI can, therefore, answer according to the user. In this way, authentication with Face Recognition is performed in the SUSI Magic Mirror Module.

Resources

 

Using OpenCV for Native Image Processing in Phimpme Android

OpenCV is very widely used open-source image processing library. After the integration of OpenCV Android SDK in the Phimpme Android application, the image processing functions can be written in Java part or native part. Taking runtime of the functions into consideration we used native functions for image processing in the Phimpme application.

We didn’t have the whole application written in native code, we just called the native functions on the Java OpenCV Mat object. Mat is short for the matrix in OpenCV. The image on which we perform image processing operations in the Phimpme Android application is stored as Mat object in OpenCV.

Creating a Java OpenCV Mat object

Mat object of OpenCV is same whether we use it in Java or C++. We have common OpenCV object in Phimpme for accessing from both Java part and native part of the application. We have a Java bitmap object on which we have to perform image processing operations using OpenCV. For doing that we need to create a Java Mat object and pass its address to native. Mat object of OpenCV can be created using the bitmap2Mat() function present in the OpenCV library. The implementation is shown below.

Mat inputMat = new Mat(bitmap.getWidth(), bitmap.getHeight(), CvType.CV_8UC3);
Utils.bitmapToMat(bitmap, inputMat);

“bitmap” is the Java bitmap object which has the image to be processed. The third argument in the Mat function indicates that the Mat should be of type 8UC3 i.e. three color channels with 8-bit depth. With the second line above, the bitmap gets saved as the OpenCV Mat object.

Passing Mat Object to Native

We have the OpenCV Mat object in the memory. If we pass the whole object again to native, the same object gets copied from one memory location to another. In Phimpme application, instead of doing all that we can just get the memory location of the current OpenCV Mat object and pass it to native. As we have the address of the Mat, we can access it directly from native functions. Implementation of this is shown below.

Native Function Definition:

private static native void nativeApplyFilter(long inpAddr);

Native Function call:

nativeApplyFilter(inputMat.getNativeObjAddr());

Getting Native Mat Object to Java

We can follow the similar steps for getting the Mat from the native part after processing. In the Java part of Phimpme, we created an OpenCV Mat object before we pass the inputMat OpenCV Mat object to native for processing. So we have inputMat and outputMat in the memory before we send them to native. We get the memory locations of both the Mat objects and pass those addresses to native part. After the processing is done, the data gets written to the same memory location and can be accessed in Java. The above functions can be modified and rewritten for this purpose as shown below

Native Function Definition:

private static native void nativeApplyFilter(long inpAddr, long outAddr );

Native Function call:

nativeApplyFilter(inputMat.getNativeObjAddr(),outputMat.getNativeObjAddr());
inputMat.release();

if (outputMat !=null){
   Bitmap outbit = Bitmap.createBitmap(bitmap.getWidth(),bitmap.getHeight(),bitmap.getConfig());
   Utils.matToBitmap(outputMat,outbit);
   outputMat.release();
   return outbit;
}

Native operations on Mat using OpenCV

The JNI function in the native part of Phimpme application receives the memory locations of both the OpenCV Mat objects. As we have the addresses, we can create Mat object pointing that memory location and can be passed to processing functions for performing native operations just like all OpenCV functions. This implementation is shown below.

#include <jni.h>
#include <opencv2/core/core.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include "enhance.h"
using namespace std;
using namespace cv;

JNIEXPORT void JNICALL
Java_org_fossasia_phimpme_editor_editimage_filter_PhotoProcessing_nativeApplyFilter(JNIEnv *env, jclass type, jlong inpAddr,jlong outAddr) {
       Mat &src = *(Mat*)inpAddr;
       Mat &dst = *(Mat*)outAddr;
       applyFilter(src, dst);
}

applyFilter() function can have any image processing operation. The implementation of edge detection function using OpenCV in the Phimpme Android is shown below. We were able to do this in very few lines which otherwise would have needed an extremely large code.  

Mat grey,detected_edges;
cvtColor( src, grey, CV_BGR2GRAY );
blur( grey, detected_edges, Size(3,3) );
dst.create( grey.size(), grey.type() );
Canny( detected_edges, detected_edges, 70, 200, 3 );
dst = Scalar::all(0);
detected_edges.copyTo( dst, detected_edges);
 

  

The general structure of an OpenCV function which is necessary for implementing custom image processing operations can be understood by referring this below-mentioned brightness adjustment function.  

int x,y,bright;
cvtColor(src,src,CV_BGRA2BGR);
dst = Mat::zeros( src.size(), src.type() );
for (y = 0; y < src.rows; y++) {
   for (x = 0; x < src.cols; x++) {
       dst.at<Vec3b>(y, x)[0] =
                  saturate_cast<uchar>((src.at<Vec3b>(y, x)[0]) + bright);
       dst.at<Vec3b>(y, x)[1] =
               saturate_cast<uchar>((src.at<Vec3b>(y, x)[1]) + bright);
       dst.at<Vec3b>(y, x)[2] =
               saturate_cast<uchar>((src.at<Vec3b>(y, x)[2]) + bright);
   }
}

    

Resources:

Integrating OpenCV for Native Image Processing in Phimpme Android

OpenCV is an open source computer vision library written in C and C++. It has many pre-built image processing functions for all kinds of purposes. The filters and image enhancing functions which we implemented in Phimpme are not fully optimized. So we decided to integrate the Android SDK of OpenCV in the Phimpme Android application.

For integrating OpenCV in the application, the files inside the OpenCV Android SDK have to be properly placed inside the project. The build and make files of the project have to be configured correspondingly.

Here, in this post, I will mention how we fully integrated OpenCV Android SDK and configured the Phimpme project.

Setting Up

First, we downloaded the OpenCV-Android-SDK zip from the official release page here.

Now we extracted the OpenCV-Android-SDK.zip file to navigated to SDK folder to see that there are three folders inside it. etc, Java and native.

All the build files are present inside the native directory and are necessarily copied to our project. The java directory inside the SDK folder is an external gradle module which has to imported and added as a dependency to our project.

So we copied all the files from jni directory of SDK folder inside OpenCV-Android-SDK to jni directory of the Phimpme project inside main. Similarly copied the 3rdparty directory into the main folder of the Phimpme project. Libs folder should also be copied into the main folder of the Phimpme but it has to be renamed to jniLibs. The project structure can be viewed in the below image.

Adding Module to Project

Now we headed to Android Studio and right clicked on the app in the project view, to add a new module. In that window select import Gradle project and browse to the java folder inside the SDK folder of OpenCV-Android-SDK as shown in figures.

Now we opened the build.gradle file of app level (not project level) and added the following line as a dependency.

  compile project(':openCVLibrary320')

Now open the build.gradle inside the openCVLibrary320 external module and change the platform tools version and  SDK versions.

Configuring Native Build Files

We removed all files related to cmake be we use ndk-build to compile native code. After removing the cmake files, added following lines in the application.mk file

APP_OPTIM := release
APP_ABI := all
APP_STL := gnustl_static
APP_CPPFLAGS := -frtti -fexceptions
APP_PLATFORM := android-25

Now we further added the following lines to the top of Android.mk file

OPENCV_INSTALL_MODULES:=on
OPENCV_CAMERA_MODULES:=on
OPENCV_LIB_TYPE := STATIC
include $(LOCAL_PATH)/OpenCV.mk

The third line here is a flag indicating that the library has to be static and there is no need for external OpenCVLoader application.

Finally, find this line

OPENCV_LIBS_DIR:=$(OPENCV_THIS_DIR)/../libs/$(OPENCV_TARGET_ARCH_ABI)

in OpenCV.mk file and replace it with

OPENCV_LIBS_DIR:=$(OPENCV_THIS_DIR)/../jniLibs/$(OPENCV_TARGET_ARCH_ABI)

And finally, add the following lines to java part. So that the OpenCV is initialized and linked to application

static {
   if (!OpenCVLoader.initDebug()) {
       Log.e( TAG + " - Error", "Unable to load OpenCV");
   } else {
       System.loadLibrary("modulename_present_in_Android.mk");
   }
}

With this, the integration of OpenCV and the configuration of the project is completed. Remove the build directories i.e. .build and .externalNativeBuild directories inside app folder for a fresh build so that you don’t face any wrong build error.

Resources

Enhancing Images using Native functions in Phimpme Android

Enhancing the image can be performed by adjusting the brightness, contrast, saturation etc. of that image. In the Phimpme Android Image Application, we implemented many enhancement operations. All these image enhancement operations are performed by the native image processing functions.

An image is made up of color channels. A gray-scale image has a single channel, colored opaque image has three channels and colored image with transparency has four channels. Each color channel of an image represents a two dimensional matrix of integer values. An image of resolution 1920×1080 has 1920 elements in its row and 1080 such rows. The integer values present in the matrices will be ranging from 0 to 255. For a grayscale image there will be a single channel. So, for that image, 0 corresponds to black color and 255 corresponds to white color. By changing the value present in the matrices, the image can be modified.

The implementation of the enhancement functions in Phimpme Application are given below.

Brightness

Brightness adjustment is the easiest of the image processing functions in Phimpme. Brightness can be adjusted by increasing or decreasing the values of all elements in all color channel matrices. Its implementation is given below.

void tuneBrightness(Bitmap* bitmap, int val) {
  register unsigned int i;
  unsigned int length = (*bitmap).width * (*bitmap).height;
  unsigned char* red = (*bitmap).red;
  unsigned char* green = (*bitmap).green;
  unsigned char* blue = (*bitmap).blue;
  signed char bright = (signed char)(((float)(val-50)/100)*127);
  for (i = length; i--; ) {
       red[i] =  truncate(red[i]+bright);
       green[i] = truncate(green[i]+bright);
       blue[i] = truncate(blue[i]+bright);
  }
}

  

low brightness, normal, high brightness(in the order) images are shown above

For the above function, the argument val is given by the seekbar implemented in java activity. Its value ranges from 0 – 100, so a new variable is introduced to change the range of the input argument in the function. You can see that in the for loop there is function named truncate. As the name suggests it truncates the input argument’s value to accepted range. It is added to the top of the c file as below

#define truncate(x) ((x > 255) ? 255 : (x < 0) ? 0 : x)

Contrast

Contrast of an image is adjusted in Phimpme application by increasing the brightness of the brighter pixel and decreasing value of the darker pixel. This is achieved by using the following formula for the adjustment contrast in editor of phimpme application.

pixel[i] = {(259 x (C + 255))/(255 x (259 - C))} x (pixel[i] - 128)

In the above formula, C is the contrast value and pixel[i] is the value of the element in the image matrix that we are modifying for changing the contrast.

 

low contrast, normal, high contrast(in the order) images are shown above

So, after this formula for modifying every pixel value, the function looks like below

void tuneContrast(Bitmap* bitmap, int val) {
  register unsigned int i;
  unsigned int length = (*bitmap).width * (*bitmap).height;
  unsigned char* red = (*bitmap).red;
  unsigned char* green = (*bitmap).green;
  unsigned char* blue = (*bitmap).blue;
  int contrast = (int)(((float)(val-50)/100)*255);
  float factor = (float)(259*(contrast + 255))/(255*(259-contrast));

  for (i = length; i--; ) {
       red[i] = truncate((int)(factor*(red[i]-128))+128);
       green[i] = truncate((int)(factor*(green[i]-128))+128);
       blue[i] = truncate((int)(factor*(blue[i]-128))+128);
  }
}

Hue

The below image explains hue shift by showing what happens when shift in hue takes place over time. The image with hue 0 looks identical with image with hue 360. Hue shift is cyclic. The definition and formulae corresponding hue is found in wikipedia page here. Using that formulae and converting them back, i.e we got rgb values from hue in Phimpme application. Its implementation is shown below.

[img source:wikipedia]

void tuneHue(Bitmap* bitmap, int val) {
  register unsigned int i;
  unsigned int length = (*bitmap).width * (*bitmap).height;
  unsigned char* red = (*bitmap).red;
  unsigned char* green = (*bitmap).green;
  unsigned char* blue = (*bitmap).blue;
  double H = 3.6*val;
  double h_cos = cos(H*PI/180);
  double h_sin = sin(H*PI/180);
  double r,g,b;

  for (i = length; i--; ) {
       r = (double)red[i]/255;
       g = (double)green[i]/255;
       b = (double)blue[i]/255;
       red[i] = truncate((int)(255*((.299+.701*h_cos+.168*h_sin)*r +  (.587-.587*h_cos+.330*h_sin)*g + (.114-.114*h_cos-.497*h_sin)*b)));

       green[i] = truncate((int)(255*((.299-.299*h_cos-.328*h_sin)*r + (.587+.413*h_cos+.035*h_sin)*g + (.114-.114*h_cos+.292*h_sin)*b)));

       blue[i] = truncate((int)(255*((.299-.3*h_cos+1.25*h_sin)*r +  (.587-.588*h_cos-1.05*h_sin)*g + (.114+.886*h_cos-.203*h_sin)*b)));
  }
}

Saturation

Saturation is the colorfulness of the image. You can see the below null saturation, unmodified and high saturated images in the respective order. The technical definition and formulae for getting the saturation value from the rgb value is given in the wikipedia page here. In Phimpme application we used those formulae to get the rgb values from the saturation value.

Its implementation is given below.

  

low saturation, normal, high saturation(in the order) images are shown above

void tuneSaturation(Bitmap* bitmap, int val) {
  register unsigned int i;
  unsigned int length = (*bitmap).width * (*bitmap).height;
  unsigned char* red = (*bitmap).red;
  unsigned char* green = (*bitmap).green
  unsigned char* blue = (*bitmap).blue;
  double sat = 2*((double)val/100);
  double temp;
  double r_val = 0.299, g_val = 0.587, b_val = 0.114;
  double r,g,b;
  for (i = length; i--; ) {
      r = (double)red[i]/255;
      g = (double)green[i]/255;
      b = (double)blue[i]/255;
      temp = sqrt( r * r * r_val +
                     g * g * g_val +
                       b * b * b_val );
      red[i] = truncate((int)(255*(temp + (r - temp) * sat)));
      green[i] = truncate((int)(255*(temp + (g - temp) * sat)));
      blue[i] = truncate((int)(255*(temp + (b - temp) * sat)));
  }
}

Temperature

If the color temperature of the image is high, i.e the image with the warm temperature will be having more reds and less blues. For a cool temperature image reds are less and blues are more. So In Phimpme Application, we implemented this simply by adjusting the brightness of the red channel matrix and blue channel matrix as we did in brightness adjustment. We didn’t modify the green channel here.

  

low temperature, normal, high temperature(in the order) images are shown above

void tuneTemperature(Bitmap* bitmap, int val) {
  register unsigned int i;
  unsigned int length = (*bitmap).width * (*bitmap).height;
  unsigned char* red = (*bitmap).red;
  unsigned char* green = (*bitmap).green;
  unsigned char* blue = (*bitmap).blue;
  int temperature = (int)1.5*(val-50);
  for (i = length; i--; ) {
       red[i] = truncate(red[i] + temperature);
       blue[i] = truncate(blue[i] - temperature);
  }
}

Tint

In Phimpme application, we adjusted the tint of an image in the same way of adjusting the temperature. But in this instead of modifying the red and blue channels, we modified the green channel of the image. An image with more tint will have a tone of magenta color and if it is decreased the image will have a greenish tone. The below shown code shows how we implemented this function in image editor of Phimpme application.

  

low tint, normal, high tint(in the order) images are shown above

void tuneTint(Bitmap* bitmap, int val) {
  register unsigned int i;
  unsigned int length = (*bitmap).width * (*bitmap).height;
  unsigned char* red = (*bitmap).red;
  unsigned char* green = (*bitmap).green;
  unsigned char* blue = (*bitmap).blue;
  int tint = (int)(1.5*(val-50));

  for (i = length; i--; ) {
       green[i] = truncate(green[i] - tint);
  }
}

Vignette

Vignetting is the reduciton in the brightness of the image towards the edges than the center. It is applied to draw the attention of the viewer to the center of the image.

 

normal and vignetted images are shown above

For implementing vignette in Phimpme application, we reduced the brightness of the pixel corresponding to a radial gradient value which is generated based on the pixel’s distance from the corner and center. It’s function in Phimpme as is shown below.

double dist(int ax, int ay,int bx, int by){
   return sqrt(pow((double) (ax - bx), 2) + pow((double) (ay - by), 2));
}

void tuneVignette(Bitmap* bitmap, int val) {
  register unsigned int i,x,y;
  unsigned int width = (*bitmap).width, height = (*bitmap).height;
  unsigned int length = width * height;
  unsigned char* red = (*bitmap).red;
  unsigned char* green = (*bitmap).green;
  unsigned char* blue = (*bitmap).blue;
  double radius = 1.5-((double)val/100), power = 0.8;
  double cx = (double)width/2, cy = (double)height/2;
  double maxDis = radius * dist(0,0,cx,cy);
  double temp,temp_s;
   for (y = 0; y < height; y++){
       for (x = 0; x < width; x++ ) {
           temp = dist(cx, cy, x, y) / maxDis;
           temp = temp * power;
           temp_s = pow(cos(temp), 4);
           red[x+y*width] = truncate((int)(red[x+y*width]*temp_s));
           green[x+y*width] = truncate((int)(green[x+y*width]*temp_s));
           blue[x+y*width] = truncate((int)(blue[x+y*width]*temp_s));
       }
   }
}

All these above mentioned functions are called from main.c file by creating JNI functions corresponding to each. These JNI functions are further defined with proper name in Java and arguments are passed to it. If you are not clear with JNI, refer my previous posts.

Resources

Iteration through the Android File System in the phimpme project

Android uses the single file system structure which has a single root. The task involved creating a custom folder chooser to whitelist folders while displaying images in the gallery in the Phimpme Photo App. The challenge arose in iterating over the files in the most efficient way. The best possible way to represent the file structure is in the form of tree data structure as given below.

Current Alternative

Currently, the MediaStore class contains metadata for all available media on both internal and external storage devices. Since it only returns a list of a particular media file format, it refrains the developer from customizing the structure in his way.

Implementation

Create a public class which represents the file tree. Since each subtree of the tree could itself be represented as file tree itself, therefore the parent of a node will be a FileTree object itself. Therefore declare a list of FileTree objects as children of the node, a FileTree object as the parent of the particular node, node’s own File object along with string values filepath and display name associated with it.

public class FileTree {
 public final String filepath;
 public final String displayName;
 public final List<FileTree> childFileTreeList = new ArrayList<>();
 public final FileTree parent;
 public boolean hasMedia = false;

 public FileTree(String filepath, String displayName, FileTree parent) {
    this.filepath = filepath;
    this.displayName = displayName;
    this.parent = parent;
 }
}

For iterating through the file system, we create a recursive function which is called on the root of the Android file system. If the particular file is a directory, with the help of Depth First traversal algorithm, the directory is traversed. Else, the file is added to the list of the file. The below code snippet is the recursive function.

public static void walkDir(FileTree fileTree, List<File> files) {
  File listFile[] = fileTree.file.listFiles();

  if (listFile != null) {
      for (File file : listFile) {
          if (file.isDirectory()) {
              FileTree childFileTree = new FileTree(file, file.getName(), fileTree);
              fileTree.childFileTreeList.add(childFileTree);
              walkDir(childFileTree, files);
          }
          else {
                  files.add(file);

          }
      }
  }
}

Conclusion

The android file system was used to whitelist folders so that the images of the folders could neither be uploaded nor edited.

For the complete guide to whitelisting folders, navigate here

Native Functions for Performing Image Processing in Phimpme Android

In Android, image processing can be performed using Java or RenderScript or Native(C/C++). The performance of native code(C/C++) for image processing is much better than Java and RenderScript. So we used native code for image processing in the photo editor of the Phimpme image application. In this blog, I explain how image processing is performed in Phimpme Android.

Setting up build script for native code.

NDK helps us to develop Android applications using native languages like C and C++ so that heavy tasks can be performed in relatively less time. We can also use libraries built using C/C++ in Android application using this NDK. NDK can be downloaded using the SDK manager of Android Studio and can be set up following instructions in Android developers’ site.

After setting up the NDK, we will create a simple application involving native code and understand the flow of functions from Java to native.

The java files are present in app/src/main/java directory. Similarly, all the native files are present in app/src/main/jni directory.

So now let’s create necessary files in jni directory.

  • main.c – Native functions are added here
  • Android.mk and Application.mk – make files for building native code using ndkbuild.

Android.mk

LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_SRC_FILES := main.c       // include the files that should be 
                                // built(.c, .cpp, .h)
LOCAL_LDLIBS += -llog
LOCAL_MODULE := modulename      //name of the module (custom)
include $(BUILD_SHARED_LIBRARY)

Application.mk

APP_OPTIM := release
APP_ABI := all                    //architectures for which native lib
                               //has to be built(can be one or many. 
                               //should be separated by comma)
APP_PLATFORM := android-25 

Add these lines to app’s build.gradle for building native code along with the Gradle build.

externalNativeBuild {
   ndkBuild {
       path 'src/main/jni/Android.mk'
   }
}

The above lines in build.gradle will run Android.mk during the Gradle build.

When the Android.mk runs, it compiles all native code and generates modulename.so files in .externalNativeBuild/ndkBuild directory for all the mentioned architectures. This .so file for a particular architecture is a library containing all the native code compatible with that architecture.

So when this library(.so file) is statically imported into Java code, native code gets linked to Java and enables calling native functions directly from Java.

Importing native Library into Java

Static import of this library can be done by writing the below lines in your Java class.

static {
    System.loadLibrary("modulename");
}

Creating a native function and calling it from Java?

Unlike normal java code, where you call a function by its name, here in native Android development, the name of the native function is different from what you call it in java. To understand this clearly, let’s see an example of simple hello world application. Define the native function in Java and call it normally like any other function.

package org.fossasia.phimpme;

import 

public class MainActivity extends Activity {
   @Override
   public void onCreate(Bundle savedInstanceState) {
       super.onCreate(savedInstanceState);
       setContentView(R.layout.activity_main);       
       textView = (TextView)findViewById(R.id.textview);
       textView.setText(helloworld());
   }

    static {
        System.loadLibrary(“modulename”);
    }

     Public native String helloworld();
}

Here the definition of the native function is present in org.fossasia.phimpme package and MainActivity class. So the name of the function in the native file should be “Java_org_fossasia_phimpme_MainActivity_helloworld”.

It follows a general structure of concatenating these strings Java, package name, Class name, function name defined in Java and replacing all full stops(.) with underscores(_).

The first two arguments in native function are JNIEnv* and jobject. These are present always. For a zero argument definition of a function in Java, there will be these two arguments in the native function. If there are two arguments defined in Java, there will be two arguments additional to these two arguments in the native function. These help in passing data in and out of the native part.

Here’s an example of a native function to output a string to Java

main.c :

#include <jni.h>
jstring Java_org_fossasia_phimpme_MainActivity_helloworld  (JNIEnv * env, jobject obj){
    return (*env)->NewStringUTF(env, "Hello World from Native");
}

Now when you run this application, you see “Hello World from Native” displayed on the screen. I hope this post clears about the flow of the native functions and how to link them with Java.

References

https://developer.android.com/ndk/guides/index.html

Implementing UNDO and REDO in Image Editor of Phimpme Android

The main feature in any image editor application like Phimpme Android other than editing the image is the ability to revert the changes (UNDO) that are done and the ability to revert the reversion i.e reperform the changes (REDO).

In Phimpme Application, we implemented this by using an ArrayList of Bitmaps. We stored the copy of the image bitmap whenever the modification is done on it. This helped us to get back to the previous image when required. But there is a problem in this method. They method may produce OutOfMemory Error when storing the bitmap in ArrayList when memory gets full. So for dealing solving this in the Phimpme application, we added a try – catch block and when the out of memory exception is caught, we recycled and removed the initial modified image from list i.e the image of index 1 in ArrayList. Index 0 is the original image on which we are working on. When we recycled that image, it gives space for adding another image, So added the recent image at the end of the list.

addToUndoList() function is shown below.

private void addToUndoList() {
   try{
       recycleBitmapList(++currentShowingIndex);
       bitmapsForUndo.add(mainBitmap.copy(mainBitmap.getConfig(),true));
   }catch (OutOfMemoryError error){
       bitmapsForUndo.get(1).recycle();
       bitmapsForUndo.remove(1);
       bitmapsForUndo.add(mainBitmap.copy(mainBitmap.getConfig(),true));
   }
}

In the above function, bitmapsForUndo is ArrayList in which we added modified bitmaps in the editor of Phimpme application. mainBitmap is the image bitmap on which all modifications are being done in the editor. The sense of integer variable currentShowingIndex is clear from its name that it points to the index of the image that is currently getting shown.

Eg. Consider a case when you perform 5 edits on an image using Phimpme Editor, then 6 image bitmaps get stored in the ArrayList including the original image and currentShowingIndex will be 5. Now if you undo the steps twice the currentShowingIndex becomes 3. The bitmaps of the index 4 and 5 have not been removed from the ArrayList yet. So they will be useful if you want to redo the changes.

  

When you make an another edit, an image bitmap gets added at index 4 and that should be the last element of the ArrayList. But you see that there is a bitmap of index 5 making that the last element of the ArrayList, not the newly added one. So in order to achieve that, the elements present in the ArrayList whose index is greater than currentShowingIndex have to recycled and removed before adding a newly modified image bitmap to the ArrayList. The first line in the try block of the above functions is referring to the function that is going to implement this. That function’s implementation is given below

private void recycleBitmapList(int fromIndex){
   while (fromIndex < bitmapsForUndo.size()){
       bitmapsForUndo.get(fromIndex).recycle();
       bitmapsForUndo.remove(fromIndex);
   }
}

Removing the bitmap from the ArrayList doesn’t clear the memory. That bitmap has to be recycled before getting removed from the ArrayList which is performed in the above function of the Phimpme application’s image editor.  The above recycleBitmapList function recycles and removes the bitmaps which have an index greater than or equal to the index that is passed as an argument to that function.

This function should also be called in onDestroy function of android activity as

recycleBitmapList(0);

This recycles and removes the whole ArrayList.

As now the implementation of the creation and recycling of the ArrayList is done, we can use this ArrayList to create getter functions for the undo and redo bitmaps. When the getUndoBitmap() is called the currentShowingIndex should decrement by one if greater than zero. When getRedoBitmap() is called the currentShowingIndex has to be incremented by one until it gets equal to the index of the last element present in the array list.

These methods are shown below.

private Bitmap getUndoBitmap(){
   if (currentShowingIndex - 1 >= 0)
       currentShowingIndex -= 1;
   else currentShowingIndex = 0;

   return bitmapsForUndo
           .get(currentShowingIndex)
           .copy(bitmapsForUndo.get(currentShowingIndex).getConfig(), true);
}

private Bitmap getRedoBitmap(){
   if (currentShowingIndex + 1 <= bitmapsForUndo.size())
       currentShowingIndex += 1;
   else currentShowingIndex = bitmapsForUndo.size() - 1;

   return bitmapsForUndo
           .get(currentShowingIndex)
           .copy(bitmapsForUndo.get(currentShowingIndex).getConfig(), true);
}

Logic part is done by here. We integrated these getter functions to button click functions of Phimpme Image Editor. setButtonVisibility() is called whenever undo or redo is button is pressed. This function sets the enable state and visibility of the button i.e the undo button is visible and enabled only if undo is possible. So does for the redo button.

setButtonVisibility() is shown below.

private void setButtonsVisibility() {
   if (currentShowingIndex > 0) {
       undo.setColorFilter(Color.BLACK);
       undo.setEnabled(true);
   }else {
       undo.setColorFilter(Color.GRAY);
       undo.setEnabled(false);
   }

   if (currentShowingIndex + 1 < bitmapsForUndo.size()) {
       redo.setColorFilter(Color.BLACK);
       redo.setEnabled(true);
   }else {
       redo.setColorFilter(Color.GRAY);
       redo.setEnabled(false);
   }
}

The above function grays the button if it is in the disabled state and will be black when the enabled state conditions are satisfied.

Finally, the OnClick() function of the editor of Phimpme is shown below.

@Override
public void onClick(View v) {
   switch (v.getId()){
       case R.id.edit_undo:
           onUndoPressed();
           break;
       case R.id.edit_redo:
           onRedoPressed();
           break;
   }
}

private void onUndoPressed() {
   if (mainBitmap != null) {
       if (!mainBitmap.isRecycled()) {
           mainBitmap.recycle();
       }
   }
   mainBitmap = getUndoBitmap();
   mainImage.setImageBitmap(mainBitmap);
   setButtonsVisibility();
}

private void onRedoPressed() {
   if (mainBitmap != null) {
       if (!mainBitmap.isRecycled()) {
           mainBitmap.recycle();
       }
   }
   mainBitmap = getRedoBitmap();
   mainImage.setImageBitmap(mainBitmap);
   setButtonsVisibility();
}

This shows how we implemented undo and redo in Image Editor of Phimpme Image Application.

Implementing a zoomable ImageView by Extending the Default ViewPager in Phimpme Android

When I was trying to give default gallery-like experience to the gallery of Phimpme Image Application, where you can zoom an image with pan and pinch controls along with the ability to navigate to another photo by swipe gestures, I faced a problem in which when the zoomed image is swiped expecting it to get panned, instead of that, the viewpager switched to another page.

This implementation of Viewpager with zoomable image in it might seem straightforward in the beginning but once you start implementing this in most common way i.e using default ViewPager for navigation between images and zooming libraries like TouchImageView, subsampling-scale-image-view or PhotoView for zooming the image with pinch and pan controls, you will notice that when you swipe left or right on the zoomed image, the pager navigates to other images instead of the zoomed image getting panned. The viewpager responds to the swipe event and causes page change and it doesn’t let zoomable view to respond to that event.

In the above screenshot, front image is zoomed and when we swipe left, instead of the image getting panned, it is switching to next page.

How to solve this?

As the problem is caused by the default ViewPager utilizing the swipe event without transferring it to child views, a custom viewpager can be created by extending default viewpager and having a touch intercept method which transfers the event to its child views. A sample implementation of this custom view pager which I used in the app is shown below.


public class CustomViewPager extends ViewPager {
    public CustomViewPager(Context context) {
        super(context);
    }

    public CustomViewPager(Context context, AttributeSet attrs)
    {
        super(context,attrs);
    }

    @Override
    public boolean onInterceptTouchEvent(MotionEvent ev) {
        try {
            return super.onInterceptTouchEvent(ev);
        } catch (IllegalArgumentException e) {
            return false;
        }
    }
}

Now we can replace normal viewpager with this custom view pager in image viewing activity and layout resource for that activity. The normal pager adapter which is used with default view pager can be used with this custom viewpager also. You can get a clear understanding of what I described here by having a look at the below implementation.

activity_imageview.xml

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout
   xmlns:android="http://schemas.android.com/apk/res/android"
   android:layout_width="match_parent"
   android:layout_height="match_parent">

   <package.name.path.to.CustomViewPager
       android:id="@+id/cviewpager"
       android:layout_width="match_parent"
       android:layout_height="match_parent"/>
</RelativeLayout>

ImageViewActivity.java

public class ImageViewActivity extends Activity {
    CustomViewPager cViewPager;
    ArrayList<String> imageList;
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_imageview);
        imageList = createList(); //some method for creating a list
        cViewPager = (CustomViewPager)findViewById(R.id.cviewpager);  
        mViewPager.setAdapter(new ImagePagerAdapter(imageList));
    }

    class ImagePagerAdapter extends PagerAdapter {
        ArrayList<String> imageList; 
        
        ImagePagerAdapter(ArrayList<String> imageList){
            this.imageList = imageList;
        }

     [email protected]
        public int getCount() {
            return (null != imageList) ? imageList.size() : 0;
        }

       [email protected]
        public boolean isViewFromObject(View view, Object object) {
            return view == object;
        }

        @Override
        public View instantiateItem(ViewGroup container, int position) {
            PhotoView photoView = new PhotoView(container.getContext());
            Glide.with(getContext())
                .load(UriFromFile(new File(imageList.get(position))))
                .asBitmap()
                .thumbnail(0.2f)
                .into(photoView);
            photoView.setMaximumScale(5.0F);
            photoView.setMediumScale(3.0F);
            container.addView(photoView, LayoutParams.MATCH_PARENT, LayoutParams.MATCH_PARENT);
            return photoView;
        }

        @Override
         public void destroyItem(ViewGroup container, int position, Object object) {
             container.removeView((View) object);
         }
     }
}

In the above pager adapter, the PhotoView library is used for a zoomable view. The image is loaded into photoview using image caching library Glide. The pager adapter here takes a List of paths to images on the device as the input argument. A simple method for creating such list had been discussed in my first post.

Here you can see that the on swiping left the zoomed image got panned.

This method of implementing zoomable view in ViewPager is used many gallery applications. One among those applications is LeafPic. We are now integrating that into Phimpme Image Application.

References:

https://developer.android.com/reference/android/support/v4/view/ViewPager.html

https://developer.android.com/training/gestures/viewgroup.html

https://github.com/MikeOrtiz/TouchImageView

https://github.com/chrisbanes/PhotoView

 

Adding Multiple Themes in Phimpme Android

In Phimpme-Android we decided to add a new feature that is providing multiple themes to the users. We have 3 types of themes in Phimpme Dark Theme, Light Theme and Amoled Theme. In this post, I am explaining how I added multiple themes support in phimpme android.

 Choose Theme in Phimpme Dialog

You need a Helper class that will store the data about the theme and all the colors related to a theme.

Before you begin you need to create a Helper class. In Phimpme I created a Helper class as ThemeHelper

public class ThemeHelper {

 public static final int DARK_THEME = 2;
 public static final int LIGHT_THEME = 1;
 public static final int AMOLED_THEME = 3;

 private PreferenceUtil SP;
 private Context context;

 private int baseTheme;
 private int primaryColor;
 private int accentColor;

 public ThemeHelper(Context context) {
  this.SP = PreferenceUtil.getInstance(context);
  this.context = context;
  updateTheme();
 }
}

Which contains all the basic method to get colors for textview, icon, toolbar, switch, imageview, background and app primary color.

Now you to provide user to select theme and it can be done using dialog box. Once the user selected any of the theme we have to update that theme and it can be done by following code :

 public void updateTheme(){
  this.primaryColor = SP.getInt(context.getString(R.string.preference_primary_color),
        getColor(R.color.md_light_blue_300));
  this.accentColor = SP.getInt(context.getString(R.string.preference_accent_color),
        getColor(R.color.md_light_blue_500));
  baseTheme = SP.getInt(context.getString(R.string.preference_base_theme), LIGHT_THEME);
 }

Now we have updated the our theme in Phimpme now we have to set the color according to a theme.

To get colors of all components we need to add some function in our helper class which will provide us the colors according to the theme.

As I said we are having 3 themes in Phimpme so I used 3 case to compare which theme user has selected.

I have added the functions to get colors for background, text and subtext as follows in phimpme.

public int getBackgroundColor(){
  int color;
  switch (baseTheme){
    case DARK_THEME:color = getColor(R.color.md_dark_background);break;
    case AMOLED_THEME:color = getColor(R.color.md_black_1000);break;
    case LIGHT_THEME:
    default:color = getColor(R.color.md_light_background);
  }
  return color;
 }



 public int getTextColor(){
  int color;
  switch (baseTheme){
    case DARK_THEME:color = getColor(R.color.md_grey_200);break;
    case AMOLED_THEME:color = getColor(R.color.md_grey_200);break;
    case LIGHT_THEME:
    default:color = getColor(R.color.md_grey_800);
  }
  return color;
 }

 public int getSubTextColor(){
  int color;
  switch (baseTheme){
    case DARK_THEME:color = getColor(R.color.md_grey_400);break;
    case AMOLED_THEME:color = getColor(R.color.md_grey_400);break;
    case LIGHT_THEME:
    default:color = getColor(R.color.md_grey_600);
  }
  return color;
 }

In the above functions, I am comparing which theme user has selected and returned the color according to the theme.

Now set the color to text by using above function you don’t need care which theme user has selected because those function will check and return the color according to the theme.

So it can be done simply,

textview.setTextColor(getTextColor());
editText.setTextColor(getTextColor());
editText.setHintTextColor(getSubTextColor());

                                     Light Theme &  Dark Theme (Phimpme)

Resources:

https://github.com/fossasia/phimpme-android/blob/development/app/src/main/java/org/fossasia/phimpme/leafpic/util/ThemeHelper.java

http://www.hidroh.com/2015/02/16/support-multiple-themes-android-app/

Preparing a release for Phimpme Android

Most of the essential features are now in a stable state in our Phimpme Android app. So we decided to release a beta version of the app. In FOSSASIA we follow branch policy where in development all current development will take place and in master branch the stable code resides.

Releasing an app is not just building an apk and submitting to the distribution platform, certain guidelines should follow.

How I prepare a released apk for Phimpme

List down the feature

We discussed on our public channel what features are now in stable state and can be released. Features such as account manager and Share Activity is excluded because it is not complete and in under development mode. We don’t want to show and under development feature. So excluded them. And made a list of available features in different category of Camera, Gallery and Share.

Follow the branch policy.

The releasable and stable codebase should be on master branch. It is good to follow the branch policy because it helps if we encounter any problem with the released apk. We can directly go to our master branch and check there. Development branch have very volatile because of active development going on.

Every Contributor’s contribution is important

When we browse our old branches such as master in case of ours. We see generally it is behind 100s of commits to the development. In case of that when we create a PR for that then it generally contains all the old commits to make the branch up to the latest.

In this case while opening and merging do not squash the commits.

Testing from Developer’s end

Testing is very essential part in development. Before releasing it is a good practice that Developer should test the app from their end. We tested app features in different devices with varying Android OS version and screen size.

  • If there is any compatibility issue, report it right away and there are several tools in Android to fix.
  • Support variety of devices and screen sizes

Changing package name, application ID

Package name, application ID are vitals of an app. Which uniquely identifies them as app in world. Like I changed the package name of Phimpme app to org.fossasia.phimpme. Check all the permission app required.

Create Release build type

Build types are great to way categorize the apps. Debug and Release are two. There are various things in codebase which we want only in Debug modes. So when we create Release mode it neglect that part of the code.

Add build types in you application build.gradle

buildTypes {
   release {
       minifyEnabled false
   }

Rebuild app again and verify from the left tab bar

Generate Signed apk and Create keystore (.jks) file

Navigate to Build → Generate Signed APK

Fill all details and proceed further to generate the signed apk in your home directory.

Adding Signing configurations in build.gradle

Copy the keystore (.jks) file to the root of the project and add signing configurations

signingConfigs {
       config {
           keyAlias 'phimpme'
           keyPassword 'phimpme'
           storeFile file('../org.fossasia.phimpme.jks')
           storePassword 'XXXXXXX'
       }
   }

InstallRelease Gradle task

Navigate to the right sidebar of Android Studio click on Gradle


Click on installRelease to install the released apk. It take all the credentials from the signing configurations.

Resources