Author: vinaysajjanapu

In Android, image processing can be performed using Java or RenderScript or Native(C/C++). The performance of native code(C/C++) for image processing is much better than Java and RenderScript. So we used native code for image processing in the photo editor of the Phimpme image application. In this blog, I explain how image processing is performed in Phimpme Android.

Setting up build script for native code.

NDK helps us to develop Android applications using native languages like C and C++ so that heavy tasks can be performed in relatively less time. We can also use libraries built using C/C++ in Android application using this NDK. NDK can be downloaded using the SDK manager of Android Studio and can be set up following instructions in Android developers’ site.

After setting up the NDK, we will create a simple application involving native code and understand the flow of functions from Java to native.

The java files are present in app/src/main/java directory. Similarly, all the native files are present in app/src/main/jni directory.

So now let’s create necessary files in jni directory.

• main.c – Native functions are added here
• Android.mk and Application.mk – make files for building native code using ndkbuild.

Android.mk

LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_SRC_FILES := main.c       // include the files that should be 
                                // built(.c, .cpp, .h)
LOCAL_LDLIBS += -llog
LOCAL_MODULE := modulename      //name of the module (custom)
include $(BUILD_SHARED_LIBRARY)

Application.mk

APP_OPTIM := release
APP_ABI := all                    //architectures for which native lib
                               //has to be built(can be one or many. 
                               //should be separated by comma)
APP_PLATFORM := android-25 

Add these lines to app’s build.gradle for building native code along with the Gradle build.

externalNativeBuild {
   ndkBuild {
       path 'src/main/jni/Android.mk'
   }
}

The above lines in build.gradle will run Android.mk during the Gradle build.

When the Android.mk runs, it compiles all native code and generates modulename.so files in .externalNativeBuild/ndkBuild directory for all the mentioned architectures. This .so file for a particular architecture is a library containing all the native code compatible with that architecture.

So when this library(.so file) is statically imported into Java code, native code gets linked to Java and enables calling native functions directly from Java.

Importing native Library into Java

Static import of this library can be done by writing the below lines in your Java class.

static {
    System.loadLibrary("modulename");
}

Creating a native function and calling it from Java?

Unlike normal java code, where you call a function by its name, here in native Android development, the name of the native function is different from what you call it in java. To understand this clearly, let’s see an example of simple hello world application. Define the native function in Java and call it normally like any other function.

package org.fossasia.phimpme;

import …

public class MainActivity extends Activity {
   @Override
   public void onCreate(Bundle savedInstanceState) {
       super.onCreate(savedInstanceState);
       setContentView(R.layout.activity_main);       
       textView = (TextView)findViewById(R.id.textview);
       textView.setText(helloworld());
   }

    static {
        System.loadLibrary(“modulename”);
    }

     Public native String helloworld();
}

Here the definition of the native function is present in org.fossasia.phimpme package and MainActivity class. So the name of the function in the native file should be “Java_org_fossasia_phimpme_MainActivity_helloworld”.

It follows a general structure of concatenating these strings Java, package name, Class name, function name defined in Java and replacing all full stops(.) with underscores(_).

The first two arguments in native function are JNIEnv* and jobject. These are present always. For a zero argument definition of a function in Java, there will be these two arguments in the native function. If there are two arguments defined in Java, there will be two arguments additional to these two arguments in the native function. These help in passing data in and out of the native part.

Here’s an example of a native function to output a string to Java

main.c :

#include <jni.h>
jstring Java_org_fossasia_phimpme_MainActivity_helloworld  (JNIEnv * env, jobject obj){
    return (*env)->NewStringUTF(env, "Hello World from Native");
}

Now when you run this application, you see “Hello World from Native” displayed on the screen. I hope this post clears about the flow of the native functions and how to link them with Java.

References

https://developer.android.com/ndk/guides/index.html

The main feature in any image editor application like Phimpme Android other than editing the image is the ability to revert the changes (UNDO) that are done and the ability to revert the reversion i.e reperform the changes (REDO).

In Phimpme Application, we implemented this by using an ArrayList of Bitmaps. We stored the copy of the image bitmap whenever the modification is done on it. This helped us to get back to the previous image when required. But there is a problem in this method. They method may produce OutOfMemory Error when storing the bitmap in ArrayList when memory gets full. So for dealing solving this in the Phimpme application, we added a try – catch block and when the out of memory exception is caught, we recycled and removed the initial modified image from list i.e the image of index 1 in ArrayList. Index 0 is the original image on which we are working on. When we recycled that image, it gives space for adding another image, So added the recent image at the end of the list.

addToUndoList() function is shown below.

private void addToUndoList() {
   try{
       recycleBitmapList(++currentShowingIndex);
       bitmapsForUndo.add(mainBitmap.copy(mainBitmap.getConfig(),true));
   }catch (OutOfMemoryError error){
       bitmapsForUndo.get(1).recycle();
       bitmapsForUndo.remove(1);
       bitmapsForUndo.add(mainBitmap.copy(mainBitmap.getConfig(),true));
   }
}

In the above function, bitmapsForUndo is ArrayList in which we added modified bitmaps in the editor of Phimpme application. mainBitmap is the image bitmap on which all modifications are being done in the editor. The sense of integer variable currentShowingIndex is clear from its name that it points to the index of the image that is currently getting shown.

Eg. Consider a case when you perform 5 edits on an image using Phimpme Editor, then 6 image bitmaps get stored in the ArrayList including the original image and currentShowingIndex will be 5. Now if you undo the steps twice the currentShowingIndex becomes 3. The bitmaps of the index 4 and 5 have not been removed from the ArrayList yet. So they will be useful if you want to redo the changes.

When you make an another edit, an image bitmap gets added at index 4 and that should be the last element of the ArrayList. But you see that there is a bitmap of index 5 making that the last element of the ArrayList, not the newly added one. So in order to achieve that, the elements present in the ArrayList whose index is greater than currentShowingIndex have to recycled and removed before adding a newly modified image bitmap to the ArrayList. The first line in the try block of the above functions is referring to the function that is going to implement this. That function’s implementation is given below

private void recycleBitmapList(int fromIndex){
   while (fromIndex < bitmapsForUndo.size()){
       bitmapsForUndo.get(fromIndex).recycle();
       bitmapsForUndo.remove(fromIndex);
   }
}

Removing the bitmap from the ArrayList doesn’t clear the memory. That bitmap has to be recycled before getting removed from the ArrayList which is performed in the above function of the Phimpme application’s image editor.  The above recycleBitmapList function recycles and removes the bitmaps which have an index greater than or equal to the index that is passed as an argument to that function.

This function should also be called in onDestroy function of android activity as

recycleBitmapList(0);

This recycles and removes the whole ArrayList.

As now the implementation of the creation and recycling of the ArrayList is done, we can use this ArrayList to create getter functions for the undo and redo bitmaps. When the getUndoBitmap() is called the currentShowingIndex should decrement by one if greater than zero. When getRedoBitmap() is called the currentShowingIndex has to be incremented by one until it gets equal to the index of the last element present in the array list.

These methods are shown below.

private Bitmap getUndoBitmap(){
   if (currentShowingIndex - 1 >= 0)
       currentShowingIndex -= 1;
   else currentShowingIndex = 0;

   return bitmapsForUndo
           .get(currentShowingIndex)
           .copy(bitmapsForUndo.get(currentShowingIndex).getConfig(), true);
}

private Bitmap getRedoBitmap(){
   if (currentShowingIndex + 1 <= bitmapsForUndo.size())
       currentShowingIndex += 1;
   else currentShowingIndex = bitmapsForUndo.size() - 1;

   return bitmapsForUndo
           .get(currentShowingIndex)
           .copy(bitmapsForUndo.get(currentShowingIndex).getConfig(), true);
}

Logic part is done by here. We integrated these getter functions to button click functions of Phimpme Image Editor. setButtonVisibility() is called whenever undo or redo is button is pressed. This function sets the enable state and visibility of the button i.e the undo button is visible and enabled only if undo is possible. So does for the redo button.

setButtonVisibility() is shown below.

private void setButtonsVisibility() {
   if (currentShowingIndex > 0) {
       undo.setColorFilter(Color.BLACK);
       undo.setEnabled(true);
   }else {
       undo.setColorFilter(Color.GRAY);
       undo.setEnabled(false);
   }

   if (currentShowingIndex + 1 < bitmapsForUndo.size()) {
       redo.setColorFilter(Color.BLACK);
       redo.setEnabled(true);
   }else {
       redo.setColorFilter(Color.GRAY);
       redo.setEnabled(false);
   }
}

The above function grays the button if it is in the disabled state and will be black when the enabled state conditions are satisfied.

Finally, the OnClick() function of the editor of Phimpme is shown below.

@Override
public void onClick(View v) {
   switch (v.getId()){
       case R.id.edit_undo:
           onUndoPressed();
           break;
       case R.id.edit_redo:
           onRedoPressed();
           break;
   }
}

private void onUndoPressed() {
   if (mainBitmap != null) {
       if (!mainBitmap.isRecycled()) {
           mainBitmap.recycle();
       }
   }
   mainBitmap = getUndoBitmap();
   mainImage.setImageBitmap(mainBitmap);
   setButtonsVisibility();
}

private void onRedoPressed() {
   if (mainBitmap != null) {
       if (!mainBitmap.isRecycled()) {
           mainBitmap.recycle();
       }
   }
   mainBitmap = getRedoBitmap();
   mainImage.setImageBitmap(mainBitmap);
   setButtonsVisibility();
}

This shows how we implemented undo and redo in Image Editor of Phimpme Image Application.

When I was trying to give default gallery-like experience to the gallery of Phimpme Image Application, where you can zoom an image with pan and pinch controls along with the ability to navigate to another photo by swipe gestures, I faced a problem in which when the zoomed image is swiped expecting it to get panned, instead of that, the viewpager switched to another page.

This implementation of Viewpager with zoomable image in it might seem straightforward in the beginning but once you start implementing this in most common way i.e using default ViewPager for navigation between images and zooming libraries like TouchImageView, subsampling-scale-image-view or PhotoView for zooming the image with pinch and pan controls, you will notice that when you swipe left or right on the zoomed image, the pager navigates to other images instead of the zoomed image getting panned. The viewpager responds to the swipe event and causes page change and it doesn’t let zoomable view to respond to that event.

In the above screenshot, front image is zoomed and when we swipe left, instead of the image getting panned, it is switching to next page.

How to solve this?

As the problem is caused by the default ViewPager utilizing the swipe event without transferring it to child views, a custom viewpager can be created by extending default viewpager and having a touch intercept method which transfers the event to its child views. A sample implementation of this custom view pager which I used in the app is shown below.

public class CustomViewPager extends ViewPager {
    public CustomViewPager(Context context) {
        super(context);
    }

    public CustomViewPager(Context context, AttributeSet attrs)
    {
        super(context,attrs);
    }

    @Override
    public boolean onInterceptTouchEvent(MotionEvent ev) {
        try {
            return super.onInterceptTouchEvent(ev);
        } catch (IllegalArgumentException e) {
            return false;
        }
    }
}

Now we can replace normal viewpager with this custom view pager in image viewing activity and layout resource for that activity. The normal pager adapter which is used with default view pager can be used with this custom viewpager also. You can get a clear understanding of what I described here by having a look at the below implementation.

activity_imageview.xml

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout
   xmlns:android="http://schemas.android.com/apk/res/android"
   android:layout_width="match_parent"
   android:layout_height="match_parent">

   <package.name.path.to.CustomViewPager
       android:id="@+id/cviewpager"
       android:layout_width="match_parent"
       android:layout_height="match_parent"/>
</RelativeLayout>

ImageViewActivity.java

public class ImageViewActivity extends Activity {
    CustomViewPager cViewPager;
    ArrayList<String> imageList;
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_imageview);
        imageList = createList(); //some method for creating a list
        cViewPager = (CustomViewPager)findViewById(R.id.cviewpager);  
        mViewPager.setAdapter(new ImagePagerAdapter(imageList));
    }

    class ImagePagerAdapter extends PagerAdapter {
        ArrayList<String> imageList; 
        
        ImagePagerAdapter(ArrayList<String> imageList){
            this.imageList = imageList;
        }

        @Override
        public int getCount() {
            return (null != imageList) ? imageList.size() : 0;
        }

        @Override
        public boolean isViewFromObject(View view, Object object) {
            return view == object;
        }

        @Override
        public View instantiateItem(ViewGroup container, int position) {
            PhotoView photoView = new PhotoView(container.getContext());
            Glide.with(getContext())
                .load(UriFromFile(new File(imageList.get(position))))
                .asBitmap()
                .thumbnail(0.2f)
                .into(photoView);
            photoView.setMaximumScale(5.0F);
            photoView.setMediumScale(3.0F);
            container.addView(photoView, LayoutParams.MATCH_PARENT, LayoutParams.MATCH_PARENT);
            return photoView;
        }

        @Override
         public void destroyItem(ViewGroup container, int position, Object object) {
             container.removeView((View) object);
         }
     }
}

In the above pager adapter, the PhotoView library is used for a zoomable view. The image is loaded into photoview using image caching library Glide. The pager adapter here takes a List of paths to images on the device as the input argument. A simple method for creating such list had been discussed in my first post.

Here you can see that the on swiping left the zoomed image got panned.

This method of implementing zoomable view in ViewPager is used many gallery applications. One among those applications is LeafPic. We are now integrating that into Phimpme Image Application.

References:

https://developer.android.com/reference/android/support/v4/view/ViewPager.html

https://developer.android.com/training/gestures/viewgroup.html

https://github.com/MikeOrtiz/TouchImageView

https://github.com/chrisbanes/PhotoView

In the process of optimizing our Phimpme Android photo application, I had to change the implementation of the gallery in the application, as its loading time and smoothness is not satisfactory. How did I implement it?

In any gallery application, the primary necessity is having a details’ list of all available images on the device. For creating such a list, it is not necessary to iterate in all folders of the device storage for images. It is very time consuming task if we perform every time the app opens. Instead we can use database created by Android system for storing details of all available media on the device which can be accessed by any application. If we query the database with proper arguments, we get a cursor pointing to our target, using which we can form a list for our need.

Querying the mediastore for images can be implemented as shown below

Uri uri;
Cursor cursor;
int column_index;
String path = null,sortOrder;
ArrayList<String> imageList = new ArrayList<>();
uri = android.provider.MediaStore.Images.Media.EXTERNAL_CONTENT_URI;
String[] projection = { MediaStore.MediaColumns.DATA }; 
//DATA is the path to the corresponding image. We only need this for loading //image into a recyclerview

sortOrder = MediaStore.Images.ImageColumns.DATE_ADDED + “ DESC”;
//This sorts all images such that recent ones appear first

cursor = getContentResolver().query(uri, projection, null,null, sortOrder);

try {
       if (null != cursor) {
           column_index = cursor.getColumnIndexOrThrow(MediaStore.MediaColumns.DATA);
           while (cursor.moveToNext()) {
                 path = cursor.getString(column_index_data);
                 imageList.add(path);
            }
            cursor.close(); 
//imageList gets populated with paths to images by here
       }
}catch (Exception e){
       e.printStackTrace();
}

As we now have the list of paths to all images, we can proceed with displaying images using image file path. We can use RecyclerView for displaying grid of images as it is more optimized that default GridView. The whole process of displaying images can be summarized in following simple steps.

1. Add RecyclerView widget to main layout and create a layout for gallery item.
2. Create an adapter for populating RecyclerView.
3. Initialize adapter with imageList and attach it to the recyclerView.

Setting Layout: 

Add the below code to the layout resource for your activity(activity_main.xml).

<android.support.v7.widget.RecyclerView
android:layout_width=”match_parent”
android:layout_height=”match_parent”
android:id=”@+id/gallery_grid”/>

Inorder to use RecyclerView in the project, the following dependency for recyclerView has to be added in the build.gradle file

compile "com.android.support:recyclerview-v7:$supportLibVer"
//change supportLibVer to its corresponding value.

Now create a layout for item in the gallery. For this, you can create something like below.

gallery_item.xml

<?xml version="1.0" encoding="utf-8"?>
 <RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
               android:layout_width="match_parent"
               android:layout_height="match_parent">
     <ImageView
         android:layout_width="match_parent"
         android:layout_height="match_parent"
         android:scaleType="centerCrop"
         android:id="@+id/image"/>
 </RelativeLayout>

Setting the Adapter:

Adapter populates the recyclerView with the list of resources given to it. For displaying images in RecyclerView, if we use normal method for setting image i.e imageview.setImageDrawable(xyz); the recyclerView would load extremely slow everytime we open the app. So, we can use an open-source image caching library Glide, for this purpose. It caches the images on its first load and lazy loads images into imageView giving a smooth experience. It can be used only if the following dependency is added to build.gradle.

compile 'com.github.bumptech.glide:glide:4.0.0-RC0'

Its sample implementation is also shown in the adapter code below.

OnItemClickListener() cannot be used with recyclerView directly, like gridView. So, a method for implementing such kind of listener is also shown in below implementation of adapter.

import android.content.Context;
import android.net.Uri;
import android.support.v7.widget.RecyclerView;
import android.view.LayoutInflater;
import android.view.View;
import android.view.ViewGroup;
import android.widget.ImageView;
import android.widget.RelativeLayout;
import android.widget.Toast;
import com.bumptech.glide.Glide;
import java.io.File;
import java.util.ArrayList;

public class GalleryAdapter extends RecyclerView.Adapter<GalleryAdapter.mViewHolder> {

static ArrayList<String> galleryImageList;
private Context context;

public GalleryAdapter(ArrayList<String> imageList){
     galleryImageList = imageList;
}

public class mViewHolder extends RecyclerView.ViewHolder implements View.OnClickListener,View.OnLongClickListener {

     ImageView imageView;
     ItemClickListener itemClickListener;

     public mViewHolder(View view) {
         super(view);
         imageView = (ImageView)view.findViewById(R.id.image);
         view.setOnClickListener(this);
         view.setOnLongClickListener(this);
     }

     public void setClickListener(ItemClickListener itemClickListener) {
         this.itemClickListener = itemClickListener;
     }

     @Override
     public void onClick(View v) {
         itemClickListener.onClick(v, getPosition(), false);
     }

     @Override
     public boolean onLongClick(View v) {
         itemClickListener.onClick(v, getPosition(), true);
         return true;
     }
}

@Override
public GalleryAdapter.mViewHolder onCreateViewHolder(ViewGroup parent, int viewType) {
     context = parent.getContext();
     View view = LayoutInflater.from(context).inflate(R.layout.gallery_item,parent,false);

//this gallery item is the one that we created above.

     DisplayMetrics metrics = new DisplayMetrics();
getWindowManager().getDefaultDisplay().getMetrics(metrics);

     RelativeLayout.LayoutParams layoutParams = new RelativeLayout.LayoutParams(metrics.widthPixels/3,metrics.widthPixels/3);    //3 columns

     view.setLayoutParams(layoutParams);
     return new mViewHolder(view);
}

@Override
public void onBindViewHolder(GalleryAdapter.mViewHolder holder, final int position) {
     Glide.with(context)
             .load(Uri.fromFile(new File(galleryImageList.get(position)))
             //get path from list and covertin to URI
             .override(100,100) //final image will be this size(100x100)
             .thumbnail(0.1f)   //instead of empty placeholder, a                                              //thumbnail of 0.1th size of original image is used as //placeholder before complete loading of image
             .into(holder.imageView);
     holder.setClickListener(new ItemClickListener() {
         @Override
         public void onClick(View view,int position, boolean isLongClick) {
             if (isLongClick)
                 Toast.makeText(context, "Long Clicked " + position, Toast.LENGTH_SHORT).show();

             else
                 Toast.makeText(context, "Clicked " + position , Toast.LENGTH_SHORT).show();
         }
     });
}

@Override
public int getItemCount() {
     return (null != galleryImageList) ? galleryImageList.size() : 0;
}
}

With this, we reached the final step. 

Attaching the adapter to RecyclerView:

A grid layout manager with a required number of columns is made and attached to recyclerView.

The adapter is also initialized with the created Image List and is set to the recyclerView as shown below.

recyclerView = (RecyclerView)findViewById(R.id.gallery_grid);
RecyclerView.LayoutManager mLayoutManager = new GridLayoutManager(this,3);
// 3 columns
recyclerView.setLayoutManager(mLayoutManager);
recyclerView.setAdapter(new GalleryAdapter(createAndGetImageList()));

createAndGetImageList() is the first method discussed in post.