phimpme

In Android, image processing can be performed using Java or RenderScript or Native(C/C++). The performance of native code(C/C++) for image processing is much better than Java and RenderScript. So we used native code for image processing in the photo editor of the Phimpme image application. In this blog, I explain how image processing is performed in Phimpme Android.

Setting up build script for native code.

NDK helps us to develop Android applications using native languages like C and C++ so that heavy tasks can be performed in relatively less time. We can also use libraries built using C/C++ in Android application using this NDK. NDK can be downloaded using the SDK manager of Android Studio and can be set up following instructions in Android developers’ site.

After setting up the NDK, we will create a simple application involving native code and understand the flow of functions from Java to native.

The java files are present in app/src/main/java directory. Similarly, all the native files are present in app/src/main/jni directory.

So now let’s create necessary files in jni directory.

• main.c – Native functions are added here
• Android.mk and Application.mk – make files for building native code using ndkbuild.

Android.mk

LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_SRC_FILES := main.c       // include the files that should be 
                                // built(.c, .cpp, .h)
LOCAL_LDLIBS += -llog
LOCAL_MODULE := modulename      //name of the module (custom)
include $(BUILD_SHARED_LIBRARY)

Application.mk

APP_OPTIM := release
APP_ABI := all                    //architectures for which native lib
                               //has to be built(can be one or many. 
                               //should be separated by comma)
APP_PLATFORM := android-25 

Add these lines to app’s build.gradle for building native code along with the Gradle build.

externalNativeBuild {
   ndkBuild {
       path 'src/main/jni/Android.mk'
   }
}

The above lines in build.gradle will run Android.mk during the Gradle build.

When the Android.mk runs, it compiles all native code and generates modulename.so files in .externalNativeBuild/ndkBuild directory for all the mentioned architectures. This .so file for a particular architecture is a library containing all the native code compatible with that architecture.

So when this library(.so file) is statically imported into Java code, native code gets linked to Java and enables calling native functions directly from Java.

Importing native Library into Java

Static import of this library can be done by writing the below lines in your Java class.

static {
    System.loadLibrary("modulename");
}

Creating a native function and calling it from Java?

Unlike normal java code, where you call a function by its name, here in native Android development, the name of the native function is different from what you call it in java. To understand this clearly, let’s see an example of simple hello world application. Define the native function in Java and call it normally like any other function.

package org.fossasia.phimpme;

import …

public class MainActivity extends Activity {
   @Override
   public void onCreate(Bundle savedInstanceState) {
       super.onCreate(savedInstanceState);
       setContentView(R.layout.activity_main);       
       textView = (TextView)findViewById(R.id.textview);
       textView.setText(helloworld());
   }

    static {
        System.loadLibrary(“modulename”);
    }

     Public native String helloworld();
}

Here the definition of the native function is present in org.fossasia.phimpme package and MainActivity class. So the name of the function in the native file should be “Java_org_fossasia_phimpme_MainActivity_helloworld”.

It follows a general structure of concatenating these strings Java, package name, Class name, function name defined in Java and replacing all full stops(.) with underscores(_).

The first two arguments in native function are JNIEnv* and jobject. These are present always. For a zero argument definition of a function in Java, there will be these two arguments in the native function. If there are two arguments defined in Java, there will be two arguments additional to these two arguments in the native function. These help in passing data in and out of the native part.

Here’s an example of a native function to output a string to Java

main.c :

#include <jni.h>
jstring Java_org_fossasia_phimpme_MainActivity_helloworld  (JNIEnv * env, jobject obj){
    return (*env)->NewStringUTF(env, "Hello World from Native");
}

Now when you run this application, you see “Hello World from Native” displayed on the screen. I hope this post clears about the flow of the native functions and how to link them with Java.

References

https://developer.android.com/ndk/guides/index.html

The main feature in any image editor application like Phimpme Android other than editing the image is the ability to revert the changes (UNDO) that are done and the ability to revert the reversion i.e reperform the changes (REDO).

In Phimpme Application, we implemented this by using an ArrayList of Bitmaps. We stored the copy of the image bitmap whenever the modification is done on it. This helped us to get back to the previous image when required. But there is a problem in this method. They method may produce OutOfMemory Error when storing the bitmap in ArrayList when memory gets full. So for dealing solving this in the Phimpme application, we added a try – catch block and when the out of memory exception is caught, we recycled and removed the initial modified image from list i.e the image of index 1 in ArrayList. Index 0 is the original image on which we are working on. When we recycled that image, it gives space for adding another image, So added the recent image at the end of the list.

addToUndoList() function is shown below.

private void addToUndoList() {
   try{
       recycleBitmapList(++currentShowingIndex);
       bitmapsForUndo.add(mainBitmap.copy(mainBitmap.getConfig(),true));
   }catch (OutOfMemoryError error){
       bitmapsForUndo.get(1).recycle();
       bitmapsForUndo.remove(1);
       bitmapsForUndo.add(mainBitmap.copy(mainBitmap.getConfig(),true));
   }
}

In the above function, bitmapsForUndo is ArrayList in which we added modified bitmaps in the editor of Phimpme application. mainBitmap is the image bitmap on which all modifications are being done in the editor. The sense of integer variable currentShowingIndex is clear from its name that it points to the index of the image that is currently getting shown.

Eg. Consider a case when you perform 5 edits on an image using Phimpme Editor, then 6 image bitmaps get stored in the ArrayList including the original image and currentShowingIndex will be 5. Now if you undo the steps twice the currentShowingIndex becomes 3. The bitmaps of the index 4 and 5 have not been removed from the ArrayList yet. So they will be useful if you want to redo the changes.

When you make an another edit, an image bitmap gets added at index 4 and that should be the last element of the ArrayList. But you see that there is a bitmap of index 5 making that the last element of the ArrayList, not the newly added one. So in order to achieve that, the elements present in the ArrayList whose index is greater than currentShowingIndex have to recycled and removed before adding a newly modified image bitmap to the ArrayList. The first line in the try block of the above functions is referring to the function that is going to implement this. That function’s implementation is given below

private void recycleBitmapList(int fromIndex){
   while (fromIndex < bitmapsForUndo.size()){
       bitmapsForUndo.get(fromIndex).recycle();
       bitmapsForUndo.remove(fromIndex);
   }
}

Removing the bitmap from the ArrayList doesn’t clear the memory. That bitmap has to be recycled before getting removed from the ArrayList which is performed in the above function of the Phimpme application’s image editor.  The above recycleBitmapList function recycles and removes the bitmaps which have an index greater than or equal to the index that is passed as an argument to that function.

This function should also be called in onDestroy function of android activity as

recycleBitmapList(0);

This recycles and removes the whole ArrayList.

As now the implementation of the creation and recycling of the ArrayList is done, we can use this ArrayList to create getter functions for the undo and redo bitmaps. When the getUndoBitmap() is called the currentShowingIndex should decrement by one if greater than zero. When getRedoBitmap() is called the currentShowingIndex has to be incremented by one until it gets equal to the index of the last element present in the array list.

These methods are shown below.

private Bitmap getUndoBitmap(){
   if (currentShowingIndex - 1 >= 0)
       currentShowingIndex -= 1;
   else currentShowingIndex = 0;

   return bitmapsForUndo
           .get(currentShowingIndex)
           .copy(bitmapsForUndo.get(currentShowingIndex).getConfig(), true);
}

private Bitmap getRedoBitmap(){
   if (currentShowingIndex + 1 <= bitmapsForUndo.size())
       currentShowingIndex += 1;
   else currentShowingIndex = bitmapsForUndo.size() - 1;

   return bitmapsForUndo
           .get(currentShowingIndex)
           .copy(bitmapsForUndo.get(currentShowingIndex).getConfig(), true);
}

Logic part is done by here. We integrated these getter functions to button click functions of Phimpme Image Editor. setButtonVisibility() is called whenever undo or redo is button is pressed. This function sets the enable state and visibility of the button i.e the undo button is visible and enabled only if undo is possible. So does for the redo button.

setButtonVisibility() is shown below.

private void setButtonsVisibility() {
   if (currentShowingIndex > 0) {
       undo.setColorFilter(Color.BLACK);
       undo.setEnabled(true);
   }else {
       undo.setColorFilter(Color.GRAY);
       undo.setEnabled(false);
   }

   if (currentShowingIndex + 1 < bitmapsForUndo.size()) {
       redo.setColorFilter(Color.BLACK);
       redo.setEnabled(true);
   }else {
       redo.setColorFilter(Color.GRAY);
       redo.setEnabled(false);
   }
}

The above function grays the button if it is in the disabled state and will be black when the enabled state conditions are satisfied.

Finally, the OnClick() function of the editor of Phimpme is shown below.

@Override
public void onClick(View v) {
   switch (v.getId()){
       case R.id.edit_undo:
           onUndoPressed();
           break;
       case R.id.edit_redo:
           onRedoPressed();
           break;
   }
}

private void onUndoPressed() {
   if (mainBitmap != null) {
       if (!mainBitmap.isRecycled()) {
           mainBitmap.recycle();
       }
   }
   mainBitmap = getUndoBitmap();
   mainImage.setImageBitmap(mainBitmap);
   setButtonsVisibility();
}

private void onRedoPressed() {
   if (mainBitmap != null) {
       if (!mainBitmap.isRecycled()) {
           mainBitmap.recycle();
       }
   }
   mainBitmap = getRedoBitmap();
   mainImage.setImageBitmap(mainBitmap);
   setButtonsVisibility();
}

This shows how we implemented undo and redo in Image Editor of Phimpme Image Application.

When I was trying to give default gallery-like experience to the gallery of Phimpme Image Application, where you can zoom an image with pan and pinch controls along with the ability to navigate to another photo by swipe gestures, I faced a problem in which when the zoomed image is swiped expecting it to get panned, instead of that, the viewpager switched to another page.

This implementation of Viewpager with zoomable image in it might seem straightforward in the beginning but once you start implementing this in most common way i.e using default ViewPager for navigation between images and zooming libraries like TouchImageView, subsampling-scale-image-view or PhotoView for zooming the image with pinch and pan controls, you will notice that when you swipe left or right on the zoomed image, the pager navigates to other images instead of the zoomed image getting panned. The viewpager responds to the swipe event and causes page change and it doesn’t let zoomable view to respond to that event.

In the above screenshot, front image is zoomed and when we swipe left, instead of the image getting panned, it is switching to next page.

How to solve this?

As the problem is caused by the default ViewPager utilizing the swipe event without transferring it to child views, a custom viewpager can be created by extending default viewpager and having a touch intercept method which transfers the event to its child views. A sample implementation of this custom view pager which I used in the app is shown below.

public class CustomViewPager extends ViewPager {
    public CustomViewPager(Context context) {
        super(context);
    }

    public CustomViewPager(Context context, AttributeSet attrs)
    {
        super(context,attrs);
    }

    @Override
    public boolean onInterceptTouchEvent(MotionEvent ev) {
        try {
            return super.onInterceptTouchEvent(ev);
        } catch (IllegalArgumentException e) {
            return false;
        }
    }
}

Now we can replace normal viewpager with this custom view pager in image viewing activity and layout resource for that activity. The normal pager adapter which is used with default view pager can be used with this custom viewpager also. You can get a clear understanding of what I described here by having a look at the below implementation.

activity_imageview.xml

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout
   xmlns:android="http://schemas.android.com/apk/res/android"
   android:layout_width="match_parent"
   android:layout_height="match_parent">

   <package.name.path.to.CustomViewPager
       android:id="@+id/cviewpager"
       android:layout_width="match_parent"
       android:layout_height="match_parent"/>
</RelativeLayout>

ImageViewActivity.java

public class ImageViewActivity extends Activity {
    CustomViewPager cViewPager;
    ArrayList<String> imageList;
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_imageview);
        imageList = createList(); //some method for creating a list
        cViewPager = (CustomViewPager)findViewById(R.id.cviewpager);  
        mViewPager.setAdapter(new ImagePagerAdapter(imageList));
    }

    class ImagePagerAdapter extends PagerAdapter {
        ArrayList<String> imageList; 
        
        ImagePagerAdapter(ArrayList<String> imageList){
            this.imageList = imageList;
        }

        @Override
        public int getCount() {
            return (null != imageList) ? imageList.size() : 0;
        }

        @Override
        public boolean isViewFromObject(View view, Object object) {
            return view == object;
        }

        @Override
        public View instantiateItem(ViewGroup container, int position) {
            PhotoView photoView = new PhotoView(container.getContext());
            Glide.with(getContext())
                .load(UriFromFile(new File(imageList.get(position))))
                .asBitmap()
                .thumbnail(0.2f)
                .into(photoView);
            photoView.setMaximumScale(5.0F);
            photoView.setMediumScale(3.0F);
            container.addView(photoView, LayoutParams.MATCH_PARENT, LayoutParams.MATCH_PARENT);
            return photoView;
        }

        @Override
         public void destroyItem(ViewGroup container, int position, Object object) {
             container.removeView((View) object);
         }
     }
}

In the above pager adapter, the PhotoView library is used for a zoomable view. The image is loaded into photoview using image caching library Glide. The pager adapter here takes a List of paths to images on the device as the input argument. A simple method for creating such list had been discussed in my first post.

Here you can see that the on swiping left the zoomed image got panned.

This method of implementing zoomable view in ViewPager is used many gallery applications. One among those applications is LeafPic. We are now integrating that into Phimpme Image Application.

References:

https://developer.android.com/reference/android/support/v4/view/ViewPager.html

https://developer.android.com/training/gestures/viewgroup.html

https://github.com/MikeOrtiz/TouchImageView

https://github.com/chrisbanes/PhotoView