OpenCV on Android: prac­tices and tips

Transfer Mat ob­jects from Android to NDK

The main idea is to use the ad­dress of a Mat ob­ject in or­der to ma­nip­u­late the data.

Basically, we have a func­tion act­ing as a bridge be­tween Java APIs and NDK:

public native void function_name(long matAddress);

To call the func­tion, we use Mat’s ad­dress by call­ing getNativeObjAddr(). All com­pu­ta­tions in NDK will af­fect the con­tent of Mat in both Java and NDK lay­ers.

In the NDK code, to use cv::Mat ob­ject re­gard­ing java Mat, we can use static_cast:

Mat& im = *(static_cast<Mat*>(addrImg));

Notes

There is a huge dif­fer­ence in im­age chan­nels be­tween Java OpenCV and NDK OpenCV. When we de­code the path or file to a bitmap in Java, we have to con­vert the bitmap to ARGB_8888 color chan­nel, oth­er­wise it does not work. Actually, it also can work on RGB_565 but for some rea­sons I can­not re­mem­ber, I al­ways use ARGB_8888 in the pro­ject.

Bitmap bm32_image = bm.copy(Bitmap.Config.ARGB_8888, true);

To ma­nip­u­late the im­age cor­rectly in NDK, we should con­vert it to the nor­mal RGB chan­nel, oth­er­wise some­times we get some bugs that are frus­trat­ing.

// convert ARGB_8888 to RGB
Mat im = new Mat();
Mat rgb_im = new Mat();
Utils.bitmapToMat(bm32_image, im);
Imgproc.cvtColor(im, rgb_im, Imgproc.COLOR_RGBA2RGB, 3);

function_name(rgb_im.getNativeObjAddr());

Another prac­tice is to put some as­ser­tions in NDK code to make sure that we use the cor­rect for­mat for the in­put.

OpenCV’s Camera

OpenCV sup­ports 3 types of cam­era:

• CameraBridgeViewBase.
• JavaCameraView.
• CameraSurfaceGLView.

JavaCameraView

1. Create the lay­out of the cam­era. For ex­am­ple, we can put the fol­low­ing lines to the Activity xml file:

<org.opencv.android.JavaCameraView
    android:layout_width="fill_parent"
    android:layout_height="fill_parent"
    android:id="@+id/opencvcamera_view"
></org.opencv.android.JavaCameraView>

2. Next, in the ac­tiv­ity which con­trols the cam­era, we have to im­ple­ment re­quired meth­ods from the CvCameraViewListener2. The 3 meth­ods are:

• onCameraViewStarted.
• onCameraViewStopped.
• onCameraFrame.

Prior to pro­cess­ing the cam­era, we need to ini­tial­ize the vari­able hold­ing call­backs of the three afore­men­tioned meth­ods:

private CameraBridgeViewBase mOpenCVCamera;

In the onCreate method:

mOpenCVCamera = (CameraBridgeViewBase) findViewById(R.id.opencvcamera_view);
mOpenCVCamera.setVisibility(CameraBridgeViewBase.VISIBLE);
mOpenCVCamera.setCvCameraViewListener(this);

Next, we im­ple­ment the 3 re­quired meth­ods:

@Override
public void onCameraViewStarted(int width, int height) {
    // initialize images, variables, settings
}

@Override
public void onCameraViewStopped() {
    // release the resources
}

@Override
public Mat onCameraFrame(CameraBridgeViewBase.CvCameraViewFrame inputFrame) {
    // retrieve the frame from `inputFrame`
    // - the grayscale frame by inputFrame.gray()
    // - the RGBA frame by inputFrame.rgba()
    Mat im = inputFrame.rgba();

    // do things
    // postprocess: convert back to the RGBA image
    return im; // `im` will show in the UI
}

Note that OpenCV’s Camera is not able to set the por­trait ori­en­ta­tion. One workaround is to set the Activity to landscape by putting the fol­low­ing line in­side the Activity tag in AndroidManifest.xml:

android:screenOrientation="landscape"

Data Manipulation

unsigned char Mat

It is trou­ble­some when we want to as­sign a value of 255 to an unsigned char Mat because Java does not sup­port unsigned char as a prim­i­tive type. One workaround is to al­lo­cate a CV_16S Mat, ma­nip­u­late that ma­trix, and fi­nally con­vert to CV_8U.

Point2f and Point

To con­vert MatOfPoint to MatOfPoint2f, we use the con­struc­tor:

MatOfPoint matofpoint = new MatOfPoint(matofpoint2f.toArray());

Accessing the pixel val­ues

In or­der to re­trieve and as­sign pixel val­ues, we use the get­ter/​set­ter from Mat.

short[] pixel = new short[nchannels];

m.get(i, j, pixel); // retrieve pixel values at (i, j)
// in this example, the pixel has 3 channels
pixel[0] = 255;
pixel[1] = 0;
pixel[2] = 125;
m.set(i, j, pixel);