Lab 7 - Video Processing

Summary

In this lab, you will learn how to process video with the OpenCV library and track objects using KCF (Kernalized Correlation Filters) tracker. A full example solution can be found here.

Downloads

Part 1 - KCF Tracking Overview

There are a variety of correlation filter based tracking algorithms. Despite the different tricks these methods employ to improve tracking accuracy and computation speed, the ideas behind them are similar, as shown in the block diagram below.

  • Generate Target Filter - The first step is to generate a proper filter defined by the pattern of the target object. This step is often called the training phase, as the tracker examine both the positive examples (image area of target object) and negative examples and then come up with a filter that will produce large response when the target is present

  • Compute Correlation Response - The next step is to apply the filter to the incoming frames and compute the correlation response, aka convolve the new frame with the filter generated in the first step. Similar to auto-correlation, this step is dramatically boosted by using 2D-FFT and 2D-iFFT.

  • Finalize Tracking Result - The final step is to estimate the new position and scale of the target object based on the correlation response results.

  • KCF Tracking - KCF is unique for it uses the area around the target object to generate negative patterns in the training phase and take advantage of circulant matrices to speed up computation.

Note

We will be using OpenCV API for KCF trackers, so you don't need to implement the algorithm in detail. However, you should understand the basic mechanism of KCF tracking, since it might appear in the quiz or the final quiz!

Part 2 - OpenCV Overview

OpenCV (Open Source Computer Vision Library) is released under a BSD license and hence it’s free for both academic and commercial use. It has C++, C, Python and Java interfaces and supports Windows, Linux, Mac OS, iOS and Android. OpenCV was designed for computational efficiency and with a strong focus on real-time applications. Written in optimized C/C++, the library can take advantage of multi-core processing. Enabled with OpenCL, it can take advantage of the hardware acceleration of the underlying heterogeneous compute platform.

Adopted all around the world, OpenCV has more than 47 thousand people of user community and estimated number of downloads exceeding 14 million. Usage ranges from interactive art, to mines inspection, stitching maps on the web or through advanced robotics.

OpenCV & OpenCV_contrib

OpenCV has two major packages, standard OpenCV Library and extra modules in OpenCV_contrib. The KCF tracker we will be using is contained in the tracking package from OpenCV_contrib, so make sure to install OpenCV_contrib.

For python, use pip install opencv_contrib_python to insall the latest OpenCV library with contrib modules included.

For Android, import the SDK package that is compiled with contrib modules. You can either download them here or build from source to utilize the latest updates.

Online Documentation

  • OpenCV official online Documentation is here. The syntax are for C++ since the source code is written in C++.
  • OpenCV official online Documentation for Android&Java is here
  • OpenCV 2.* Documentation is here. We don't use OpenCV 2.* in our labs, but the APIs are mostly the same and this documentation is more user friendly.

In the past, students complained a lot about the lack of OpenCV documentation for Java and Android, and the situation hasn't change much even today. Searching for OpenCV Java/Android APIs directly tend to be difficult, however there is no lack of C++/Python documentation for OpenCV. Since OpenCV Java/Python is merely an wrapper around the OpenCV C++ source code, the APIs are extremely similar. It is often that case that one can infer the Java API easily by combining the documentation for C++/Python and the wrapper function declaration in Java.

Tip

Android Studio and many other major IDEs support "Go to Declaration" to explore the declaration of the function you are calling.

Part 3 - Program Workflow

This tracking app is an interactive one, meaning that the user needs to press button to control the behavior of the app. The general workflow is as follows.

  • Open the camera and start previewing. The user uses the scroll bar on the right to modify window color and size. The window represents the ROI (region of interest). tracking_flag is -1, meaning that the app is not tracking.
  • When the user presses Button, tracking_flag is set to 0 (if not currently tracking), meaning the app should initialize tracking. The scroll bar is hidden since modifying ROI size is not allowed once tracking has started.
  • When tracking_flag is 0, create a KCF tracker and initialize the tracker with the current frame and ROI. tracking_flag is then set to 1, meaning the app has started tracking.
  • When tracking_flag is 1, call the KCF tracker to update the tracking result. Draw a rectangle around the target if tracking is successful, or display the text "Tracking failure occured!" on the top left of the frame if the target is missing. Calculate and display the text of FPS value (eg."FPS@30") on the top right of the frame.
  • When the user presses Button again, tracking_flag is then set back to -1 (if the app has started tracking), meaning the app has stopped tracking. Set the ROI back to the center of the frame and clear the KCF tracker.

Python

Your task for the Python portion of this lab will be to prototype your Android system. You will be given python source code and test clip to start with. Make sure you have python OpenCV with contib installed.

Note

The python part follows the same workflow with minor changes. The program can open either the computer webcam or load a video clip. Pressing the button is simulated by pressing ENTER. User can select the ROI by pressing ENTER and draw an rectangle in the pop-up window. User can stop current tracking by pressing ENTER again. Program will be terminated whenever ESC is pressed.

Part 4 - OpenCV Video API

We utilize the OpenCV video packages to extract frames from video clip or webcam. The following lines will try to open a video stream.

1
2
3
4
5
6
7
8
 
# Load video, 0 for webcam, str for path to video
video = cv2.VideoCapture(0)
# video = cv2.VideoCapture('test_clip.mp4')

# Exit if video not opened.
if not video.isOpened():
    print('Could not open video!')
    sys.exit()

Once the video is opened successfully, we will read it frame by frame with the following loop which simulates the Android part when the camera preview is delivered frame by frame.

1
2
3
4
5
6
7
8
 
# Loop simulate Camera Preview Callback
while True:

    # Read Next frame.
    read_success, frame = video.read()
    if not read_success:
        print('Cannot read video file!')
        sys.exit()

The middle part of code is where you need to complete. Use OpenCV functions to draw text and calculate FPS (frames per second). We have created the variables for you here.

1
2
3
4
5
 
# Tracker Variables
tracker = None
roi = (0, 0, 0, 0)
# -1 for not tracking, 0 for init tracking, 1 for update tracking
tracking_flag = -1

Part 5 - KCF Tracker in Python

The completed program should be able to track simple objects that move smoothly, in particular, the referee's head in the provided test_clip.mp4. Useful functions include cv2.TrackerKCF_create(), cv2.putText(), cv2.getTickFrequency().

Assignment

Implement the Python system described above. In particular, you must show:

  • Rectangle around the target object
  • Text "Tracking failure occured!" if target object is missing
  • Text "FPS@"+fps (eg."FPS@30") when tracking
  • Correct program behavior described above

This will count for two demo points.

Question

What is the FPS of your tracker? What effect will the ROI size have on FPS?

Android

For this lab, we will be implementing in Java like lab 6. The procedure is exactly the same with Python part. Your code will reside in java\ece420\lab7.

Part 6 - Import OpenCV

We have imported OpenCV for this lab, this section is provided for reference.

  1. Download OpenCV Android SDK. This lab uses OpenCV 3.3.1 Android SDK with contrib. Unzip into './OpenCV Android SDK'.

  2. From Android Studio import OpenCV into your project as a module: File -> New -> Import_Module -> Select './OpenCV Android SDK/sdk/java' Module name: Android studio automatically fills Module name with openCVLibrary331dev (the exact name depends on the version you download). Click on next. You get a screen with three checkboxes and questions about jars, libraries and import options. All three should be checked. Click on 'Finish'. Android Studio starts to import the module and you are shown an import-summary.txt file that has a list of what was not imported (mostly javadoc files) and other pieces of information.

  3. Open the project structure dialogue as File -> Project_Structure. Select the app module, click on the Dependencies tab and add openCVLibrary331dev as a Module Dependency. When you select Add/Module_Dependency it should appear in the list of modules you can add. It will now show up as a dependency.

  4. At this point the module should appear in the project hierarchy as openCVLibrary331dev, you can navigate in its java directory to examine the Java wrapper APIs. You should be able to import and use OpenCV in your source code.

  5. However, the library source code is still not included into your final app build, any user that attempts to run your app will be asked to install OpenCV Manager on their device. To avoid this, copy the './OpenCV Android SDK/sdk/native/libs' directory (and everything under it) to your Android project, to './OpenCVLibrary331dev/src/main/, and then rename the copied folder from libs to jniLibs.

  6. Go to the onCreate method of MainActivity.java and append this code:

1
2
3
4
5
 
if (!OpenCVLoader.initDebug()) {
    Log.e(this.getClass().getSimpleName(), "  OpenCVLoader.initDebug(), not working.");
} else {
    Log.d(this.getClass().getSimpleName(), "  OpenCVLoader.initDebug(), working.");
}

  1. (Optional) Navigate to './OpenCVLibrary331dev/build.gradle' and modify compileSdkVersion, minSdkVersion, targetSdkVersion to match the values in './app/build.gradle'.

  2. Clean and rebuild project as OpenCV is ready for your project.

Part 7 - KCF Tracker in Android

Use OpenCV packages to complete the android app that has the same workflow as described above. The completed program should be able to track simple objects that moves smoothly. We recommend using the grayscale frame mGray for tracking and only use mRgba for display and preview purposes.

Assignment 2

Implement the Python system described above. In particular, you must show:

  • Rectangle around the target object
  • Text "Tracking failure occured!" if target object is missing
  • Text "FPS@"+fps (eg."FPS@30") when tracking
  • Correct program behavior described above

This will count for two demo points.

Attention

Android devices tend to have limited performance when battery is low. It is because the system is designed to turn off a lot of hardware resources when the battery gets too low so that the device will last for longer. Make sure your tablet is fully charged before demo or your app might be slower than your algorithm should expect.

Part 8 - Get Ready for Assigned Project Labs

Starting next week, you will be working on the assigned project labs, consult the assigned project lab page for more information. Note that the project proposal will be due soon.

Submission Instruction

Refer to the Submission Instruction page for more information.

Grading

Your lab will be graded as follows:

  • Prelab [2 points]

  • Lab [4 points]

    • Python:

      • Draw rectangle around the target object [0.5 point]

      • Add text "Tracking failure occured!" if target object is missing [0.5 point]

      • Add text "FPS@"+fps (eg."FPS@30") when tracking [0.5 point]

      • Correct program behavior [0.5 point]

    • Android:

      • Draw rectangle around the target object [0.5 point]

      • Add text "Tracking failure occured!" if target object is missing [0.5 point]

      • Add text "FPS@"+fps (eg."FPS@30") when tracking [0.5 point]

      • Correct program behavior [0.5 point]

  • Quiz (2 points)