Category: Tracking in video

Pavol Zbell

Introduction

In our work we focus on basics of motion analysis and object tracking. We compare MeanShift (non-parametric, finds an object on a back projection image) versus CamShift (continuously adaptive mean shift, finds an object center, size, and orientation) algorithms and effectively utilize them to perform simple object tracking. In case these algorithms fail to track the desired object or the object travels out of window scope, we try to find another object to track. To achieve this, we use a background subtractor based on a Gaussian Mixture Background / Foreground Segmentation Algorithm  to identify the next possible object to track. There are  two suitable implementations of this algorithm in OpenCV – BackgroundSubtractorMOG and BackgroundSubtractorMOG2. We also compare performance of both these implementations.

Used functions: calcBackProject, calcHist, CamShift, cvtColor, inRange, meanShift, moments, normalize

Solution

1. Initialize tracking window:
   • Set tracking window near frame center
2. Track object utilizing MeanShift / CamShift
   • Calculate HSV histogram of region of interest (ROI) and track

   int dims = 1;
   int channels[] = {0};
   int hist_size[] = {180};
   float hranges[] = {0, 180};
   const float *ranges[] = {hranges};
   roi = frame(track_window);
   cvtColor(roi, roi_hsv, cv::COLOR_BGR2HSV);
   // clamp > H: 0 - 180, S: 60 - 255, V: 32 - 255
   inRange(roi_hsv, Scalar(0.0, 60.0, 32.0), Scalar(180.0, 255.0, 255.0), mask);
   calcHist (&roi_hsv, 1, channels, mask, roi_hist, dims, hist_size, ranges);
   normalize(roi_hist, roi_hist, 0, 255, NORM_MINMAX);
   ...
   Mat hsv, dst;
   cvtColor(frame, hsv, cv::COLOR_BGR2HSV);
   calcBackProject(&hsv, 1, channels, roi_hist, dst, ranges, 1);
   clamp_rect(track_window, bounds);
   print_rect("track-window", track_window);
   Mat result = frame.clone();
   if (use_camshift) {
   	RotatedRect rect = CamShift(dst, track_window, term_criteria);
   	draw_rotated_rect(result, rect, Scalar(0, 0, 255));
   }
   else {
   	meanShift(dst, track_window, term_criteria);
   	draw_rect(result, track_window, Scalar(0, 0, 255));
   }
   

3. Lost tracked object?
   • In other words, is the centeroid of MOG mask out of tracking window?

   bool contains;
   if (use_camshift) {
   	contains = rect.boundingRect().contains(center);
   }
   else {
   	contains = center.inside(track_window);
   }
   

4. When lost, reinitialize tracking window:
   • Set tracking window to centeroid of MOG mask
   • Go back to 2. and repeat

   mog->operator()(frame, mask);
   center = compute_centroid(mask);
   track_window = RotatedRect(Point2f(center.x, center.y), Size2f(100, 50), 0).boundingRect();
   

Samples

As seen on Fig. 1, MeanShift (left) operates with fixed size tracking windows which can not be rotated. On the contrary, CamShift (right) utilizes the full potential of dynamic size rotated rectangles. Working with CamShift yielded significantly better tracking results in general. On the other hand we recommend to use MeanShift when the object is in constant distance from the camera and moves without rotation (or is represented by a circle), in such case MeanShift performs faster than CamShift and produces sufficient results without any rotation or size change noise.

Comparison of BackgroundSubtractorMOG and BackgroundSubtractorMOG2 is depicted on Fig. 2. MOG approach is simpler than MOG2 as it considers only binary masks whereas MOG2 operates on a full gray scale masks. Experiments shown that in our specific case MOG performed better as it yielded less information noise than MOG2. MOG2 will probably produce better results than MOG when utilized more effectively than in out initial approach (simple centeroid from mask extraction).

Summary

In this project explored the possibilities of simple object tracking via OpenCV APIs utilizing various algorithms such as MeanShift and CamShift, Background Extractor MOG and MOG2, which we also compared. Our solution performs relatively well, but we can certainly improve it by fine tuning histogram calculation, MOG, and other parameters. Other improvements can be done in MOG usage, as now the objects are only recognized by finding MOG mask centeroids. This also calls to better tracking window initialization process.

This example shows how to separate and track moving object using OpenCV. First, the background of the video is being calculated and moving objects detected, then it is filtered and tracked.

Used: cv::BackgroundSubtractorMOG2; cv::getStructuringElement; cv::morphologyEx; cv::BackgroundSubtractorMOG2.operator();

The process

1. Initialize the background extraction object

   BackgroundSubtractorMOG2 bg( 500, 64, false);
   

2. Process video frame by background extraction object by it’s method operator and receive mask of moving object

   bg.operator()( origi, mask);
   

   

3. Process mask by morphologyEx’s open to remove noise in mask

   morphologyEx( mask, mask, MORPH_OPEN, element1 );
   

4. Process mask by morphologyEx’s close to close gaps in mask

   morphologyEx( mask, mask, MORPH_CLOSE, element2 );
   

   

5. Apply mask on video frame

   origi.copyTo( proci0, mask);
   

6. Find good features to track and apply KLTracker

   goodFeaturesToTrack( proci0, points[0], MAX_COUNT, 0.1, 10, Mat(), 3, 0, 0.04);
   calcOpticalFlowPyrLK( proci1, proci0, points[0], points[1], status, err, winSize, 3, termcrit, 0, 0.00001);
   

   

7. Combine with initial frame

   size_t i, k;
   for (i = k = 0; i < points[1].size(); i++)
   {
   	if (!status[i]) continue;
   	points[1][k++] = points[1][i];
   	circle(finI, points[1][i], 3, Scalar(0, 255, 0), -1, 8);
   	circle(procI2, points[1][i], 3, Scalar(0, 255, 0), -1, 8);
   }
   points[1].resize(k);
   

   

8. Find contours of the mask, find it’s bounding rectangle and draw it onto output frame

   findContours(procI3, contours, CV_RETR_LIST, CV_CHAIN_APPROX_SIMPLE);
   if (contours.size() > 0)
   for (int i = 0; i < (int)contours.size(); i++)
   	rectangle(finI, boundingRect(contours[i]), Scalar(0, 255, 0));
   

   

Bounding rectangle hints position of moving object on the scene and could be used to approximate it’s coordinates

The goal of this project is to implement algorithm that segments foreground using OpenCV library. We assume that background is static, objects in foreground are moving and video is taken from static camera. We detect moving  vehicles (foreground) with 2 methods.

Background detection

First method is computing an average image from video frames.

1. Each frame is added into accumulator with a certain small weight (0.05 and smaller)
   1. accumulateWeighted(frame, background, alpha);
   2. at this point we have in Mat background actual backgorund image
2. To detect foreground we have to compute difference between current frame and current accumulated background image
   1. absdiff(frame, background, diff);
   2. in Mat diff is color image, we need to transform it into grayscale image
3. To detect relevant changes (more than given threshold) we use simple
   1. thresholdthreshold(diff, foregroundMask, 20, 255, CV_THRESH_BINARY);
   2. in Mat foregroundMask is foreground mask
4. We can use morphological operations (dilatation, erosion) to expand foreground region

Each steps (1-4) are illustrated in next figures

Frames history

Second method is compare current frame with older frame. It is enough if the stack is 5-20 elements large- it depends on the speed of vehicles.

1. Add current frame into stack (if stack is full, first element is erased and the rest is shifted)
   1. framesHistory.add(frame);
2. Compute difference between current frame and first element in the stack
   1. first = framesHistory.first();
   2. absdiff(frame, first, diff);
3. To detect relevant changes (more than given threshold) we use simple threshold
   1. threshold(diff, foregroundMask, 20, 255, CV_THRESH_BINARY);
   2. in Mat foregroundMask is foreground mask
4. We can use morphological operations (dilatation, erosion) to expand foreground region

Each steps (1-4) are illustrated in next figures. LocalBackgound is the older frame (5 frames old)

Combination of methods

These 2 methods are combined to create more accurate output. In the next picture we can see that first way (computing average image) creates “tails” behind vehicles. Comparing with older frame doesn’t create „tails“.

We use simple sum of binary masks

Mat sumMask = mask1 & mask2

How to create more precise segmentation, future work

To create more precise vehicle segmentation we have to use another methods. Shadows of cars, lights of cars make these 2 methods to hard use. We can only segment region where could be a car, but for more precisely detection we have to use template matching (champfer matching), graph cut method. In this project we experimented with these other two, but these were too complex, so the time complexity was unacceptable to use in video.