Category: Object detection, Event detection

Description

Dominant orientation templates (DOT) is a method for real-time object detection, which works well for detection of untextured objects and is related to method Histogram of oriented gradients. DOT is neither based on statistical learning of object’s shapes nor on feature point detection, but it uses real-time Template Matching recognition with locally most dominant orientations from HoG.

OpenCV function used

cvCaptureFromAVI, cvtColor

The process

1. Computation of gradients for each pixel in template and input image
1. Provided by convolution kernel
2. For each pixel
3. Gradient is defined by magnitude and direction
4. 0-180° instead of 0-360° range
5. Directions can be discretized from 0-180° into bins (e.g. 9 bins by 20° )

for (int r=0;r<=area.rows-region_size;r+=region_size)
{
    for (int s=0;s<=area.cols-region_size;s+=region_size)
    {
        int mag=gradienty_template.gradient[i][j].magnitude;
        n0=histogram_group(gradienty_template.gradient[i][j].direction);
        if (mag>min_magnitude)
            template_hist.hist_matrix[hx][hy].bins[n0]+=mag;
    }
}

2. Dividing pixels into regions

   

   for (int r=0;r<=area.rows-region_size;r+=region_size)
       // moving in the picture with step size of 7 or 9
   

3. Computing most dominant gradient orientations for each region

   

   if (template_hist.hist_matrix[i][j].bins[k]>max) //now only the most dominant
   {
       if  (template_hist.hist_matrix[i][j].bins[k]>min_magnitude)
       {
           max=template_hist.hist_matrix[i][j].bins[k];
           max_index=k;
       }
   }
   

4. Template matching and comparing of most dominant orientations.
5. Evaluating comparison.

   

   

Description

The main purpose of this work is to detect eyes and recognize when are open and when close. To execute this purpose we must use video camera or video file with person face.

Eye Detection

To detect eye blinking we need to recognize face and eyes on image. For this intention we use Viola Jones algorithm which detect this features and bounded it with rectangles.

Because this algorithm is performance consuming, we use tracking which is more faster. We use Good Features to Track algorithm which return set of points suitable to track, and then with Lucas-Kanade tracker algorithm we track it on every frame.

// track points
calcOpticalFlowPyrLK(prevGray, gray, features, cornersB, status, error, Size(31, 31), 1000);

if(!calculateIntegralHOG(gray(rectangleFace)))
    text = "CLOSED";

There is problem with points which is not precisely targeted to next frame, so we remove them from set of tracking points. When number of points is not enough to track, we repeat eye detection method again.

Eye Blinking Detection

To detect if eyes are open or close we use HOG descriptor which return array of floats representing lines orientation. Because HOG descriptor is usable only on images with specific resolution, we use sliding window with this resolution which covers our image.

cv::gpu::HOGDescriptor gpu_hog(win_size, Size(16, 16), Size(8, 8), Size(8, 8), 9, 0.8, 0.00015, true);

// calculate HOG for every window
GpuMat gpuMat;
gpuMat.upload(cropped);
GpuMat descriptors;
gpu_hog.getDescriptors(gpuMat, win_size, descriptors);
Mat descriptorMat = Mat(descriptors);

In next step we take array of floats returned from HOG descriptor and transform it to histogram. We notice, that when eye is closed the local maximum of this histogram is much lower than local maximum of opened eye, so we define the value which separate opened and closed eye.

Because we use sliding window, we average all this local maxims and based on returned value, we decide if specified area contains open or close eyes.

Description

The goal of this project is to implement algorithm that finds dots on dices. Motivation was idea / question how to create home-made random number generator? We can throw dices and our application will be able to recognize summary value on dices.

This program uses fitEllipse() function to find dots on dices. The basics steps are as follows:

Process

1. Open video stream

   CvCapture* capture = cvCaptureFromCAM( CV_CAP_ANY );
   

2. For the whole stream we create single frame

   IplImage* frame = cvQueryFrame( capture );
   

3. Invert color
4. Use adaptive threshold

   adaptiveThreshold(image, bimage, 255, ADAPTIVE_THRESH_GAUSSIAN_C, CV_THRESH_BINARY, 15, -10);
   

5. We can use morphological operations (dilatation, erosion) to expand/minimize contours
6. To find circles we use following:

   Mat pointsf;
   Mat(contours[i]).convertTo(pointsf, CV_32F);
   RotatedRect box = fitEllipse(pointsf);
   

7. If difference between box.size.width and box.size.height is lower than treshold, we consider ellipse as circle.
8. At this point we have a lot of “circles”. An experimenting helped us to determine, which circle in the picture is real point on dice. Based on size of real dices points we can isolate only real dices points.

Example of process

Using custom settings we are able to improve results in specific situations.

findContours(bimage, contours, CV_RETR_LIST, CV_CHAIN_APPROX_NONE);

Vector<pair<RotatedRect*, int>> vec, finalVec;
Mat cimage = Mat::zeros(bimage.size(), CV_8UC3);
int i;
int w, h, wAhThr, angleThr, centerThr;
int  hwDifferenceThreshold, histThreshold;						
centerThr = settCenterThr;
wAhThr = settwAhThr;
hwDifferenceThreshold = settHWdifferenceThr;
histThreshold = settHistogramThr;

for(i = 0; i < contours.size(); i++)
{
	size_t count = contours[i].size();
	if( count > 50 || count  < 6)
	      continue;

	Mat pointsf;
	Mat(contours[i]).convertTo(pointsf, CV_32F);
	RotatedRect box = fitEllipse(pointsf);

	w = box.size.width;
	h = box.size.height;

	int hwDifference = abs(h - w);
	if (hwDifference > hwDifferenceThreshold)
	      continue;

	if (w < wAhThr || h < wAhThr)
	      continue;

	vec.push_back(pair<RotatedRect*, int>(new RotatedRect(box), i));
}

int asdf = vec.size();
Vector<pair<RotatedRect* ,int>>::iterator it ,iend, it2;
int MAXHIST = 200;
int* histVals = new int[MAXHIST];
for (int i = 0; i < MAXHIST; i++)
	histVals[i] = 0;

int histIter = 0;
RotatedRect * box;
RotatedRect * box2;
int maxWidth = 0;
int distanceOfCenters;
for (it = vec.begin(), iend = vec.end(); it != iend; it++)
{
	box = (it->first);
	for (it2 = it + 1; it2 != iend; it2++)
	{
		box2 = (it2->first);
		distanceOfCenters = (int)std::sqrt((box->center.x - box2->center.x) * (box->center.x - box2->center.x) + (box->center.y - box2->center.y)  * (box->center.y - box2->center.y));
		if (distanceOfCenters < centerThr)
		{
			if (box->size.width > box2->size.width)
				it2->second = -1;
			else
				it->second = -1;
			break;
		}

	}
}

Description

The goal of this project is to implement algorithm that creates curvature scale space (CSS) image of given shape using OpenCV library. “The CSS image consists of several arch-shape contours representing the inflection points of the shape as it is smoothed. The maxima of these contours are used to represent a shape. The CSS representation is robust with respect to scale, noise and change in orientation.”[1]

Process

1. Find contour coordinates of given shape

findContours(im, contours, CV_RETR_LIST, CV_CHAIN_APPROX_NONE );

Following steps are repeated with increased sigma until there are no zero-crossing points:

2. Gaussian kernel is the base for upcoming steps:

   transpose(getGaussianKernel(width, sigma, CV_64FC1), G);
   

3. Curve evolution can be computed by convolution of contour points with Gaussian kernel. Smoothed contour is not needed for CSS computation; it is used only to visualize the process:

   filter2D(X, Xsmooth, X.depth(), G);
   filter2D(Y, Ysmooth, Y.depth(), G);
   

   

4. To compute 1^st and 2^nd derivation of contour points, Gaussian kernel derivations will be needed:

   Sobel(G, dG, G.depth(), 1, 0, 3);
   Sobel(G, ddG, G.depth(), 2, 0, 3);
   

5. Convolution of contour points using derivatives of Gaussian kernel. According to OpenCV documentation: filter2D does actually computes correlation, not the convolution. That is, the kernel is not mirrored around the anchor point. If you need a real convolution, flip the kernel using flip() and set the new anchor to (kernel.cols – anchor.x – 1, kernel.rows – anchor.y – 1) :

   flip(dg, dg, 0);
   flip(ddg, ddg, 0);
   Point anchor(dg.cols - fwhm -1, dg.rows - 0 - 1);
   filter2D(X, dX, X.depth(), dG, anchor);
   filter2D(Y, dY, Y.depth(), dG, anchor);
   filter2D(X, ddX, X.depth(), ddG, anchor);
   filter2D(Y, ddY, Y.depth(), ddG, anchor);
   

6. Finally, we calculate the curvature and find zero crossings:

   

7. Zero-crossing points are plotted to the final CSS image. X-axis represents position of point on the curve; Y-axis represents the value of sigma:
   

   

Practical Applications

• Finding similar shapes  (Used as shape descriptor in MPEG-7 standard)
• Corner detection

References

[1] Sadegh Abbasi, Farzin Mokhtarian, Josef Kittler: Curvature Scale Space Image in Shape Similarity Retrieval. Multimedia Syst. 7(6): 467-476 (1999)

[2] Farzin Mokhtarian, Alan K. Mackworth: A Theory of Multiscale, Curvature-Based Shape Representation for Planar Curves. IEEE Trans. Pattern Anal. Mach. Intell. 14(8): 789-805 (1992)

Description

Convolutional Neural Networks(CNNs) are multi-layered neural networks with standard hidden layers and with at least one convolutional layer. They are suitable for visual processing because they exploit the tolopogy of inputs.

Because we are interested in more general structures of networks and layers, the CNN is implemented with automatic differentiation (AD) in mind. This means that one has to provide only the implementation of forward pass of any structure. It is important to note that AD is based on certainly different principle as finite differences. The important difference of AD is that it yields precise results.

Architecture

The architecture of convolutional networks is shown in figure 1. Convolutional layers are located right after inputs. After each convolutional layer the process of subsampling is performed. By subsampling we improve translational invariance and significantly reduce complexity. After convolutional layers, standard fully connected hidden layers are used. These are then mapped to desired outputs.

When the signal passes into standard hidden layer, it is no longer meaningful to pass it again into another convolutional layers. However it can be meaningful to have heterogeneous layers consisting of convolutional and also standard units.

Experiments

Note that it is meaningful for input to be two-dimensional. We have used CNN for handwritten digits recognition of US Postal Service.

• 2,5% human error rate
• 2,0% best error rate: combination of multiple classifiers
• 4,7% best achievement of our CNN

We note, that CNNs are sensitive to parameter settings including number and size of convolutional kernels. However, when properly set, CNNs perform well. We can see an example output of convolutional layers at figure 2.