Category: Object detection, Event detection

This project focuses on the usage of computer vision within the field of board games. We propose a new approach for extracting the position of game board, which consists of the detection of empty fields based on the contour analysis and elipse fitting, locating the key points by using probabilistic Hough lines and of finding the homography by using these key points.

Functions used: Canny, findContours, fitEllipse, HoughLinesP, findHomography, warpPerspective, chamerMatching

Input

The process

1. Canny edge detector

   Mat canny;
   Canny(img, canny, 100, 170, 3);
   

2. Contour analysis – extraction contours and filtering out those that don’t match our criteria

   vector<vector<Point>> contours;
   vector<Vec4i> hierarchy;
   findContours(canny, contours, hierarchy, CV_RETR_TREE, CV_CHAIN_APPROX_NONE);
   

3. Ellipse fitting – further analysis of contours, final extraction of empty fields

   RotatedRect e = fitEllipse(contours[i]);
   

4. Extraction of the game board model – 4 key points are needed for locating this model
5. Locating the key points in the input image – using Hough lines & analysing their intersections

   Mat grayCpy;
   vector<Vec4i>& lines;
   HoughLinesP(grayCpy, lines, 1, CV_PI/180, 26, 200, 300);
   

6. Finding the homography and final projection of the game board model into the input image

   findHomography(Mat(modelKeyPoints), Mat(keyPoints));
   warpPerspective(modelImg, newImg, h, Size(imgWithEmptyFieldsDots.cols, imgWithEmptyFieldsDots.rows), CV_INTER_LINEAR + CV_WARP_FILL_OUTLIERS);
   chamerMatching(canny, piece, results, costs, 1.0, 30, 1.0, 3, 3, 5, 0.9, 1.1);
   

Sample

Result

Mat findImageContours(const Mat& img, vector<vector<Point> >& contours, vector<Vec4i>& hierarchy)
{
    // detect edges using canny:
    Mat canny;
    Canny(img, canny, 100, 170, 3);

    findContours(canny, contours, hierarchy, CV_RETR_TREE, CV_CHAIN_APPROX_NONE);

    // draw contours:   
    Mat imgWithContours = Mat::zeros(canny.size(), CV_8UC3);
    for (unsigned int i = 0; i < contours.size(); i++)
    {
        // process "holes" only:
        if (hierarchy[i][3] == -1) continue;
        // apply ratio + size + contourArea filters:
        if (!checkContour3(contours[i])) continue;

        // fit and draw ellipse:
        RotatedRect e = fitEllipse(contours[i]);
        if (e.size.height < 50)
        {
            line(imgWithContours, e.center, e.center, Scalar(255, 255, 255),3);
        }
    }
    return imgWithContours;
}

This project shows a possible way of finding contour lines on maps. These properties of the contour lines are considered here:

• contour lines are closed or they end at the edges of the map,
• in some sections more neighbor contour lines are nearly parallel,
• they are mainly slightly curved only (the lines do not have large angles like roads or buildings).

The algorithm uses the OpenCV library.

Functions used: cv::medianBlur, cv::Sobel, cv::magnitude

The process

1. Image preprocessing – using median blur

   cv::Mat bl;
   cv::medianBlur(input, bl, params_.medianBlurKSize);
   

2. Detecting lines and their directions – using Sobel filter (magnitudes are obtained using the magnitude function and directions are computed using atan2 from horizontal and vertical gradients)

   cv::Mat_<double> grad_x;
   cv::Sobel(beforeSobel, grad_x, CV_64F, 1, 0, params_.sobelKSize);
   cv::Sobel(beforeSobel, grad_y, CV_64F, 0, 1, params_.sobelKSize);
   

3. Finding some contour line seeds – points at lines with approximately equal directions.

   cv::Mat_<double> magnitude;
   cv::magnitude(grad_x, grad_y, magnitude);
   

4. Tracing lines beginning at the seeds – we are going from each seed to both directions to find the line while checking if the curves do not exceed a threshold (the more curved lines are probably not the contour lines).
5. Filtering of the traced lines – only the lines having both ends at the image boundaries or the closed lines are considered as the map contour lines.

The result image shows a map with some contour lines detected. The seeds and line points are marked as follows:

• yellow – seed points
• red – closed line points
• green – points of the first part of a line ending at the image edge
• blue – points of the second part of a line ending at the image edge

Problems and possible improvements

These algorithm properties cause problems and need to be considered in the algorithm improvements:

• line intersections are not being detected – one line from each pair of the intersecting lines should always be removed,
• the algorithm uses a global magnitude threshold (the threshold determines if a point belongs to a line), but the line intensities change in most images,
• the algorithm has too many parameters which were not generalized to match more possible images,
• some contour lines are not continuous (they are splitted by labels) and thus not being detected by the algorithm.

This project shows object recognition using local features-based methods. We use four methods for keypoints detection and description: SIFT/SIFT, SURF/SURF, FAST/FREAK and ORB/ORB. Keypoints are used to compute homography. Object is located in scene with RANSAC algorithm. RGB and hue-saturation histograms are used for RANSAC verification.

Functions used: FeatureDetector::detect, DescriptorExtractor::compute, knnMatch, findHomography, warp, calcHist, compareHist

Input

The process

1. Keypoints detection

   FeatureDetector * detector;
   detector = new SiftFeatureDetector();
   detector->detect( image, key_points_image );
   
   DescriptorExtractor * extractor;
   extractor = new SiftDescriptorExtractor();
   extractor->compute( image, key_points_image, des_image );
   

2. Keypoints description
3. Keypoints matching

   DescriptorMatcher * matcher;
   matcher = new BruteForceMatcher<L2<float>>();
   matcher->knnMatch(des_object, des_image, matches, 2);
   

4. Calculating homography

   findHomography( obj, scene, CV_RANSAC );
   

5. Histograms matching

   calcHist( &hsv_img_object, 1, channels, Mat(), hist_img_object, 2, histSize, ranges, true, false );
   compareHist( b_hist_object, b_hist_quad, CV_COMP_BHATTACHARYYA );
   

6. Outline recognized object

Sample

drawMatches( gray_object, key_points_object, image,
             key_points_image, good_matches, img_matches,
             Scalar::all(-1), Scalar::all(-1), vector<char>(),
             DrawMatchesFlags::NOT_DRAW_SINGLE_POINTS );

	if (good_matches.size() >= 4)
	{
	for( int i = 0; i < good_matches.size(); i++ )
	{
	//
	obj.push_back( key_points_object[ good_matches[i].queryIdx ].pt );
	scene.push_back( key_points_image[ good_matches[i].trainIdx ].pt );
	}

	H = findHomography( obj, scene, CV_RANSAC );

	perspectiveTransform( obj_corners, scene_corners, H);
	//*******************************************************

	Mat quad = Mat::zeros(rgb_object.rows, rgb_object.cols,
                   CV_8UC3);

	//warping object back to tamplate rotation
	warpPerspective(frame, quad, H.inv(), quad.size());

	...

The project shows detection and recognition of euro money bill from input image (webcam). For each existing euro money bill is chosen template that contains number value of bill and also its structure. For matching templates with input images is used Flann Based matcher of local descriptors extracted by SURF algorithm.

Functions used: medianBlur, FlannBasedmatcher, SerfFeatureDetector, SurfDescriptorExtractor, findHomography

Process

1. Preprocessing – Conversion to grayscale + median filter

   cvtColor(input_image_color, input_image, CV_RGB2GRAY);
   medianBlur(input_image, input_image, 3);
   

2. Compute local descriptors

   SurfFeatureDetector detector( minHessian );
   vector<KeyPoint> template_keypoints;
   detector.detect( money_template, template_keypoints );
   SurfDescriptorExtractor extractor;
   extractor.compute( money_template, template_keypoints, template_image );
   detector.detect( input_image, input_keypoints );
   extractor.compute( input_image, input_keypoints, destination_image );
   

3. Matching local descriptors

   FlannBasedMatcher matcher;
   matcher.knnMatch(template_image, destination_image, matches, 2);
   

4. Finding homography and drawing output

   Mat H = findHomography( template_object_points, input_object_points, CV_RANSAC );
   perspectiveTransform( template_corners, input_corners, H);
   drawLinesToOutput(input_corners, img_matches, money_template.cols);
   

Sample

Project is focused on the image detection which major components are cities and buildings. Buildings and cities detection assumes occurence of the edges as implication of the Windows and walls, as well as presence of the sky. Algorithm creates the feature vector with SVM classification algorithm.

Functions used: HoughLinesP, countNonZero, Sobel, threshold, merge, cvtColor, split, CvSVM

The process

1. Create edge image

   cv::Sobel(intput, grad_x, CV_16S, 1, 0, 3, 1, 0, cv::BORDER_DEFAULT);
   cv::Sobel(intput, grad_y, CV_16S, 0, 1, 3, 1, 0, cv::BORDER_DEFAULT);
   

2. Find lines in the binary edge image

   cv::HoughLinesP(edgeImage, edgeLines, 1, CV_PI / 180.0, 1, 10, 0);
   

3. Count numbers of lines in specified tilt
4. Convert original image to HSV color space and remove saturation and value

   cv::cvtColor(src, hsv, CV_BGR2HSV);
   

5. Process the image from top to bottom , if pixel is not blue then all pixels under him are not sky
6. Classification with SVM

   CvSVMParams params;
   params.svm_type  = CvSVM::C_SVC;
   params.kernel_type = CvSVM::LINEAR;
   params.term_crit   = cvTermCriteria(CV_TERMCRIT_ITER, 5000, 1e-5);
   
   float OpencvSVM::predicate(std::vector<float> features)
   {
      std::vector<std::vector<float> > featuresMatrix;
      featuresMatrix.push_back(features);
      cv::Mat featuresMat = createMat(featuresMatrix);
      return SVM.predict(featuresMat);
   }
   

Example