Opened and closed hand gesture detection

Opened and closed hand gesture detection

We detect the gesture of the opened and closed hand with sensor Kinect. State of the hand was divided into 2 parts, when it is opened (palm) or closed (fist). We assume that hand is rotated in a parallel way with the sensor and is captured her profile.

Functions used:

The process

  1. Get point in the middle of the hand and limit around her window
    Point pointHand(handFrameSize.width, handFrameSize.height);
    Rect rectHand = Rect(pos - pointHand, pos + pointHand);
    Mat depthExtractTemp = depthImageGray(rectHand); //extract hand image from depth image
    Mat depthExtract(handFrameSize.height * 2, handFrameSize.width * 2, CV_8UC1);
    

    Limiting red window with hand

  2. Find the minimum depth value in the window
    int tempDepthValue = getMinValue16(depthExtractTemp);
    
  3. Convert window from 16bit to 8bit  and use as mean value of the minimum depth
    ImageExtractDepth(&depthExtractTemp, &depthExtract, depthValue );
    

    Conversion 16bit to 8bit image

  4. Cut half hand in the window
    1. for the right hand from the center to the right
    2. for the left hand from the center to the left
    3. Cropping half the hand in the window

  5. Use thresholding, create mask and cut distant hand (finger)
    Mat depthThresh;
    threshold( depthThresh, depthThresh, 180, 255, CV_THRESH_BINARY_INV);
    

    Cropping half the hand in the window

  6. Determine the size of the rectangle surrounding this part of the hand
    Mat depthExtract2;
    morphologyEx(depthExtract2, depthExtract2, MORPH_CLOSE, structElement3);
    vector<vector<Point>> contours;
    vector<Vec4i> hierarchy;
    findContours(depthExtract2, contours, hierarchy, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE, cvPoint(0,0));
    fitEllipse(Mat(contours[i]));
    
  7. If aspect ratio of width and height of the rectangle is greater than 1, then hand is opened, else hand is closed

    Right hand shape and left detection rectangle

Limitation

  • Maximal distance detection is 2 meters
  • Maximal slope hand is up or down by 25 degrees
  • Profile of hand must be turned parallel with the sensor

Result

Detection of both hand (right and left) takes 4ms.

Opened and closed hand

Augmented Reality with hand detection