Detecting Human Walking Motion
Using OpenCV


Goal
 The goal of my project was to detect human walking motion, specifically gait, from a sequence of images, by looking at the periodicity of the movement.
Techniques
 Motion History
 The main techniques and series of functions used are from the motion analysis and object tracking set of OpenCV tools. Specifically, UpdateMotionHistory, ClacMotionGradient, CalcGlobal Orientation, and SegmentMotion. These provided the information on the moving regions, i.e., what areas are moving, and what is the angle of each movement.

Filter / Predictor
 The second technique needed was a way to predict the next motion, based on the centroids represented (explained below), in order to differentiate between walking motion and non-walking motion, thus detecting human motion. Two different methods were tried.
The first method was the Kalman estimator, which proved to be unsuccessful after close examination of the example code provided with the OpenCV toolkit. The problem was that the estimator did not perform very well. A working Kalman estimator should improve over time, however in this case, the best performance was at the beginning of the program's execution. By this I mean that the position of the predicted motion (indicated by a green '+') was closest to the actual motion (indicated by a red '+') only at the very beginning. As the program continued to run, the predicted motion symbol moved farther and farther away from the actual motion symbol. This signifies that the estimator's accuracy decreased over time.

The second, and successful method was the Alpha-Beta-Gamma (abg) filter. This filter works by measuring the object, then predicting the motion by adding values for velocity and acceleration, providing a trajectory of the motion.
Implementation
 This section lists and explains the steps necessary to achieve the goal. The source code can be found here.
  1. Define the filter to be used.
  2. Open and read from a text file containing the names of each image in the sequence (which can be found here).
  3. Create the motion history image. This includes converting the image to grayscale, finding the difference between two frames at a time in sequence order, which creates a silhouette image of the layered history of the motion. A threshold function then is applied to the silhouette image, and the motion history is updated.
  4. Calculate the orientation of the motion using the CalcMotionOrientation function.
  5. Perform motion segmentation. The distinguishes between the different connected regions of movement.
  6. The angle of the movement is calculated, and a centroid with a line depicting the direction of motion is drawn on the MHI image.
  7. Next we use the abg filter to predict the flow of motion. To do so, first we measure the x,y location of the centroid, then pass that information to the filter to predict the motion trajectory. This information is written to a file (which can be found here).
Parameter Modifications
 The parameters that were changed / modified to elicit results that (theoretically) can be reproduced for any sequence of images are:
  • MHI_DURATION represented in milliseconds, and accounts for how long images are stored in the motion history. The original code mistakenly used numbers based on seconds. This error caused the angles of the centroids in the MHI image not to move. Therefore, the number needed to be increased.
  • MAX_TIME_DELTA and MIN_TIME_DELTA are used for thresholding on the values of the MHI image. The situation for these parameters was the same as above, therfore these numbers also needed to be increased.
  • cvCvtColor one parameter passed to this function was changed from CV_BGR2GRAY to CV_RGB2GRAY based on the image type that was used. My understanding was that the color images from the camera were in RGB order. However, I think this was a minor change, and the program still functions correctly without this.
  • Read from file instead of capturing the motion directly from a camera.
  • Filter.h class (which can be found here ) was included.
  • abg_filter accuracy and time for predicting were changed. Changing these numbers resulted in less overshooting of the prediced target.
Analysis and Results
 An analysis of the data after executing the program shows that the gait of human walking motion indeed, can be detected, based on the periodicity represented by plotting (with gnuplot) the 'x' coordinate for both the measured target and the predicted target.

When launching the program, 3 windows appear, which are represented by the images below.



The image on the left shows the motion history. The center image is the actual image, and the image on the right contains the information about the moving regions.

The moving regions are segmented out and depicted by the centroids, which are the red circles with the lines. The red lines in the circles show the angle and direction (from the center outward) of the movement of each region. The number of centroids is equal to the number of detected moving regions. Pressing the "Enter" key over and over causes the program to advance through the sequence of images and stores the centroid information in a file which was discussed above.


After the file has been created, the measured target motion and the predicted target motion can be plotted using Gnuplot with the following command:
> plot 'output.txt' using 1 with lines, 'output.txt' using 3 with lines
The resulting graph is depicted in the image to the left. The red line shows the actual movement, and the green line shows the predicted movement. As you can see, the periodicity is not always perfect - there are pockets in which more than one centroid appear. Currently, I have hardcoded the first position of the predictor to be at the same position of the first centroid, and I do not have a way to distinguish between centroids. Further work will include an algorithm for this, which is discussed below.


The range of the 'x' coordinate can be set with the following command, in order to focus in on a specific portion of the graph:
> set xrange[15:35]
> replot
This results in the image to the right. As you can see, the two lines are almost identical, indicating good prediction, as well as periodicity. This means that the goal has been achieved.



Unsolved Problem
 The main problem that has yet to be solved is to choose the correct centroid for the cases in which two or more centroids (e.g., moving regions) are shown. One necessary step for achieving this is to calculate the distance between the centroids to make sure that the chosen one is, indeed, the next one in that range of motion. This step was attempted, however the results were incorrect - the value indicating distance occasionally was negative - indicating that I must have an error somewhere in my equation.
Upon rechecking of my equation, I think it is correct.

distance = SQRT((x2-x1)^2+(y2-y1)^2)

The results are: most of the time either 0 or a large positive number, but occasionally the distance is a large negative number. Upon further examination, I think this may be correct, because sometimes the centroid moves backward in successive frames. However, I still do not understand why the number is so large.

Andrea Kleinsmith ~ University of Aizu ~ September 8th, 2003
andi@andisplanet.com