CN101564300B - Gait cycle detection method based on regional characteristics analysis - Google Patents

Abstract

The invention provides a gait cycle detection method based on regional feature analysis. Including the acquisition of pedestrian target contours and gait cycle detection; the method for obtaining the pedestrian target contours is to first extract a single frame image from the video for grayscale transformation, and then calculate the median value of each pixel point frame by frame, As the background image of the entire sequence, the background subtraction method is used to extract the human body target, the hole in the binarized image is filled by mathematical morphology, and the silhouette of the person is extracted by simple connectivity analysis, the human body is centered, and the size of the image is unified to 64*64 pixels The gait cycle detection is to convert the gait cycle analysis problem into a single-frame graph area feature analysis problem, that is, to analyze the gait cycle according to the feature change of the graph area in each frame. The invention not only has a small amount of calculation, but also has reached the accuracy of subjectively judging the gait cycle, and provides the possibility for real-time gait recognition.

Description

Gait Cycle Detection Method Based on Regional Feature Analysis

(1) Technical field

The invention relates to a pattern recognition technology, in particular to a gait cycle detection method in gait recognition.

(2) Background technology

The purpose of gait recognition is to identify people based on their walking posture. Gait recognition can obtain the gait characteristics without the research object being aware of it. It has the advantages of non-invasiveness, non-contact, low requirement for system resolution, long-distance, difficult to camouflage, and little environmental impact. Therefore, from the viewpoint of video surveillance, gait is the most potential biometric feature in long-distance situations. Gait recognition has broad application prospects and economic value in the fields of access control systems, security monitoring, human-computer interaction, medical diagnosis, etc., so it has stimulated the enthusiasm of researchers at home and abroad.

The gait sequence image is a periodic spatio-temporal joint signal. If the whole video is studied to identify individuals, it will not only cause the disadvantage of large amount of data, but also there will be redundancy in the information. Therefore, it is necessary to use the method of cycle analysis to determine the start frame and end frame, and then determine a gait cycle, so as to extract features in a cycle to achieve the purpose of final recognition. Many researchers at home and abroad have done research on gait cycle detection. BenAbdelkader et al. determined the gait cycle by calculating the self-similarity of human body contours; BenAbdelkader et al. also analyzed the periodicity of gait according to the width of the bounding rectangle; Collins et al. analyzed the periodic changes in the height and width of the human body, and then observed the gait cycle; Kale et al. analyzed the periodic characteristics of the gait by observing the norm of the human body width vector over time; Boulgouris et al. used the foreground pixel and the autocorrelation to judge the cycle of gait; Sarkar et al. used the periodic characteristics of the number of pixels in the lower half of the human body area to determine the periodic change of gait; Li et al. arranged the gait into a self-similar map (SSP), Then, the linear local embedding (LLE) non-linear dimensionality reduction method is used to extract one-dimensional features that retain the original geometric shape to analyze the periodicity of gait; Chen Shi et al. take all pedestrian contour areas in the gait sequence as a rectangular frame In the image area, within the 1/4 height of the image area from bottom to top, three areas are equally divided horizontally, the cumulative contour points of each area are calculated, and the corresponding point distribution histogram features are obtained to detect the gait cycle; these methods are commonly used to calculate The disadvantage of large quantity. Since the accuracy of gait cycle segmentation seriously affects the accuracy of gait recognition problems, most of the existing literature proposes gait recognition algorithms under the assumption that the gait cycle segmentation is very good.

Public reports related to the present invention include:

[1]BenAbdelkader C, Culter R, Davis L.Motion based recognition of people in eigengaitspace[C]In:proceedings of the IEEE International Conference on Automatic Face and GestureRecognition.Washington DC, USA, 2002:254-259P;

[2] Boulgouris N V, Plataniotis K N, Hatzinakos D. Gait recognition using dynamic timewarping [C]. 2004 IEEE 6th Workshop on Multimedia Signal Processing, 2004: 263-266;

[3] Li Hong-gui, Shi Cui-ping, Li Xing-guo. LLE based gait recognition [C]. In: Proceedings of2005 International Conference on Machine Learning and Cybernetics, 2005, 7: 4516-4521;

[4] Chen Shi, Ma Tianjun, Huang Wanhong, et al. Multi-layer window image moments for gait recognition. Journal of Electronics and Information Technology, 2009, 31(1): 116-119.

(3) Contents of the invention

The purpose of the present invention is to provide a gait cycle detection method based on regional feature analysis that can effectively improve the speed and accuracy of gait cycle detection, thereby providing possibility for real-time gait recognition.

The purpose of the present invention is achieved like this:

Including the acquisition of pedestrian target contours and gait cycle detection; the method for obtaining the pedestrian target contours is: first extracting a single frame image from the video for grayscale transformation, and then calculating the median value of each pixel point frame by frame, As the background image of the entire sequence, the background subtraction method is used to extract the human body target, the hole in the binarized image is filled by mathematical morphology, and the silhouette of the person is extracted by simple connectivity analysis, the human body is centered, and the size of the image is unified to 64*64 pixels ; The gait cycle detection is to convert the gait cycle analysis problem into a single-frame graphic area feature analysis problem, that is, analyze the gait cycle according to the characteristic change situation of the graphic area in each frame, from the first occurrence of local The extremum to the third occurrence of the local extremum is a gait cycle.

The present invention may also include:

1. The feature of the graphic area is one of the area, centroid, fitted ellipse or circle, special point, and bounding box feature of the graphic area in each frame.

2. The area mentioned is one of the intrinsic area, the convex hull area, the area after filling the cavity, or the ratio of the intrinsic area to the convex hull.

3. The fitting ellipse or circle is the major axis, minor axis and eccentricity of the ellipse having the same second-order space moment in the human body region or a circle equal to the area of the human body region.

4. The bounding box feature is that the width of the bounding box of the moving body changes frame by frame or the proportion of the moving body in the bounding box changes.

The main effect of the invention is that not only the amount of calculation is small, but also the accuracy of subjectively judging the gait cycle has been achieved, which provides the possibility for real-time gait recognition.

(4) Description of drawings

The flowchart of Fig. 1 gait cycle detection;

Figure 2(a)-Figure 2(e) The preprocessing process of human target extraction, in which: Figure 2(a) grayscale transformation, Figure 2(b) background reconstruction, Figure 2(c) background subtraction, Figure 2( d) Human body contour, Fig. 2(e) normalized centering;

Figure 3 Gait sequence images;

Fig. 4 schematic diagram of convex hull;

Figure 5(1)-Figure 5(20) various methods to detect the gait cycle, in which Figure 5(1) is based on the intrinsic area, Figure 5(2) is based on the area of the convex hull, and Figure 5(3) is based on the area after filling the cavity , Figure 5(4) is based on the proportion of graphics to the convex hull, Figure 5(5) is based on the ordinate of the centroid, Figure 5(6) is based on the abscissa of the centroid, Figure 5(7) is based on the long axis of the fitted ellipse, Figure 5 (8) According to the minor axis of the fitted ellipse, Figure 5(9) is based on the eccentricity of the fitted ellipse, Figure 5(10) is based on the diameter of the fitted circle, Figure 5(11) is based on the right-top abscissa, Figure 5(12 ) according to the right-top ordinate, Fig. 5(13) according to the right-bottom abscissa, Fig. 5(14) according to the right-bottom ordinate, Fig. 5(15) according to the left-bottom abscissa, Fig. 5(16) according to The left-bottom ordinate, Figure 5 (17) is based on the left-top abscissa, Figure 5 (18) is based on the left-top ordinate, Figure 5 (19) is based on the width of the human body bounding box, and Figure 5 (20) is based on the figure. The scale of the bounding box;

Figure 6 shows a single-frame image of a gait with holes. The four frames in Figure 6 are frames 10, 12, 22, and 49 in Figure 3 from left to right, and the circles mark the holes that appear on the real gait outline;

Fig. 7 The ellipse fitted by the graphic area, where 1 is the focus of the ellipse, 2 is the major axis of the ellipse, 3 is the minor axis of the ellipse, and 4 is the ellipse;

Figure 8(1)-Figure 8(2) The positioning of the eight special points, Figure 8(1) is the general situation, Figure 8(2) is the special case of (1), where 1 is top-right, 2 is right -top, 3 is right-bottom, 4 is bottom-right, 5 is top-left, 6 is left-top, 7 is left-bottom, 8 is bottom-left;

Figure 9. Bounding boxes of human body regions.

(5) Specific implementation methods

The gait cycle detection method based on regional feature analysis of the present invention includes the acquisition of pedestrian target contours and gait cycle detection; the method for obtaining the pedestrian target contours is: first extracting a single frame image from the video for gray scale transformation, Then calculate the median value of each pixel in frame by frame as the background image of the entire sequence, and finally use the background subtraction method to extract the human target, use mathematical morphology to fill the hole in the binary image, and extract the silhouette of the person through simple connectivity analysis. The human body is centered, and the size of the image is unified to 64*64 pixels; the gait cycle detection is to transform the gait cycle analysis problem into a single frame graphic area feature analysis problem. That is, the gait cycle is analyzed according to the characteristic change of the graphic area in each frame; from the first occurrence of the local extremum to the third occurrence of the local extremum is a gait cycle.

The gait cycle detection method is as follows: analyzing the gait cycle according to the changes of features such as the area, center of mass, fitting ellipse or circle, special point and bounding box of the graphic area in each frame.

The cycle of analyzing the gait according to the area change of the graphic area in each frame is: according to the inherent area of the moving human body, the area of the convex hull, the area after filling the cavity, and the ratio of the inherent area to the convex hull to perform the gait cycle Detection, from the first occurrence of local extremum to the third reappearance of local extremum is a gait cycle.

The described analysis of the cycle of gait according to the change of the center of mass of the graphic area in each frame is: to detect the gait cycle according to the change of the center of mass ordinate and abscissa of the moving human body frame by frame, from the first occurrence of local extremum to The third occurrence of the local extremum is a gait cycle.

The cycle of analyzing the gait according to the variation of the fitting ellipse (or circle) of the graphic area in each frame is: according to the long axis, the short axis and the eccentricity of the ellipse with the same second-order space moment as the human body area or with The gait cycle is detected by the frame-by-frame change of the diameter of the circle with the same area in the human body area. From the first occurrence of the local extremum to the third occurrence of the local extremum is a gait cycle;

The gait cycle is analyzed according to the change of special points in the graphic area in each frame: since the shoulders shake back and forth relative to the center of mass of the human body during walking, the right-top point is used to determine the gait cycle Sexual changes are effective, from the first occurrence of local extremum to the third reappearance of local extremum is a gait cycle;

The described analysis of the cycle of gait according to the change of the bounding box of the graphics area in each frame is: according to the frame-by-frame change of the width of the bounding box of the moving human body or according to the change of the proportion of the bounding box of the moving human body, the gait cycle detection is performed, from the first A gait cycle is defined as the occurrence of a local extremum once to the third reappearance of a local extremum.

The present invention is described in more detail below in conjunction with accompanying drawing example:

1. In order to extract the human target, first extract a single frame image from the original video for grayscale transformation (as shown in Figure 2(a)); then calculate the median value of each pixel point in each frame as the background image of the entire sequence ( As shown in Figure 2(b)); finally, the background subtraction method is used to extract the human body target (as shown in Figure 2(c)), and mathematical morphology is used to fill the holes in the binarized image, and the simple connectivity analysis is used to extract the silhouette of the person (as shown in Figure 2(c) d). In order to eliminate the influence of image size on recognition, the human body should be centered, and the size of the image should be unified to 64*64 pixels (as shown in Figure 2(e)).

2. Gait cycle detection

After the above-mentioned image sequence preprocessing, the problem of gait cycle analysis is transformed into the problem of feature analysis of the graphic region of a single frame. Take a sequence containing 53 frames (as shown in Figure 3) as an example to analyze its gait cycle. Five types of 10 gait cycle detection methods based on regional feature analysis are proposed, which are based on the area, center of mass, attributes of the fitted ellipse (or circle), some special points, and changes in the bounding box of the moving human body in the image. Determine the period of the gait.

2.1 Gait cycle detection method based on area feature-area

Here, for the area feature of the graphic area, we mainly consider the inherent area of the moving human body, the area of the convex hull, and the area after filling the cavity.

Let the pixel set in the region be R, if f(r,c)=1, A represents the intrinsic area of the region:

$\mathrm{<span\; class="notranslate">\; A</span>}<span\; class="notranslate">\; =</span>\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

f(r, c) represents the gray value at (r, c). A means the number of pixels in the image area.

In computational geometry, the convex hull of a point set X in the real vector space V is the boundary of the area formed by the minimum containing this point set, and this point set is a non-empty finite convex set on the plane, as shown in Figure 4 An intuitive picture of a convex hull. For a planar object, the convex hull is easily conceived as an elastically stretchable polygonal wireframe surrounding the particular object, and this elastically stretchable polygonal wireframe is assumed to be a membrane under tension. However, the equilibrium surface (minimum energy surface) may not be a convex hull in this case. In order to illustrate the existence of a convex hull of the set X in the real vector space V, and express the form of the convex set more directly, let the convex set of X be described as a convex set from a finite subset of points in X, the points in the set The form is ∑ _j=1 ⁿ t _j x _j , where n is any natural number, t _j is a non-negative number, and ${\mathrm{\&Sigma;}}_{j=1}^{\mathrm{no}}{t}_j=1,$ x _j (j=1,...,n) is a point in X. So, the convex set of X is expressed as H _convex (X):

${\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; convex</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; \{</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; \&Greater\; Equal;</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1,2</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; \&CenterDot;</span><span\; class="notranslate">\; \&CenterDot;</span><span\; class="notranslate">\; \&CenterDot;</span><span\; class="notranslate">\; \}</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

Note that X contains at least one convex set (eg: the entire space V), and the intersection of any convex set containing X is the intersection of all convex sets containing X. In fact, if X is a subset of an N-dimensional vector space, it is sufficient to combine at most N+1 bumps according to the above definition. Finite point sets and geometric objects in two-dimensional plane and three-dimensional space are special cases.

2.1.1 Observing the periodicity of gait according to the inherent area change of the moving human body in each frame image

As shown in Figure 5(1), when the angle of the limbs of the human body is the largest (such as the 5th, 17th, 29th and 41st frames in the image sequence), the white area of the binary image is the largest, and the moving human body in the binary image at this moment The area of the body is the largest; when the limbs are closed (such as the 11th, 23rd, 35th and 47th frames in the image sequence), the area of the moving human body at this moment is the smallest. Therefore, the periodicity of the gait sequence image can be seen through the area of the moving human body in a single frame image. For example, from frame 5 to frame 28 is a gait cycle; from frame 11 to frame 34 is also a complete cycle. Many groups of gait cycles can be found in this gait sequence, since the start point of the cycle is not defined. In particular, we can define: the starting point of each gait cycle is the maximum angle of limb opening, from this frame to the second occurrence of the maximum angle of limb opening is half a cycle; from this frame to the third The time when the angle of opening of the limbs is the largest for the first time is the whole cycle.

2.1.2 Observe the periodicity of gait according to the periodic change of the convex hull area of the moving human body area in each frame of image

Using the number of pixels in the convex hull to detect the periodicity of the gait is shown in Figure 5(2), but if there are noise points, this method will be affected to a certain extent. Since the noise is removed during preprocessing, this method can be used to estimate the periodicity of the gait. We find that the smallest convex hull area occurs at frames 11, 23, 35, and 48, and the largest convex hull area occurs at frames 6, 17, 29, and 42. Then this method has a similar conclusion to method 1. The difference from method 1 is that method 1 calculates the number of pixels in the actual human body area, while this method approximates the human body area to find the number of pixels in the convex hull. Therefore, this method (as shown in Fig. 5(2)) has a larger area for each frame than method 1 (as shown in Fig. 5(1)), but the periodicity is basically unchanged.

2.1.3 Observing the periodicity of gait according to the periodic change of the area after filling the cavity in the moving human body area in each frame of image

This method can solve the problem of hole defects in the gait contour in the early image preprocessing, but at the same time, for the situation that the holes appearing in the 10th, 12th, 22nd and 49th frames are indeed the real gait contour Void filling. The four frames in Figure 6 are the 10th, 12th, 22nd and 49th frames in Figure 3 from left to right, and the circles mark the holes that appear on the real gait contour. Figure 5(3) shows the effect diagram of judging periodicity by this method, which is very similar to the effect diagram 5(1) of method 1, and also shows obvious periodicity.

2.1.4 Observing the periodicity of the gait according to the change in the proportion of the moving human body in the convex hull in each frame of image

This method is a combination of Method 2.1.1 and Method 2.1.2. As shown in Fig. 5(4), in the 6th, 17th, 31st and 42nd frames, the ratio has a local minimum value; in the 12th, 24th, 36th and 48th frame, the ratio has a local maximum value. Similarly, it can be determined that from frame 6 to frame 30, from frame 17 to frame 41, from frame 12 to frame 35, and from frame 24 to frame 47 are all separate gait cycles.

2.2 Gait cycle detection method based on regional features-centroid

The horizontal and vertical coordinates of the centroid of the region R are:

$\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; A</span>}}\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

$\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; A</span>}}\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

As shown in Figure 5 (5) and (6), they are the frame-by-frame changes of the ordinate and abscissa of the center of mass, respectively. The effect of determining the cycle through the change of the abscissa of the center of mass is not as good as using the ordinate of the center of mass. So here we judge the gait cycle based on the change of the ordinate of the center of mass. We found that the local minimum point of the ordinate of the center of mass appeared in the 11th, 23rd, 35th, and 48th frames, so the 11th to the 34th frame A frame is a cycle, and frames 23 to 47 are also a cycle. However, there are some differences between this method and the method of judging the period according to the inherent area of the human body, but not so much. And we observe with the naked eye that there is not much difference in the shape of the moving human body in the 47th frame and the 48th frame.

2.3 Gait cycle detection method based on regional features - fitting ellipse (or circle)

2.3.1 Observing the periodicity of gait according to the major, minor axis and eccentricity of the ellipse with the same second-order space moment as the graph area

There are three second-order spatial moments of the region, which are respectively expressed as the second-order row moment μ _rr , the second-order column moment μ _cc and the second-order mixing moment μ _rc , which are defined as follows:

${\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; rr</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; A</span>}}\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; )</span>$

${\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; cc</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; A</span>}}\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 6</span>}<span\; class="notranslate">\; )</span>$

${\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; rc</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; A</span>}}\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 7</span>}<span\; class="notranslate">\; )</span>$

μ _rr represents the row variation from the row mean, μ _cc represents the column variation from the column mean, and μ _rc represents the row and column variation from the center. They do not change with the translation and scale of the two-dimensional shape, so they are often used to describe simple shape.

If the region R is an ellipse with its center at the origin, then R can be expressed as:

R={(r,c)|dr ² +2erc+fc ² ≤1} (8)

Then the relationship between the coefficients d, e and f of the elliptic equation and the second order moments μ _rr , μ _cc and μ _rc is:

$\left(\begin{array}{cc}\mathrm{<span\; class="notranslate">\; d</span>}& \mathrm{<span\; class="notranslate">\; e</span>}\\ \mathrm{<span\; class="notranslate">\; e</span>}& \mathrm{<span\; class="notranslate">\; f</span>}\end{array}\right)<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; rr</span>}}{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; cc</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; rc</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}\left(\begin{array}{cc}{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; cc</span>}}& <span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; rc</span>}}\\ <span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; rc</span>}}& {\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; rr</span>}}\end{array}\right)<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 9</span>}<span\; class="notranslate">\; )</span>$

With the coefficients d, e and f of the elliptic equation, we can determine the length, minor axis and direction of the ellipse. Since the coefficients of the elliptic equation have the above-mentioned relationship with the second-order moments μ _rr , μ _cc and μ _rc , we have μ _rr , μ _cc and μ _rc can determine the major axis, minor axis and direction of the ellipse. The discussion is shown in Table 1. Note: the direction angle below is the direction of turning counterclockwise from the vertical axis.

Table 1 Calculation of the major and minor axes and directions of the fitted ellipse based on the second-order space moment

Figure 7 illustrates the positioning of the ellipse and the positioning of the major and minor axes. The direction angle of the ellipse is the angle between the horizontal dotted line and the major axis of the ellipse. According to the major axis, minor axis and eccentricity of the ellipse with the same second-order space moment as the graphic area, the periodic law of gait is found. The experimental results are shown in Figure 5 (7), (8) and (9) respectively. Because when the limbs are closed, the short axis of the fitting ellipse is the shortest and the eccentricity is the largest, while the long axis of the fitting ellipse does not change significantly during the pedestrian walking process, so the effects of Figure 5 (8) and Figure (9) are better. There is a clear cyclicality. In Fig. 5(8), the local minimum value of the short axis occurs in the 11th, 23rd, 35th and 47th frame, and it happens to be the local maximum value of the eccentricity at the same time. We found that the lengths of the minor axis in frame 47 and frame 48 are very close, which explains why there is a difference between the method based on the center of mass and the method based on the intrinsic area when determining the gait cycle.

2.3.2 Determine the periodicity of gait according to the change of the diameter of the fitted circle

其中：Area表示人体区域的面积，也即这个区域含有的像素点数。如图5(10)所示为通过拟合圆的直径来观测步态周期性的效果图，它与图5(1)形状类似，而且该方法与基于固有面积判定的周期定位是一致的。We use a circle to fit the area of the moving human body so that the area of this circle is equal to the area of the human body area. Then we use the diameter of the circle relative to the human body area of each frame to observe the periodicity of the gait. What we actually calculate is

Among them: Area represents the area of the human body area, that is, the number of pixels contained in this area. As shown in Figure 5(10), the effect diagram of observing gait periodicity by fitting the diameter of the circle is similar to that in Figure 5(1), and this method is consistent with the periodic positioning based on the inherent area determination.

2.4 Gait cycle detection method based on regional features - some special points

Figure 8 defines eight special points: top-left, top-right, left-top, right-top, left-bottom, right-bottom, bottom-left and bottom-right, where (2) is the special point of (1) Condition.

Since the limbs shake back and forth relative to the center of mass of the human body during walking, it is effective to use the boundary of the right limb to determine the periodic change of gait, as shown in Figure 5(11) and (12) It is the change of the abscissa and ordinate of the right-top point with the frame of the gait sequence. It can be seen that using the ordinate of this point to detect the period is not effective, so the change of the abscissa of the point is used to judge, the 12th frame The 35th frame and the 24th frame to the 47th frame are all separate gait cycles. The conclusion of this cycle judgment is basically the same as that of method 1; The horizontal and vertical coordinates change with the frame of the gait sequence. We can still see that using the vertical coordinate of this point to detect the period is not effective. Most of the left-bottom and left-top points correspond to the position of the left limb boundary, so the periodic changes of gait can also be seen through these two points, as shown in Figure 5 (15), (16), (17), ( 18) The left-bottom abscissa, left-bottom ordinate, left-top abscissa and left-top ordinate change with the gait sequence frame. Although the cycle effect is not good, you can see the gait through these points. periodic changes in state. The periodic change of gait is determined according to the position changes of some special points in the moving body area of each frame of image. Although this method is simple, it must be ensured that the early preprocessing is good enough. If there are a large number of noise points in the image, it will lead to Segmentation errors of the gait cycle.

2.5 Gait cycle detection method based on region feature----bounding box

Sometimes in order to roughly understand where an area is located in an image, the bounding box of the area is used at this time. The bounding box is a rectangle that surrounds the entire area by four horizontal and vertical sides and is connected to the top, The bottommost, leftmost and rightmost points meet. Figure 9 shows the bounding box of the human body area.

Using the width change of the bounding box in each frame to observe the periodic change of gait is shown in Fig. 5(19). Since its local minimum point has little difference from its nearby points, we use its local Maximum points (for example: corresponding to frames 6, 17, 29 and 43) to estimate the periodicity of the gait, from frame 6 to frame 28 and from frame 17 to frame 42 are separate gait cycles . We can also determine the periodicity of the gait according to the change in the proportion of the moving body area to the bounding box, as shown in Figure 5 (20). The local minima in extreme values are less different from the points near them, so it is not appropriate to use local minimum points for judgment. However, the local maximum point in the extreme value is quite different from the points near them. When the 12th, 24th, 35th and 48th frame, the local maximum value of the ratio appeared, so from the 12th frame to the 34th frame and from the Frames 24 to 47 are all individual gait cycles.

3 Comparison of various gait cycle detection methods

Summarize the above five simple and effective gait cycle detection methods, of which 10 are feasible based on regional feature analysis (including: intrinsic area, convex hull area, cavity removal area, proportion of human body to convex hull, centroid ordinate, fitting ellipse The short axis, the eccentricity of the fitted ellipse, the diameter of the fitted circle, the right-top (bottom) abscissa and the proportion of the human body to the bounding box, etc.) are summarized in Table 2. Their advantage is that the calculation is simple, the amount of calculation is far less than the literature [1] [2] [3] [4], easy to implement, and has reached the accuracy of human subjective judgment. The gait cycle signals obtained by convex hull area method, fitting ellipse minor axis method, fitting ellipse eccentricity method and right-top (bottom) abscissa method are all relatively smooth. The inherent area method, the void area method, the centroid ordinate method, the fitting ellipse minor axis method, the fitting ellipse eccentricity method, the fitting circle diameter method and the human body to bounding box ratio method are more robust to noise, while The convex hull area method, the right-top (bottom) abscissa method and the human body to convex hull ratio method are less robust to noise, so when using these three methods for periodic detection, the image sequence must be pre-prepared. Process work. Except for the minor axis and eccentricity method of fitting ellipse, other methods must be carried out after the standard centering of preprocessing, and because the minor axis and eccentricity method of fitting ellipse have scale and translation invariance, the two cycles The method of detection can be performed prior to standardization of preprocessing. The inherent area method, the void area method, the fitting ellipse minor axis method, the fitting ellipse eccentricity method, the fitting circle diameter method and the right-top (bottom) abscissa method can all get the period of this gait sequence to be 24 frame conclusion, other methods exist to judge the conclusion of 23 or 25 frames. It is normal for these methods to determine the difference in gait cycle, because it is difficult to distinguish the difference between two frames even with the naked eye.

Table 2 Summary of various methods to determine gait cycle characteristics

This patent transforms the problem of gait cycle analysis into a single-frame graphic region feature analysis problem, and proposes five types of region-based feature analysis (including: area, centroid, fitted ellipse (or circle), some special points and bounding boxes) gait cycle detection method. It includes 10 simple and feasible gait detection methods: intrinsic area method, convex hull area method, hollow area method, human body to convex hull ratio method, centroid ordinate method, fitting ellipse minor axis method, fitting ellipse centrifugal Rate method, diameter method of fitting circle, right-top (bottom) abscissa method, human body proportion method, etc. They all achieve the accuracy of human judgment of gait cycle. Fitting ellipse minor axis method and fitting ellipse eccentricity method have the best performance, not only the obtained gait cycle signal is smooth, robust to noise, but also has scale and translation invariance. The method can be performed before standard preprocessing, which greatly reduces the calculation amount of other periodic frames related to image processing, and also shortens the time of preprocessing work for gait recognition.

Claims (1)

1. A gait cycle detection method based on regional feature analysis, comprising the acquisition of the pedestrian target profile and the gait cycle detection; it is characterized in that: the method for the acquisition of the pedestrian target profile is to first extract a single frame from the video The image is grayscale transformed, and then the median value of each pixel in each frame is calculated as the background image of the entire sequence. Finally, the background subtraction method is used to extract the human target, and the hole in the binarized image is filled with mathematical morphology. Simple connectivity analysis Extract the silhouette of people, make the human body center, and unify the size of the image to 64*64 pixels; the gait cycle detection is to convert the gait cycle analysis problem into a single frame graphic area feature analysis problem, that is, according to the The characteristic changes of the graphic area are used to analyze the cycle of gait, and it is a gait cycle from the first occurrence of local extremum to the third occurrence of local extremum; Fitting ellipse; said fitting ellipse is the major axis, the minor axis and the eccentricity of the ellipse with the same second-order space moment in the human body region, according to the major axis, minor axis, eccentricity to observe the periodicity of gait, There are three second-order spatial moments of the region, which are respectively expressed as the second-order row moment μ _rr , the second-order column moment μ _cc and the second-order mixing moment μ _rc , which are defined as follows: ${\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; rr</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; A</span>}}\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$ ${\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; cc</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; A</span>}}\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$ ${\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; rc</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; A</span>}}\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; )</span>$ A represents the intrinsic area of the region, is the abscissa of the centroid of the region R,

is the ordinate of the centroid of the region R; μ _rr represents the row variation deviated from the row mean, μ _cc represents the column variation deviated from the column mean, μ _rc represents the row and column variation deviated from the center, and they do not change with the translation and scale of the two-dimensional shape while changing, If the region R is an ellipse with its center at the origin, then R is expressed as: R={(r,c)|dr ² +2erc+fc ² ≤1} Then the relationship between the coefficients d, e and f of the elliptic equation and the second order moments μ _rr , μ _cc and μ _rc is: $\left(\begin{array}{cc}\mathrm{<span\; class="notranslate">\; d</span>}& \mathrm{<span\; class="notranslate">\; e</span>}\\ \mathrm{<span\; class="notranslate">\; e</span>}& \mathrm{<span\; class="notranslate">\; f</span>}\end{array}\right)<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; rr</span>}}{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; cc</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; rc</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}\left(\begin{array}{cc}{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; cc</span>}}& {<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; rc</span>}}\\ {<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; rc</span>}}& {\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; rr</span>}}\end{array}\right)$ With the coefficients d, e and f of the ellipse equation, the length, minor axis and direction of the ellipse are determined. Since the coefficients of the ellipse equation have the above-mentioned relationship with the second-order moments μ _rr , μ _cc and μ _rc , it is determined by μ _rr , μ _cc and μ _rc determines the major and minor axis of the ellipse and its direction.

Images