KR20090049438A - Pose robust human detection and tracking method using the stereo image - Google Patents

Abstract

The human detection method using stereo images according to the present invention comprises a two-dimensional space-depth histogram transformed from a two-dimensional space-time histogram generated using visual difference information in a stereo image with four-directional directional two-dimensional elliptic filters. By detecting the candidate of the person through the multiplicative operation and verifying the detected person candidate through face detection and head-shoulder shape matching, high human detection rate is provided even when the pose angle of the person in the stereo image is not in front. do.

Description

Force robust human detection and tracking method using the stereo image}

The present invention relates to a method for detecting a person using a stereo image, and more particularly, to a two-dimensional space-depth histogram generated by using the disparity information of a stereo camera. -Oriented 2D Elliptical Filters (MO2DEFs) and Convolutional Operations (MO2DEFs) to detect human candidates, face detection and head-to-shoulder matching to verify the human, using stereo images to track the position of the person in real time The present invention relates to a method for detecting a person who is robust to a pose.

In the case of detecting a person through background extraction, an image obtained through a single camera generally used cannot be detected correctly. Accordingly, various detection methods for effectively detecting a person through stereo images have been proposed.

In the conventional stereoscopic image detection method, a person is detected by height map and head detection from a stereoscopic image obtained by installing a stereo camera on the ceiling. In the case of a similar shape to a human head, there was a problem such as incorrectly judged as a human.

In addition, since a person must be detected by an image captured from a camera fixed to the ceiling, it cannot be applied because the camera cannot be fixed in the same manner on a mobile platform such as a robot.

Therefore, not only a person can be detected by using a stereo image photographed using a camera installed on a mobile platform such as a robot, but also a person can be detected as an image of a person who is partially rotated in one direction instead of the front side. There has been a demand.

The technical problem to be achieved in the present invention is to detect a human candidate regardless of the rotation angle of the person using a stereo image captured on a mobile platform, such as a robot and consists of face detection and head-shoulder shape matching The present invention provides a method for detecting a person who finally extracts a person by determining whether a real person is correct through a person verification operation.

Human detection method using a stereo image for achieving the object of the present invention,

(a) generating a two-dimensional spatial-depth histogram by obtaining a disparity map from a stereo image;

(b) designing directional two-dimensional elliptical filters, similar to the human shape, each rotated by a plurality of predetermined angles, by a two-dimensional kernel function;

(c) detecting a human candidate through a convolution operation between the two-dimensional space-depth histogram and the directional two-dimensional elliptic filter; And

(d) predicting a pose angle of the detected human candidate based on the results of the multiplication of the directional two-dimensional elliptic filter.

In addition, the detecting step includes: (e) verifying a person by face detection from the detected person candidate; And

(f) if the verification is not performed in the step (e), characterized in that it comprises the step of verifying the person through the head-shoulder shape matching.

The plurality of angles may be set to 0 °, 45 °, 90 °, and 135 °.

In addition, the step (a),

calculating a visual difference map from the stereo image;

(a2) creating a two-dimensional spatial-disparity histogram by projecting the visual difference map in a vertical direction; And

(a3) converting the two-dimensional space-time histogram into a two-dimensional space-depth histogram.

In addition, the step (c),

(c1) calculating four histograms as a result of filtering by performing a multiplicative operation with the two-dimensional space-depth histogram and the four directional two-dimensional elliptic filters;

(c2) detecting a human candidate centered on the position of the largest value among the four filtered histograms.

In addition, the step (f),

(f1) A head-shoulder silhouette is extracted from the motion information of a person, which is a result of a double time difference, and the silhouette information, which is a result of detecting a candidate of a person in a visual difference map. step;

(f2) extracting the head-shoulder shape from the head-shoulder silhouette; And

(f3) comparing the extracted head-shoulder shape with the similarity between the previously registered head-shoulder shape.

In addition, in the step (f1), the head-shoulder silhouette is extracted from the double time difference result calculated by performing the temporal difference twice in the stereo image and the human candidate detection result in the visual difference map. It is done.

In addition, extracting the head-shoulder shape of the (f2),

And sampling a specific number of points on the edge of the head-shoulder silhouette.

In addition, the step (c2) is characterized in that it comprises the step of setting the area of the person candidate as a range by the left and right average width of the person and the thickness value of the front and back on the basis of the position of the largest value.

The human detection method according to the present invention extracts a human candidate by loading a two-dimensional space-depth histogram generated from a stereo image and a directional two-dimensional elliptic filter in four directions, and detects a face and performs a head-shoulder on the extracted human candidate. Verification is performed through shape matching to provide a high detection rate even when the pose angle of the person in the image is not in front.

Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a flowchart illustrating a person detection method using stereo image information according to the present invention.

에서 사람 키 방향인 수직 축 방향으로 투상(project)함으로써 2차원 공간-시각차 히스토그램

을 산출하고, 이를 근거로 2차원 공간-깊이 히스토그램을 산출한다(100 내지 102 단계). Calculate the visual difference map from the stereo image obtained by the stereo camera, and obtain the three-dimensional visual difference map

Histogram of Two-Dimensional Space-Visual Differences by Projecting from the Vertical Axis Direction in the Human Key Direction

Is calculated and a two-dimensional space-depth histogram is calculated based on this (steps 100 to 102).

FIG. 2A is an image of a person when the stereo camera is face to face, and FIG. 2B is a time difference image as a result of calculating a time difference map for each stereo image of FIG. 2A, and a bright part may be a person in contrast to a background. This is high.

FIG. 2C is a spatial-time difference histogram obtained from the visual difference image of FIG. 2B, and FIG. 2D is a space-depth histogram transformed for application of a directional two-dimensional elliptic filter from the space-time difference histogram of FIG. 2C.

를 공간-시각차 히스토그램

에서 픽셀 크기로 사람을 나타나는 공간 좌표라고 하고,

를 공간-깊이 히스토그램

에서 실제 크기로 사람을 나타내는 공간 좌표라고 하면, 2차원 공간-깊이 히스토그램은 2차원 공간-시각차 히스토그램으로부터 다음 수학식 1과 같이 산출된다.

Space-Visual Histogram

Is called the spatial coordinate in which people are represented in pixels,

Space-depth histogram

In the case of the spatial coordinates representing the person at the actual size, the two-dimensional space-depth histogram is calculated from Equation 1 from the two-dimensional space-time difference histogram.

는 2차원 공간-시각차 히스토그램에서 공간 좌표축의 중앙이며,

는 스테레오 카메라의 초점길이,

는 스테레오 카메라의 기선(baseline)이며, z는 깊이를 나타낸다. here

Is the center of the spatial coordinate axis in the two-dimensional space-time difference histogram,

Is the focal length of the stereo camera,

Is the baseline of the stereo camera, and z is the depth.

Subsequently, a directional two-dimensional elliptic filter having a human shape rotated at a plurality of predetermined angles is created.

만큼 회전한 방향성 2차원 타원형 필터 V_θ는 타원형 형태로 다음과 같이 나타낼 수 있다.Angle

The directional two-dimensional elliptic filter V _θ rotated by may be expressed in an elliptical form as follows.

와

는 사람 몸의 평균 너비와 두께이다.here

Wow

Is the average width and thickness of a person's body.

는 다음과 같이 나타내어진다. Also, two-dimensional kernel function of directional two-dimensional elliptic filter

Is expressed as follows.

가 타원의 윤곽으로부터 멀어질수록 감소한다. 도 3a 내지 도 3d는 0°, 45°, 90°, 135°로 회전된 방향성 2차원 타원형 필터들의 2차원 커널 함수들로서 패턴을 나타낸 것이다. The filter weight has a maximum of 1 in the outline of the ellipse,

Decreases away from the contour of the ellipse. 3A-3D show patterns as two-dimensional kernel functions of directional two-dimensional elliptical filters rotated at 0 °, 45 °, 90 °, 135 °.

The directional two-dimensional oval filter has an elliptic shape similar to the outline of a human body and has a directional two-dimensional filter in consideration of the rotation of the person.

The filter shown in FIGS. 3A to 3D uses only half of the two-dimensional kernel function because the camera of the robot captures only one side of the person in front, thus only half of the elliptical form is shown.

In addition, the generated 2D kernel function performs normalization to equalize the sum of the filtering results and sets the sum of the 2D kernel filters to 1.

The filtered result in step 104 is calculated as a product of the four-directional two-dimensional elliptic filter by the space-depth histogram and the θ value of Figure 2d and can be represented by the following equation (4).

는 승적 연산자이며, F_θ는(·,·)는 2차원 커널 함수이다.here

Is a multiplier operator and F _θ is a two-dimensional kernel function.

Step 106 extracts the human candidate from the four filtering results based on the θ values.

FIG. 4 is a detailed flowchart for explaining a process of extracting a human candidate shown in FIG. 1.

Each person candidate is extracted based on the average width, thickness, and two-dimensional width of the person. Assuming that FIG. 2D is a filtered two-dimensional depth histogram, the width of the person and the distance on the vertical axis are calculated as thicknesses between the left and right sides of the semi-elliptic shape shown in the center. To this end, the first two-dimensional depth histogram of FIG. 2D is converted into a one-dimensional depth histogram, and the front position z _f of the human candidate is found using the average two-dimensional width of the person as a threshold. We then predict the back z _b of the person candidate based on the average thickness of the person. Thereafter, only a region corresponding to the front and rear ranges of the person candidate found in the filtering result is converted into the one-dimensional spatial histogram (steps 400 to 406).

이라고 보고, 그 중심을 기준으로 사람의 평균 너비를 기준으로 하여 사람 후보에 해당하는 영역의 왼쪽 끝 x_l과 오른쪽 끝 x_r을 예측한다. 그리고 필터링 결과에서 전후좌우로 찾아낸 영역을 기준으로 사람 후보를 나타내는 봉우리의 가장자리를 찾아서 실제 사람 후보의 영역을 확정한다. 사람이 더 이상 찾아지지 않을 때까지 이와 같은 연산을 반복한다(408 내지 412 단계). The point at which the largest value appears in this histogram.

The left end x _l and the right end x _r of the area corresponding to the person candidate are predicted based on the average width of the person based on the center. The area of the real person candidate is determined by finding the edges of the peaks representing the person candidates based on the areas found in the filtering results. This operation is repeated until a person is no longer found (steps 408-412).

Next, the process of predicting the pose angle of the human candidate in step 108 consists of three steps.

(θ=0°, 45°, 90°, 135°)에서 찾아진 사람 후보 영역을 둘러싸는 사각형들을 얻음으로써 사람 후보를 분리한다. 다음, 각 히스토그램들에서 얻어진 사각형의 위치가 일치하는 것끼리 합친다. 관련된 사각형 전체를 둘러싸는 영역이 되므로 영역이 더 커지게 된다. Filtered 2-D Space-Depth Histograms

The human candidate is separated by obtaining squares surrounding the human candidate region found at (θ = 0 °, 45 °, 90 °, 135 °). Next, the positions of the squares obtained from the histograms coincide with each other. The area becomes larger because it becomes the area surrounding the entire associated rectangle.

의 최고치를 찾는다. 이 과정은 각각의 필터링 결과인 모두 4개의 히스토그램에 대해서 계산된다.2) Each filtered two-dimensional space-depth histograms based on the merged region for separated person candidates

Find the highs. This process is calculated for all four histograms as a result of each filtering.

 3) A two-dimensional elliptic filter having the highest of the four histogram peaks predicts the corresponding angle as the human pose angle.

Step 110 determines whether the detected person candidate is a real person. First, a head candidate is detected based on a person's average head height and a two-dimensional head size in a region corresponding to the detected person candidate. The person is then validated by performing Adaboost-based face detection on that head candidate. This method works well for people in front of the camera, but if it is not in front of the face is not detected, another method is needed.

In step 112, the person candidate having no face detection verifies the person using head-shoulder shape matching based on a preset shape matching according to the pose angle of the person candidate. Person verification using head-shoulder shape matching is performed by the following process for head-shoulder silhouette extraction, head-shoulder shape extraction, and head-shoulder shape matching.

1) Head-shoulder silhouette is extracted using motion and visual difference information. The silhouette is extracted from the result of the double temporal differencing which performs the temporal difference twice in the input image and the result of the detection of the human candidate in the visual difference map, and fills the holes of the silhouette with dots.

2) After extracting the head-shoulder shape, the shape is extracted by sampling a point at the edge of the extracted head-shoulder silhouette. Since the number of points at a particular location used for shape matching must always be fixed and the direction must be the same, sampling is always performed in a particular direction, eg clockwise.

와 등록된 머리-어깨 형태의 형태 인자 벡터

간의 마할라노비스 거리(Mahalanobis distance)가 작을 경우에 유사한 것으로 판단한다. 여기서 회전각

는 사람 포즈 각도 예측에서 얻어진 사람 포즈 각도이며, 만약 거리가 임계치

보다 작으면 그 형태는 사람이라고 판단하여 검출 결과를 출력한다(112 내지 118 단계).3) The extracted head-shoulder shape matches the pre-registered head-shoulder shape. The extracted head-shoulder shape is not yet aligned, so it cannot be directly applied to shape matching. As a preprocessing of shape matching, the alignment process is performed to remove similarity transformation factors. It is determined whether the sorted input form is similar to the predefined head-shoulder shape. This judgment is the shape parameter vector of the extracted head-shoulder shape.

And registered head-shoulder form factor vector

It is judged to be similar when the Mahalanobis distance of the liver is small. Rotation angle

Is the person pose angle obtained from the person pose angle prediction, and if the distance is the threshold

If smaller, the form is determined to be a person and outputs a detection result (steps 112 to 118).

5A to 5E are diagrams showing the results of detecting one person in the 0 degree, 45 degree, 90 degree, 135 degree, and 180 degree directions, respectively, by the detection method of the present invention described above, and FIGS. 6A to 6B are 0 Fig. 45 shows the results of detecting two people in the 45 degree and 135 degree directions.

Although a preferred embodiment of the present invention has been described in detail above, those skilled in the art to which the present invention pertains may make various changes without departing from the spirit and scope of the invention as defined in the appended claims. It will be appreciated that modifications or variations may be made. Therefore, changes in the future embodiments of the present invention will not be able to escape the technology of the present invention.

1 is a flowchart illustrating a person detection method using stereo image information according to the present invention.

2A is an image of a person when facing the stereo camera in front.

FIG. 2B is a time difference image as a result of calculating the time difference map for each stereo image of FIG. 2A.

FIG. 2C is a spatial-time difference histogram obtained from the time difference image of FIG. 2B.

FIG. 2D is a space-depth histogram converted from the space-time difference histogram of FIG. 2C.

3A to 3D show the directional filters rotated 0 degrees, 45 degrees, 90 degrees, 135 degrees and 180 degrees, respectively, according to the present invention.

FIG. 4 is a detailed flowchart for explaining a process of extracting a human candidate shown in FIG. 1.

5A to 5E are diagrams showing the results of detecting one person in the directions of 0 degrees, 45 degrees, 90 degrees, 135 degrees, and 180 degrees, respectively.

6A to 6B are diagrams showing the results of detecting two people in the directions of 0 degrees, 45 degrees, and 135 degrees.

Claims (8)

(a) generating a two-dimensional space-depth histogram by obtaining a visual difference map from a stereo image; (b) designing directional two-dimensional elliptical filters, which are similar to human shapes, each rotated by a plurality of predetermined angles, as a two-dimensional kernel function; (c) detecting a person candidate through a multiplicative operation between the two-dimensional space-depth histogram and each of the directional two-dimensional elliptic filters; And (d) A human detection method using stereo images for predicting the pose angle of the detected human candidate based on the result of the multiplication of the directional two-dimensional elliptic filter. The method of claim 1, (e) verifying a person by detecting a face from the detected person candidate; And (f) if the verification is not performed in the step (e), further comprising the step of verifying a person by matching the head-shoulder shape. The method of claim 1, wherein the plurality of angles are set to 0 °, 45 °, 90 °, and 135 °. The method of claim 1, wherein step (a) comprises: calculating a visual difference map from the stereo image; (a2) creating a two-dimensional space-visual histogram by projecting the visual difference map in a vertical direction; (a3) converting a two-dimensional space-time histogram into a two-dimensional space-depth histogram. The method of claim 1, wherein step (c) comprises: (c1) calculating four histograms as a result of filtering by performing a multiplicative operation with the two-dimensional space-depth histogram and the four directional two-dimensional elliptic filters; (c2) detecting a person candidate centered on the position of the largest value among the four filtered histograms. The method of claim 2, wherein (f) comprises: (f1) extracting a head-shoulder silhouette from the motion information of a person, which is a double time difference result calculated by performing the temporal difference twice in the stereo image, and the silhouette information that is a result of detection of a person candidate in the visual difference map; (f2) extracting the head-shoulder shape from the head-shoulder silhouette; And (f3) matching the extracted head-shoulder shape with a previously registered head-shoulder shape to compare similarity. The method of claim 6, wherein the extracting the head-shoulder form of (f2) comprises: And detecting and extracting a specific number of points on the edge of the head-shoulder silhouette. 6. The method of claim 5, wherein the step (c2) comprises: setting a region of the person candidate as a range based on the left and right average widths of the person and the thickness values before and after the person based on the position of the largest value. Human detection method using a stereo image.

Images