JP2001351193A - Device for detecting passenger - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a passenger detecting device capable of precisely detecting any passenger. SOLUTION: A distance to an object is detected as two-dimensional distance information, and multiple feature values for detecting any passenger are calculated from the distance information, and each feature value is weighted as preliminarily decided, and whether or not the object is any passenger is determined, based on each weighted feature value. For example, the object width and vertical dispersion of an object area are used for the feature values. The feature values of the shape of the passenger are calculated, and each feature value is preliminarily weighted by experiment so that whether or not the object is nay passenger can be discriminated. Thus, it is possible to discriminate the candidate of any passenger from the other objects, and to precisely detect the passenger.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for detecting a pedestrian using an image processing technique.

[0002]

2. Description of the Related Art As a conventional pedestrian detection device, there is one described in, for example, Japanese Patent Application Laid-Open No. 7-200986. In this conventional example, when a moving object is detected in an imaging area by calculating an inter-frame difference between a latest frame photographed by a camera and a previously photographed frame, and the moving object moves in a lateral direction. Is determined to be a pedestrian.

[0003]

However, in the above-described pedestrian detection device, the object detected by the inter-frame difference is not always a pedestrian, so that an object other than a pedestrian may be included. There was a problem that was large. Further, when photographing from a moving vehicle, even if the object is a stationary object, the object position on the image moves horizontally between frames, and there is a problem that a stationary object on the road may be erroneously determined as a moving object. .

[0004] The present invention has been made to solve the problems of the prior art as described above, and has as its object to provide a pedestrian detection device capable of performing accurate pedestrian detection.

[0005]

In order to achieve the above object, the present invention is structured as described in the appended claims. That is, according to the first aspect of the present invention, a distance to an object is detected as two-dimensional distance information, a plurality of feature amounts for pedestrian detection are calculated from the distance information, and each of the feature amounts is determined in advance. A predetermined weighting is performed, and it is configured to determine whether or not a pedestrian is based on each weighted feature amount.

Further, in the invention according to claim 2,
At least the object width and the vertical variance of the object region are calculated as the feature amounts. Note that the vertical distribution means the distribution of height data at each point corresponding to the distance information. Also,
According to a third aspect of the present invention, at least an object width, a vertical variance, and a vertical center of gravity of the object area are calculated as the feature amounts. Note that the vertical center of gravity means the center of gravity of height data at each point corresponding to the distance information. According to a fourth aspect of the present invention, the luminance information of the object (pedestrian candidate), which is temporarily determined to be a pedestrian, on the two-dimensional range image and on the image at a position higher than the region is vertically integrated. The histogram is calculated, and it is determined again whether or not the person is a pedestrian based on the distribution of the integrated value in the vertical direction. Also, the invention described in claim 5 shows an example of a specific configuration of the present invention, and the processing subsequent to the calculation of the characteristic amount is described in claims 2 to 4.
Is the same as any one of the above.

[0007]

According to the first aspect of the present invention, the pedestrian candidate and other objects are obtained by determining the characteristic amount of the shape of the pedestrian and weighting each characteristic amount in advance by an experiment to determine whether the object is a pedestrian. This makes it possible to perform accurate pedestrian detection.

According to a second aspect of the present invention, the object width and the vertical variance are used as the feature amounts, and weighting for discriminating whether or not the pedestrian is a pedestrian determined in advance by an experiment is performed. A more accurate distinction from an object becomes possible.

According to the third aspect of the present invention, since the object width, the vertical variance, and the vertical center of gravity are used as the characteristic amounts, even if the characteristic amounts greatly change due to the lateral movement of the pedestrian, the characteristic amounts are not changed. By adding the vertical center of gravity, pedestrians can be accurately detected.

According to a fourth aspect of the present invention, for a pedestrian candidate area temporarily detected as a pedestrian, the detected area is extended to the upper end of the image, and an integrated histogram of luminance values is obtained in the area. Is determined again based on the distribution of the integrated value of. For example, by calculating a kurtosis which is a statistic indicating a shape characteristic of a histogram,
This makes it possible to distinguish between a telephone pole, which is a stationary object on the road, and a pedestrian, thereby enabling more accurate pedestrian detection.

According to a fifth aspect of the present invention, a three-dimensional positional relationship between the own vehicle and the object in real space is grasped from a positional relationship between the parallax calculated from images captured by the two cameras and the cameras. Therefore, it is possible to distinguish between a stationary object and a moving object even when the camera position is moving. Further, since position data in the real space can be calculated by stereo image processing, it is possible to calculate a feature amount that is hardly affected by the distance between the camera and the object.

[0012]

FIG. 1 is a block diagram showing a configuration of an embodiment of the present invention. In FIG. 1, reference numerals 1 and 2 denote electronic cameras, which are installed at the front of the host vehicle so as to face forward. The optical axes of both cameras are parallel to each other, and the horizontal axes of the imaging surfaces are the same. It is installed so as to be aligned with the top. In addition, it is also possible to arrange so as to face the rear part of the vehicle so as to detect an object behind the vehicle. Also,
The same applies to the case where a stereo camera having two lenses for one camera is used. Reference numerals 3 and 4 denote memories for storing image signals input from the cameras 1 and 2, respectively. Reference numeral 5 denotes an arithmetic unit, which is configured by a microcomputer including, for example, a CPU, a RAM, a ROM, and the like. Reference numeral 6 denotes a detection target existing in front of the host vehicle, which is a vehicle in FIG. 1, but mainly detects a pedestrian in the present invention.

FIG. 2 shows a triangulation source using stereo images.
Explain the principle of finding the distance from the camera to the object
FIG. In FIG. 2, the X axis is horizontal to the ground surface, and the Y axis is
Perpendicular to the ground surface, the Z axis is the optical axis of the left camera (in front of the vehicle
Direction). The left camera (Fig. 1)
Camera 1) and right camera (equivalent to camera 2 in FIG. 1)
Optical axes are parallel to each other and the horizontal axis of the imaging surface is the same.
It is installed so that it is aligned on the screen. In addition, the lens of both cameras
Is not shown in the figure, but is a focal length f before the X axis.
Exists at the location indicated by the local coordinate system. Actual picture
The image is formed at the position of the X axis, but in FIG.
Shown in the table. In the above configuration, the point P on the space
In the left image obtained by shooting (x, y, z), the point P
Projected P _LX_LY_LCoordinates in the local coordinate system
P_L(X_L, Y_L), And similarly, the point P in the right image is
The local coordinate at the time of projection is P_R(X_R, Y_R)
You. At this time, the distance between the two cameras is h, and the focal length is f
X, y, z coordinates of point P (x, y, z)
Can be obtained by the following equation (Equation 1).

[0014] _{x = (x L · h)} / (x L -x R) y = (y L · h) / (x L -x R) = (y R · h) / (x L -x R) ... (Equation 1) z = (f · h) / (x _L −x _R ) Obtaining a point when a point in the space is projected on the left and right images from these equations, in other words, using the left image and the right image By determining the corresponding points, the three-dimensional coordinates of each point in the image can be determined. In particular, in the above optical system, the line connecting the left and right cameras is horizontal and parallel to the X axis, and y _L
= Because it is y _R, it may be determined corresponding relationship between the horizontal line pixels present in the left and right images. That is, the matching positions x _L and x _R (the positions of the right image where the same image as the left image exists) between the two images are obtained, and thereby the parallax (x _L
By obtaining the -x _R), it is possible to measure the distance from the camera to the object by z in equation (1) below.

FIG. 3 shows an image obtained by dividing one of the input left and right images (for example, the left image) into small regions, and using the characteristic edge existing in the region to determine the matching position between the two images. It is a figure showing the result of having calculated. FIG. 3A is a diagram showing a state in which the left image is divided into a plurality of small areas.
The example in which the preceding vehicle exists in front of the road is illustrated. FIG.
(B) is a diagram showing the right image, showing a state in which a part (matching) that matches a certain area in the left image is obtained, and the parallax between the two is obtained. FIG. 3C is a diagram in which a block of an area having the same parallax value is extracted.

As shown in FIG. 3, the parallax (x _L
-X _R ), the distance to the object imaged in the area can be determined by the above (Equation 1). An image in which the distance of an object inside each region is obtained as shown in FIG. 3 is called a “distance image”. In the present embodiment, the above distance image is two-dimensional distance information. The images captured by the two cameras are images corresponding to the luminance of each pixel, and are called “luminance images”. In the above distance image, the distance obtained for each region is a distance to a characteristic edge of an object in the region. Therefore, if there is a similar distance in an adjacent region, it can be determined that the two objects are the same object. For example, a portion in which a region is indicated by a rectangular frame in FIG. 3C is a portion where the value of parallax is the same, and this portion corresponds to a preceding vehicle.

When two cameras are installed so that the optical axes are parallel as shown in FIG. 2, the matching positions of the images are on the same line as shown in FIG. From this, the search for the target region of the right image that matches the left image is performed by shifting the left and right regions on the same line by one pixel in the parallax direction, using the matching degree H (difference value ) Can be performed. H = Σi | Li−Ri | (Equation 2) Since the degree of coincidence H is represented by a difference value, the smaller the value, the more the two coincide.

In the above equation (2), L and R are the luminances of the left image and the right image, respectively, and among the pixels included in one area (window), the positions corresponding to the left image and the right image The absolute value | Li−Ri | of the difference between the two pixels is obtained, and the sum thereof is set to H. The value of H becomes smaller as the brightness of the pixel at the corresponding position between the left image and the right image is closer. From the distance images obtained as described above, an object area is detected as a group of adjacent areas having similar distances. Also, a two-dimensional scanning laser radar (laser range finder) can be used to create a distance image indicating distance information to a forward object.

Next, the x, y, and z coordinates in the real space are calculated from the detected distance image in the object area by the above equation (Equation 1), and the feature quantity indicating the three-dimensional shape of the object is calculated. .
That is, when the feature amount is calculated from the coordinates in a general image, the feature amount greatly varies depending on the distance to be imaged. Therefore, in the present invention, x, y, z in the real space as described above
The configuration is such that calculating the coordinates prevents a change in the feature amount due to the distance.

Further, as the characteristic amount indicating the pedestrian, the object width and the vertical dispersion are taken. FIG. 4 is a diagram for explaining processing in a case where the object width and the vertical dispersion are characterized, and FIG.
And (B) are images in front of the vehicle taken at intervals of t seconds, (C)
Shows a graph in which the object width and the vertical dispersion are taken. Note that the vertical distribution means the distribution of height data at each point corresponding to the distance information. The above two feature amounts in each frame of the pedestrian moving in the horizontal direction are calculated, and the vertical variance is plotted on the Y axis and the object width is plotted on the X axis, and plotted as shown in FIG. However, when the pedestrian moves in the lateral direction, the width increases when the foot is opened and becomes narrower when the foot is closed, so that the object width varies according to the movement of the foot. Therefore, FIG.
Comparing the distributions of pedestrians and other objects in, there is a large variation in the object width of pedestrians, so there is an overlapping part (the part filled in FIG. 4C) in both distributions, and the two groups are completely separated. Can not be done. Therefore, as shown in the following equation (3), when the distribution of the weighted total score S in which a and b are appropriately determined is obtained, the distribution can be divided into upper and lower portions of the broken line in FIG. And the two groups can be completely separated. S = Y−aX + b (Equation 3) The total score S shown in the above (Equation 3) is called a discriminant function.

The above-mentioned discriminant function is obtained by calculating a plurality of
Significant features from features by multivariate discriminant analysis
Can be created by selecting. Representative features and
The distance, object width, horizontal center of gravity, horizontal dispersion, vertical center of gravity, vertical division
Dispersion and dispersion ratio. Multiple features like this
Is weighted, and the following (Equation 4), (Equation 5),
6) A discriminant function as shown in the equation is created, and the value of the total score S is 0
If it is above, it can be determined to be a pedestrian once,
Supplement. S = a₁x₁+ A₂x₂+ B₁x₁ ²+ B₂x₂ ²+ C (Equation 4) S = a₁x₁ ^-1+ A₂x₂ ^-1+ B₁x₁+ B₂x₂+ C (Equation 5) S = a₁x₁+ A₂x₂+ ... + a_ix_i  + B₁x₁ ²+ B₂x₂ ²+ ... + b_ix_i ²  + M₁x₁ ⁿ+ M₂x₂ ⁿ+ ... + m_ix_i ⁿ+ P (Equation 6).

Next, a pedestrian whose total score S becomes 0 or more
Focus on the luminance image of the candidate area. Total score S is 0 or more
Objects include not only pedestrians but also elongated objects such as telephone poles
It is. Therefore, the upper end of the pedestrian candidate area in the brightness image
Spread to the top of the image, and within that area the product of the vertical luminance values
Create an arithmetic histogram. At this time, the histogram
The shape has characteristics as shown in FIG. Note that FIG.
(A) is an image in front of the vehicle, (B) is a distribution of telephone poles,
(C) shows the distribution of pedestrians. As shown in FIG.
In addition, the telephone pole is the same at a specific location
Because there is such a luminance value, the histogram shape is horizontal
At a specific position. On the other hand, FIG.
As shown, when the pedestrian accumulates to the top of the image,
The histogram shape becomes flat due to the influence of the scenery. Accordingly
In order to determine pedestrians with high accuracy,
Is a statistic that indicates features of the histogram shape such as
Calculate the degree k. And when the kurtosis is below the threshold
Is determined to be a pedestrian. k = [n (n + 1) / (n-1) (n-2) (n-3)] Σ [(g_i−g) / S]⁴  -3 (n-1)²/ (N−2) (n−3) (Expression 7) In the expression (Expression 7), g_iAt the horizontal position i of the luminance image
Integrated value of vertical luminance value ingIs the histogram pair
The average value of the brightness of the entire elephant part, s is the standard deviation
It is. Walking car candidate judged to be a pedestrian as described above
By calculating the kurtosis k and determining again,
Pedestrians can be determined more accurately.

Next, an embodiment for detecting a pedestrian using these principles will be described. Here, a stereo camera installed so that the optical axes of the cameras are parallel as shown in FIG. 2 is used. FIG. 6 is a flowchart showing the processing of the embodiment. 6, first, in step S100, the stereo image (right image and left image) of FIG. 2 is input. Next, in step S101, one of the stereo images is divided into a plurality of regions (windows) of the same size. The parallax is calculated for each of the divided areas to determine the distance. The detection of parallax is performed by calculating a difference value between images as shown in Expression (2) as a degree of coincidence. Next, in step S102, a parallax is obtained by performing a matching process. Here, all the coincidences in the search range are obtained, a position where the coincidence is minimum is searched, and the position is set as the parallax of this area. In step S103,
By performing such an operation on all the regions, the distance to the object in each region is calculated by the above-described (Equation 1), and is set as a distance image. Next, in step S104, an object region is detected by finding a cluster of continuous regions in which the distances obtained above have the same value.

Next, in step S105, a feature amount is calculated. The feature amounts effective in the pedestrian detection include the vertical center of gravity (X ₁ ), the vertical variance (X ₂ ), and the object width as shown in the following Expression (8), Expression (9), and Expression (10). (X ₃ ) is used. However, when calculating the feature value, the data in the vertical direction is normalized so that the height is the height from the road surface. Note that the vertical center of gravity means the center of gravity of the height data at each point corresponding to the distance information.

[0025] _{X 1 = Σ ij (y ij} -yob) / n ... Number _{8 X 2 = (1 / n} ) Σ ij _[(y ij _-yob) - _y] ² ... number 9 _X 3 ₌ x r -x _l ... Equation 10 In the above equations (8), (9) and (10), y
_ij and x _ij are x and y coordinates calculated from the equation (1) in the distance image (i, j), yob is the height of the road surface in the real space, and y is all of (y _ij -yob) Is the average of the values.

The height yob of the road surface is set to the height of the road surface if a white line is detected in the distance image, and is set to 0 otherwise. The white line drawn on the road has a strong edge (dark → bright → dark edge in the case of the white line) at the boundary with the asphalt on the road surface, has a higher brightness value than the road surface, and several tens of cm.
The feature is that it has the above width. x
_r, x _l is the value calculated by the the value of the left end point and right point in the distance image of the object area (the number 1).

Next, in step S106, a pedestrian candidate area is extracted. Here, the values of a, b, c, and d, which are linear expressions such as the following Expression (11), are obtained in advance by experiments or the like by a discriminant analysis method. And a feature quantity of the detected object area are substituted into the following equation (11), obtaining a value S _m of S in the object region m. As a result, when the value of the weighted total score S of the three feature amounts as in Expression (11) is equal to or greater than 0, the object region m is set as a pedestrian candidate. S = aX ₁ + bX ₂ + cX ₃ + d (Equation 11) When the value of S is 0 or more, the process proceeds to the next step S107, and when the value of S is less than 0, the process returns to step S105, and the next process is performed. The feature amount is calculated in the object area S _{m + 1} . Such a process is performed for all object regions m.

Next, in step S107, the final decision of the pedestrian is made. The luminance image corresponding to the pedestrian candidate region m, extends the longitudinal direction of a region m to image the upper end, is calculated by the kurtosis k _m showing a histogram shape of the region and the (number 7). As a result, when k _m is the threshold value t ₁ below determines a pedestrian, to calculate the distance and lateral position in step S108. Further, it is determined that the time is greater than the threshold value t ₁ is a utility pole or the like, proceeds to process the next region. Such processing is performed for all pedestrian candidate areas.

In the above description, the case where the present invention is applied to a pedestrian detecting device for a vehicle has been exemplified. However, the present invention is not limited to this case. The present invention can also be applied to a device that detects an approach and automatically opens a door.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of an embodiment of the present invention.

FIG. 2 is a view for explaining the principle of obtaining a distance from a camera to an object based on the principle of triangulation using stereo images.

3A and 3B are diagrams illustrating a result of finding matching positions in input left and right images, where FIG. 3A illustrates a state in which a left image is divided into a plurality of small regions, FIG. 3B illustrates a right image,
FIG. 4C is a diagram in which a block of an area having the same parallax value is extracted.

FIGS. 4A and 4B are diagrams for explaining a process in a case where the object width and the vertical variance are featured. FIGS. 4A and 4B are images taken in front of the vehicle at t-second intervals, and FIG. Graph with width and vertical dispersion.

FIG. 5 is a diagram showing characteristics of a histogram shape;
(A) is an image in front of the vehicle, (B) is a distribution of utility poles, (C)
Is the distribution of pedestrians.

FIG. 6 is a flowchart showing processing in the embodiment.

[Description of Signs] 1, 2 ... Electronic camera 3, 4 ... Memory 5 ... Calculation unit 6 ... Detection target existing in front of own vehicle

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (reference) G06T 1/00 330 G01B 11/24 K G08G 1/04 NF term (reference) 2F065 AA04 AA06 AA22 AA53 BB05 BB15 CC11 CC16 FF05 FF09 GG04 JJ03 JJ05 JJ26 MM16 PP01 QQ03 QQ17 QQ24 QQ25 QQ26 QQ27 QQ28 QQ38 QQ41 QQ42 QQ43 2F112 AC03 BA06 CA05 CA12 FA03 FA21 FA36 FA38 FA45 5B057 AA16 CH11 DA07 DB03 CC03 DC02 DC03 DC02

Claims (5)

[Claims] 1. A distance information detecting means for detecting a distance to an object existing ahead as two-dimensional distance information; and a calculating means for calculating a plurality of predetermined pedestrian detection feature amounts from the distance information. A pedestrian detection device, comprising: a weighting unit that weights each of the feature amounts in advance and determines whether or not the pedestrian is based on each of the weighted feature amounts. 2. A method according to claim 1, further comprising detecting means for detecting an object area where an object is present from the distance information, wherein the calculating means calculates at least an object width and a vertical variance of the object area as a characteristic amount. The pedestrian detection device according to claim 1, wherein a predetermined weight is assigned to the object width and the vertical variance, and whether the object is a pedestrian is determined based on the weighted object width and the vertical variance. 3. A detecting means for detecting an object area in which an object is present from the distance information, wherein the calculating means calculates at least an object width, a vertical variance, and a vertical center of gravity of the object area as a characteristic amount, The determining means performs predetermined weighting on the object width, the vertical variance, and the vertical center of gravity, and determines whether the object is a pedestrian based on the weighted object width, the vertical variance, and the vertical center of gravity. 1 is a pedestrian detection device. 4. An integrated histogram calculating means for calculating a vertical integrated histogram of luminance information for an area on a two-dimensional distance image of an object once determined to be a pedestrian and an area at a position higher than the area on the image. The pedestrian according to any one of claims 1 to 3, further comprising: re-determining means for determining again whether or not the person is a pedestrian based on the distribution of the integrated value in the vertical direction. Detection device. 5. A camera, wherein the optical axes of the cameras are mounted parallel to each other, two cameras for photographing the front, a memory for storing images photographed by the two cameras as digital images, and Means for dividing one of the captured images into small areas of a predetermined size, and detecting, from each of the divided areas, a position of an area having the highest matching degree with that area from the other image. Means for calculating the parallax of each area by using the parallax between each area and the positional relationship between the two cameras. The three-dimensional shape of the pedestrian existing in the area in the real space based on the principle of triangulation. A pedestrian detection device comprising: means for determining a characteristic amount and determining a pedestrian based on the characteristic amount;