JP2010020404A - Image processor and method thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor capable of detecting an object as a candidate area even in such a situation that the object is near or far from another object. <P>SOLUTION: A distance distribution acquisition part 101 acquires the distance distribution information on an object space. A plane calculation part 102 divides the object space into a set of three-dimensional planes from the acquired distance distribution information. A selection part 103 computes the actual size of a plane and compares it with a model of the object, for the set of planes obtained by the plane calculation part 102. The plane determined to have high degree of similarity with the model is sent to an image acquisition part 105, and the plane determined to have low degree of similarity is sent to an extraction part 107. An image pickup part 104 picks up the object space in front of a vehicle. The image obtained in the image pickup part 104 is acquired by a luminance distribution acquisition part 106 through the image acquisition part 105. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and method for detecting a candidate area that is a candidate for an object from an image captured by an imaging unit.

  A technique for detecting an object such as a person, a bicycle, or a motorcycle from an image captured by an image capturing unit such as a camera plays an important role for environment recognition and ensuring safety in a moving object such as an automobile. In many cases, a discriminator based on an image pattern is used to determine whether or not the object is an object.

  In this way, when the object is detected by the classifier, the three-dimensional object candidate region is extracted using the parallax distribution information obtained by stereo vision for the purpose of reducing the amount of calculation and preventing erroneous detection, There is a method of performing determination by a classifier only on a three-dimensional object candidate region.

In Patent Literature 1, a parallax image is generated from a pair of stereo images, and the entire image is grouped based on the continuity of parallax. For each of these groups, the degree of protrusion to the camera side relative to the surroundings is evaluated, and the protruding area is determined as a three-dimensional object candidate area.
JP 2006-72495 A

  In the method of extracting a three-dimensional object candidate region based only on the relative difference in parallax as in Patent Document 1, when a target such as a pedestrian or a bicycle is in front of a large three-dimensional object, the relative parallax is determined. Since the difference between the two is small, there is a problem that the three-dimensional object candidate region is missed.

  In addition, since the disparity in the disparity becomes smaller as the object is farther away, there is a problem that extraction is difficult only with the disparity.

  In other words, the difference in parallax is effective when the object is sufficiently away from the surrounding three-dimensional object or when the object exists near the imaging unit. It wo n’t work.

  Therefore, the present invention provides an image processing apparatus and method capable of extracting a candidate area of a target object even in a situation where the target object is near a standing object or far away from an imaging unit. It is aimed.

  The present invention is an image processing device that detects an object in a target space from an image captured by an imaging unit, and distance distribution information from the imaging unit to a plurality of three-dimensional objects existing in the target space. A distance distribution acquisition unit to acquire, a luminance distribution acquisition unit to acquire luminance distribution information from the image, and a projection plane obtained by projecting the plurality of three-dimensional objects on the two-dimensional imaging surface of the target space viewed from the imaging unit A plane is calculated from the distance distribution information, and each of the projected plane areas is converted into a converted size that is a size in a predetermined real space, each of the converted sizes, A ratio calculation unit that calculates a ratio to a predetermined object size based on the object, and the projection plane area in which the ratio is determined to be within a predetermined range; Based on an edge feature amount in the luminance distribution information corresponding to the projection plane area, with respect to the selection section that selects as a candidate area where an object exists and the projection plane area in which the ratio is determined to be larger than the certain range The image processing apparatus includes: an extraction unit that extracts the candidate region from the projection region; and an output unit that outputs the candidate region selected and extracted by the selection unit and the extraction unit.

  According to the present invention, it is possible to extract a candidate area of a target object even in a situation where the target object is near a standing object or is far from an imaging unit.

  Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings. In the present embodiment, the case where the image processing apparatus is mounted on a vehicle and a pedestrian in front of the vehicle is detected is exemplarily shown, but the present invention is not limited to this. In addition, the space ahead of the vehicle for detecting a pedestrian is called "target space" in the following embodiment.

  A block diagram of the configuration of the image processing apparatus is shown in FIG. As shown in FIG. The image processing apparatus includes a distance distribution acquisition unit 101, a plane calculation unit 102, a selection unit 103, an imaging unit 104, an image capturing unit 105, a luminance distribution acquisition unit 106, an extraction unit 107, and an output unit 108. The functions of these units 101 to 103 and 105 to 108 can be realized by a computer by a computer-readable program stored or transmitted in the computer.

  The distance distribution acquisition unit 101 acquires distance distribution information from image data of a target space imaged by a plurality of imaging units described later. The plane calculation unit 102 divides the target space into a set of three-dimensional planes from the distance distribution information obtained by the distance distribution acquisition unit 101. The selection unit 103 calculates the actual size of the plane for the set of planes obtained by the plane calculation unit 102 and compares it with the model of the object. The selection unit 103 selects a plane having a high similarity with the model based on the comparison result. The plane selected by the selection unit 103 is sent to the output unit 108. On the other hand, the plane determined as having a low similarity by the selection unit 103 is sent to the extraction unit 107. The imaging unit 104 images the target space in front of the vehicle. Image data obtained by the imaging unit 104 is converted into digital image data by the image capturing unit 105 and can be processed by a computer. The luminance distribution acquisition unit 106 acquires luminance distribution information from the image capturing unit 105.

  First, the distance distribution acquisition unit 101 will be described in detail. The distance distribution acquisition unit 101 acquires distance distribution information corresponding to the measurement range of the luminance distribution acquisition unit 106 and outputs the distance distribution information to the plane calculation unit 102. Here, a case where distance distribution information is acquired by stereo image processing will be described as an example. The configuration of the distance distribution acquisition unit 101 at this time is shown in FIG.

  As shown in FIG. 2, the first imaging unit 201 and the second imaging unit 202 image the target space in front of the vehicle. At least two imaging units are required, and the reliability of distance information can be improved by increasing the number of imaging units. In FIG. 2, two image pickup units are shown to simplify the illustration, but may be three or more. The image capturing unit 203 converts the captured information into digital image data and performs various processes. The roles of the first imaging unit 201, the second imaging unit 202, and the image capturing unit 203 are the same as those of the luminance distribution acquisition unit 106. One of the first imaging unit 201 and the second imaging unit 202 and the image capturing unit 203 can use the imaging unit 104 and the image capturing unit 105 of the luminance distribution acquisition unit 106 in common. The parallax estimation processing unit 204 estimates the parallax value distribution between images by performing stereo matching of the image data obtained from the first imaging unit 201 and the second imaging unit 202. The distance distribution information is obtained by using the parallax value of each pixel in the image, the parameter values such as the focal length of the first imaging unit 201 and the second imaging unit 202, and the baseline length between the imaging units.

Here, the simplest parallax estimation process in a stereo image device with two cameras will be described. First, as a premise, it is assumed that the two first imaging units 201 and the second imaging unit 202 are installed in parallel in the vehicle front direction. In the case of such a parallel camera, the vertical coordinates of corresponding points match in the left and right images. In order to obtain the parallax, it is necessary to search the point corresponding to a certain pixel in the image of one imaging unit from the image of the other imaging unit. In this case, since it is known that the vertical coordinate of the corresponding point is the same as that of the original pixel, the search may be performed while changing only the horizontal coordinate. For the search for corresponding points, as shown in FIG. 4, the periphery 404 of the target pixel 403 of the image 401 of one imaging unit is cut out, and the partial image 405 of the same size is cut out of the image 402 of the other imaging unit. evaluate. The degree of similarity can be evaluated using the normalized cross-correlation of the following equation (1).

Then, the larger the _{RNCC of the} formula (1), that is, the closer to 1, the higher the similarity. The similarity calculation of the equation (1) is performed while moving the position of the partial image 405 to search for a point with the highest similarity. Next, when the corresponding point is found, the partial area 405 is fixed and matching is performed on the image 401. If the similarity is maximized around the target pixel, the matching is considered to be successful, and the horizontal coordinate of the partial image is determined. The difference is set as the parallax value of the target pixel 403. A disparity value distribution is obtained by repeating the above processing over the entire image.

  Another process is required to obtain the distance distribution information from the parallax value distribution. In the case of the two first imaging units 201 and 202 installed in parallel, the depth distance Z with respect to the parallax value d is obtained by the following equation (2).

Z = fB / d (2)

In the equation (2), f is a focal length of the camera, and B is a baseline distance between the first imaging unit 201 and the second imaging unit 202. Therefore, these values must be obtained in advance by calibration. If this is repeated over the entire image, distance distribution information can be obtained. However, since the distance is uniquely determined from the parallax value, the following processing is described using the parallax value as it is. These places can be processed in the same way even if they are replaced with distances.

  The distance distribution acquisition unit 101 is not limited to the apparatus that performs the stereo image processing described here, and a laser range finder, a millimeter wave radar apparatus, or the like may be used. In this case, the resolution of the distribution and the accuracy of the distance are different, but the distance distribution acquisition unit 101 and later can be processed in the same manner as the stereo image processing.

  Next, the operation of the image processing apparatus according to this embodiment that detects the pedestrians 504 and 505 from the target space in front of the vehicle where the road surface 501, the side walls 502 and 503, and the pedestrians 504 and 505 exist as shown in FIG. 5. This will be described with reference to the flowchart of FIG.

  In step ST301, the distance distribution acquisition unit 101 acquires distance distribution information of the target space. The operation of the distance distribution acquisition unit 101 is as described above. Although the description will be continued here assuming that the parallax distribution information is obtained when stereo processing is performed instead of the distance distribution information, the same applies to the distance distribution information. In the following description, it is assumed that parallax distribution information corresponding to the two-dimensional imaging surface of the target space in front of the vehicle where the road surface 501, the side walls 502 and 503, and the pedestrians 504 and 505 exist as shown in FIG. 5 is obtained. . However, in the case of stereo image processing, a portion where parallax cannot be obtained occurs, but the parallax in that case contains a value indicating invalidity, and it can be excluded from the processing target below.

  Steps ST302 to ST305 correspond to the processing procedure of the plane calculation unit 102, and interpret the target space from the distance distribution information as a set of three-dimensional planes, that is, as a projected plane area in which a three-dimensional object is projected on a two-dimensional imaging surface. It is the processing method for.

  First, according to the procedure of step ST302 and step ST303, the parallax distribution information corresponding to the non-three-dimensional object is removed from the parallax distribution information, and only the three-dimensional object candidate region is left. In the example of FIG. 5, the road surface 501 is a non-solid object.

In step ST302, the plane calculation unit 102 votes individual parallax distribution information in a u (horizontal coordinate) -d (parallax) space. That is, the number of data having a certain horizontal coordinate u1 and parallax d1 is calculated, and the distribution is obtained. The stereo first imaging unit 201 and second imaging unit 202 of the distance distribution acquisition unit 101 are installed facing the front of the vehicle and substantially horizontal. Therefore, it can be considered that the part with a large number of votes in the ud space is a three-dimensional object standing perpendicular to the ground. Therefore, binarization of the ud space by threshold processing based on the voting result can be divided into a solid object and a non-solid object. There are the following two methods for determining the threshold. In the first method, a constant value T _ud is used for the entire _ud space. In the second method, in the case of a stereo camera, the parallax is large, that is, the closer an object tends to appear larger, so that the threshold is set by the function of d as shown in Equation (3) so as to be proportional to the parallax d. decide.

T _ud (d) = αd + t ₀ (3)

In the equation (3), α and t ₀ are preset constants. In the case of a stereo camera, setting a threshold according to the parallax d is more robust against noise such as erroneous parallax.

  In step ST303, the plane calculation unit 102 performs processing for leaving only a portion corresponding to the three-dimensional object in the distance distribution information as a projection plane area based on the result of the three-dimensional object-non-three-dimensional object determination in step ST302. The procedure of this process is as follows. First, the plane calculation unit 102 reads the parallax distribution information estimated by the parallax estimation processing unit 204. Next, since u (horizontal coordinates), v (vertical coordinates), and d (parallax) data are obtained for each piece of parallax distribution information, coordinates (u, d) are obtained from the binarized ud space in step ST302. Read the data. Next, if the read data is a three-dimensional object, the parallax distribution information is left, and if the read data is a non-three-dimensional object, the parallax distribution information is invalidated. By repeating the above processing for the entire parallax distribution information, parallax distribution information for only the three-dimensional object candidate region can be obtained. In the remaining steps ST304 and ST305, the plane calculation unit 102 actually divides the parallax distribution information of the three-dimensional object candidate area into a projected plane area set.

  In step ST304, the plane calculation unit 102 divides the parallax distribution information for each horizontal line, and divides the one-dimensional parallax distribution information into a set of line segments. FIG. 6 is a diagram illustrating a state in which the parallax distribution information is divided for each horizontal line. One-dimensional parallax distributions corresponding to the horizontal lines 602, 603, and 604 are a distribution 605, a distribution 606, and a distribution 607, respectively. In FIG. 6, a portion where the parallax becomes an invalid value is illustrated as having no data. FIG. 7 shows a processing procedure for dividing the one-dimensional parallax distribution information into a set of line segments.

  First, in step ST701, the plane calculation unit 102 resets the coordinate u from which the parallax distribution information is read to 0. That is, processing is performed in order from the end of the one-dimensional parallax set data.

  Next, in step ST702, the plane calculation unit 102 reads the parallax value d at the coordinate u.

  Next, in step ST703, the plane calculation unit 102 determines whether or not the parallax value d is a valid value. If the parallax distribution information is invalidated for a non-three-dimensional object and the parallax value d is invalid (“No” in step ST703), the process proceeds to step ST712. If the parallax value d is a valid value (“Yes” in step ST703), the process proceeds to step ST704.

The plane calculation unit 102 adds data (u, d) to the element set. Therefore, the element set includes a plurality of horizontal coordinate-parallax coordinates. Let p be the newly added element.

Next, in step ST705, the plane calculation unit 102 applies a least square method to the element set to obtain a straight line passing through the ud space. This means that a straight line fitting is applied to a part of the data distribution in the ud space such as the distribution 605, the distribution 606, and the distribution 607. At this time, the slope a and intercept c of the straight line are

(4) After obtaining a straight line from (5), the straight line

ax + by + c = 0 (6)

And the coefficients are normalized so that a ² + b ² = 1.

Next, in step ST706, the plane calculation unit 102 measures the distance between the obtained straight line and each element of the element set. The distance between the element (u, d) and the straight line ax + by + c = 0 is

| Au + bd + c | (7)

Is required. This calculation is performed for each element in the set to determine the maximum distance D.

  Next, in step ST707, the plane calculation unit 102 determines whether or not the maximum distance value D is greater than a predetermined maximum value T. If D <= T, the degree of matching between the element set including p and the obtained straight line is considered high, and the process proceeds to step ST712. On the other hand, if D> T is satisfied, in steps ST708 and 709, the plane calculation unit 102 has deviated from the straight line when p is included in the element set. Again, straight line fitting is performed by the method of least squares. At the same time, since the u coordinates of the start point and end point of the element set are determined, the output data becomes line segment data of straight line equation + start point + end point.

  Next, in step ST710, the plane calculation unit 102 adds the line segment L thus obtained to the final line segment set. Since it has been found that the element p is not included in the line segment L, the first element of the next line segment is p.

  Therefore, in step ST711, the plane calculation unit 102 once deletes all elements of the element set and then adds p. When such processing is completed, the process proceeds to step ST712.

  Next, in step ST712, the plane calculation unit 102 increments u. That is, the object is moved to the next point.

  Next, in step ST713, the plane calculation unit 102 checks whether or not the end of the data has been reached, and if not, returns to step ST702 and repeats the processing so far.

  If the last data has been reached, in steps ST714 to 716, the plane calculation unit 102 regards the data remaining in the element set as a line segment, determines its parameter, and adds it to the line segment set.

  Finally, the line segment set is output and the process ends.

  The above is the processing procedure of step ST304 in the plane calculation unit 102. Repeat this procedure for all horizontal lines.

  When the processing is performed as described above, a line segment set including a large amount of line segments is generated after step ST304. These line segments can be considered as a part of the projected plane area. Therefore, in step ST305, the plane calculation unit 102 restores the projected plane area by clustering similar line segments. FIG. 8 shows the procedure of the plane division process.

  In step ST801, the plane calculation unit 102 sets the value of the vertical coordinate v to 0, and performs processing in order from the top. At the same time, the uncertain cluster set with the possibility of adding an element and the definite cluster set without adding a line segment are initialized to an empty set. The indeterminate cluster has a counter indicating the lifetime, and when this value becomes 0, it transits to the established cluster. Since a cluster is a set of line segments, it is assumed that it has a line segment parameter representing the set as an attribute.

  Next, in step ST802, the plane calculation unit 102 decreases the value of the counter by 1 for all unconfirmed clusters.

  Next, in step ST803, the plane calculation unit 102 checks whether all line segments existing at the vertical coordinate v have been processed. If the process is finished, the process proceeds to step ST809. If unprocessed line segments remain, the process proceeds to step ST804.

  Next, in step ST804, the plane calculation unit 102 selects one of the line segments and sets the selected line segment to L.

  Next, in step ST805, the plane calculation unit 102 checks the similarity between all undefined clusters and the line segment L. Similar conditions are as follows: (First condition) There is an overlapping portion of the start point-end point u-coordinates. (Second condition) The slope difference of the line segment is equal to or less than a predetermined threshold value. (Third condition) It is determined that they are similar if all three conditions that the difference in slope of the line segment is equal to or less than a predetermined threshold are satisfied. In some cases, the line segment L may be determined to be similar to a plurality of undefined clusters.

  Next, in step ST806, the plane calculation unit 102 checks whether there is an undetermined cluster similar to the line segment L. If there is, the process proceeds to step ST807.

  Next, in step ST807, the plane calculation unit 102 performs a combination process with all unconfirmed clusters whose similarity with the line segment L is confirmed. In the joining process, the same label information is assigned to the line segment L and the line segment included in the target uncertain cluster, and the line segment parameter representing the cluster is updated. The parameter is updated with a weighted sum corresponding to the number of line segments included in the line segment L and the undefined cluster. When the joining process is performed, the count of the indeterminate cluster is returned to the initial value c (predetermined value). That is, as long as the joining process continues, the cluster continues to grow, and conversely, if there is no similar line segment nearby, the growth is aborted. When the combining process ends, the process returns to step ST803. On the other hand, when there is no unconfirmed cluster similar to L, a new unconfirmed cluster is generated in step ST808. The element at this time is only the line segment L. The counter is set to the initial value c, added to the undefined cluster set, and the process returns to step ST803.

  In step ST803, the plane calculating unit 102 proceeds to step ST809 after processing all the line segments.

  Next, in step ST809, the plane calculation unit 102 checks the count of indeterminate clusters to check whether there is a zero one.

  Next, in step ST810, when there is a count 0 undetermined cluster, the plane calculating unit 102 adds the cluster to the confirmed cluster set and terminates further growth.

  Next, in step ST811, the plane calculation unit 102 increments the value of the vertical coordinate v when the cluster state transition processing ends.

  Next, in step ST812, the plane calculation unit 102 confirms whether all the data has been checked, and if there is any remaining data, returns to step ST802 and repeats the processing so far.

  Next, in step ST813, since the plane calculation unit 102 has reached the end of the data, the remaining undefined cluster is added to the confirmed cluster set.

  Finally, the plane calculation unit 102 outputs a definite cluster set, that is, a projected plane area, as a plane division result.

  The above is the processing content for acquiring the plane set in step ST305 in the plane calculation unit 102.

  The cluster set element that is finally output represents one projective plane area in each target space. From the contents of the elements of the cluster, it is possible to obtain information such as how each projection plane area extends in the real space and which area occupies in the stereo image. Using these pieces of information, it is determined in the subsequent steps whether or not the object is a candidate region.

As an example, it is assumed that projection plane regions 901, 902, and 903 represented by the hatched pattern in FIG. It is assumed that the projection plane region 903 comes out with the pedestrian 505 and the side wall 503 integrated. For each projection plane area, (1) a mask pattern (corresponding to the hatched portion in FIG. 9) representing the projection plane area in the image, (2) a projection plane area in u (horizontal coordinate) -d (parallax) space. Information such as a line segment parameter to be expressed is obtained from the result of step ST305. Rectangle including the whole projective plane area such as 1001, 1002, 1003 from the information (a) 10, a simplified like (b) rectangular left end of the disparity value _{d L,} and the right end of the disparity value _{d R} It is also possible to obtain information on the projective plane area.

  Step ST306 to step ST308 correspond to the processing procedure performed in the selection unit 103.

First, in step ST306, the selection unit 103 converts the projection plane area into a size in the real space (real world), compares it with the model of the object, and verifies the similarity. That is, it is determined that a three-dimensional object having the same size as the target object may be the target object. The “object model” here is simply an average size (at least one of width and height) of the object, and in the case of a pedestrian, it is a numerical value such as 65 cm in width and 165 cm in height. is there. Since the actual size of the pedestrian does not match the average size, similarity determination with a certain width is performed. In order to compare with the object model, it is necessary to calculate the actual size from the plane information. This can be simply calculated by the following equations (8) and (9).

In these equations (8) and (9), W and H are the height and width of the projection plane area converted to the real space conversion size, and w and h are the width and height of the rectangle including the projection plane area in the stereo image. That's it. d _L and d _R are parallax values at the left end and the right end of the rectangle, respectively. B is the baseline length between the stereo cameras, and is obtained in advance by calibration.

In step ST307, the selection unit 103 performs the following verification. Once the actual size of the projection plane area is obtained, the comparison with the object model is the ratio between the two sizes.

W / W _M (10)

H / H _M (11)

And verify that each is within the range of the lower limit value and the upper limit value. Here, W _M and H _M are the width and height of the object model, respectively. The upper limit value and lower limit value necessary for determination are set in advance. If it is recognized that there is similarity, the process proceeds to step ST311. In step ST311, the selection unit 103 adds a similar projection plane area to the candidate area set, and performs similar verification in another projection plane area. On the other hand, if it is determined that the similarity with the object model is low, the process proceeds to step ST308.

  In step ST308, the selection unit 103 verifies whether the ratio between the actual size of the projection plane region and the object model is smaller than the lower limit value. If one of the width and the height is below the lower limit value, it is determined that the object is not included in the projection plane area, and the process proceeds to step ST312. Otherwise, the projection plane area is larger than the object, and the process proceeds to step ST309 to verify whether or not there is a possibility that the object is included in the projection plane area.

  Step ST309 and step ST310 correspond to the processing procedure of the extraction unit 107 that processes the luminance distribution information of the target space in front of the vehicle obtained by the luminance distribution acquisition unit 106 and the information of the projection plane region. A flowchart of this process is shown in FIG.

  First, in step ST1101, the extraction unit 107 extracts an edge from the projection plane region of the luminance distribution information. The extraction method is as follows. In the first method, a vertical edge detection Sobel filter composed of the coefficients shown in FIG. 12 is applied. In the second method, the horizontal edge detection Sobel filter shown in FIG. 13 is applied to a projection plane area of luminance distribution information (that is, normal image data). In the third method, the Laplacian filter shown in FIG. 14 is applied. The fourth method applies Canny's edge detection method. In the case of a Sobel filter or a Laplacian filter, the absolute value of the output of the filter becomes large in the vicinity of the corresponding edge. Therefore, by binarizing this value with an appropriate threshold value, whether each pixel in the image is an edge or not Can be determined. Canny's edge detection method also finally obtains binarized information as to whether each pixel is an edge or not by threshold processing. Considering the object, the Sobel filter if the vertical or horizontal edge is characteristic, the Laplacian filter if the edge direction is not important, the Canny method if you want to acquire continuous edges, etc. The edge extraction method may be switched depending on

  Next, in step ST1102, the extraction unit 107 calculates in advance an edge feature amount model for the object. The purpose of this series of processing is to search for a partial region similar to the edge feature amount. A specific example of the edge feature amount will be described later. That is, the edge feature amount model searches for a partial region similar to the edge feature amount. The u coordinate (horizontal coordinate) at the left end of the projection plane area is u1, and the right end is u2.

  Next, in step ST1103, the extraction unit 107 initializes the value of the variable u to u1.

Next, in step ST1104, the extraction unit 107 determines the size of the window width w and height h according to the value of the variable u. This value is determined by the size on the image when the object model is at that position. For this purpose, first, the parallax value on the projection plane area when the horizontal coordinate is u is obtained by complementation. disparity value d _L when the u = _u1, disparity value d at any u since disparity value is d _R when the u = u2 by linear interpolation,

Can be obtained. Once the parallax value d is determined, the width and height of the object model in the image are

w = W _M d / B (13)

h = H _M d / B (14)

It is obtained by. W _M and H _M are the width and height of the object model, respectively, and B is the baseline length of the stereo camera.

  Steps ST1105 and 1106 performed by the extraction unit 107 will be described in detail.

  Since the window size has been determined, in step ST1105, the extraction unit 107 next determines the edge feature in the window whose horizontal coordinate range is [u−w / 2, u + w / 2] and height h from the u−edge amount distribution. Calculate the amount.

As a first specific example of the edge feature amount, there is a total of the number of pixels determined as an edge in the window or a ratio of the area of the pixel determined as an edge in the window and the area of the window. When another object such as a pedestrian is present in front of a large wall surface, a discontinuity in brightness occurs at the boundary between the two objects, and therefore there should be an edge there. Based on such an assumption, in step ST1106, the extraction unit 107 determines whether or not it is a candidate area. The determination method in the first specific example is performed by the edge amount or the area ratio of the edge amount. In this case, the edge feature quantity model is a fixed range of the ratio with the edge quantity or area. Therefore, the comparison with the edge feature amount model in step ST1106 is to determine whether the edge amount or the area ratio is within a predetermined range. If the target is a pedestrian, the vertical edge is characteristic, so it is possible to extract and determine only the vertical edge with a Sobel filter. When a stereo camera is used, the corresponding area in the other image corresponding to the window in one image can be calculated if the camera parameter and the positional relationship between the two cameras are known. If the corresponding area is known, the same determination can be made for the other image. Thereby, the influence of the edge in a different distance from a pedestrian can be removed.

  As a second specific example of the edge feature amount, the shape of the edge itself is used.

  The edge feature amount model in this case is an edge model that shows a typical example of the detection target edge as shown in FIG. In addition, FIG. 15 is an example in case an object is a pedestrian. The determination method in the second specific example is determined by how similar the edge feature model and the edge pattern in the window are. For comparison of the similarity between edge patterns, a chapter distance obtained by totaling the distances from the edge points of one pattern to the edge point nearest to the other pattern at all edge points can be used. If “Chamfer Distance” is equal to or less than a predetermined threshold value, it is determined that the pattern in the window has a high degree of similarity with the edge feature amount model. Even when the edge shape is evaluated, it is possible to perform the determination more stably by calculating the Champion Distance for both of the stereo image pairs.

  As another determination method in the first specific example of the edge feature amount, as shown in FIG. 16, the ratio of the edge amount or area in the window 1601 and the area 1602 thereabove is obtained, and the edge amount of the window 1601 is obtained. There is also a method of confirming that is higher than the threshold and the edge amount of the region 1602 is lower than the threshold. By using such a feature, a tall object such as a utility pole can be removed from the candidate region.

  If the determination in step ST1106 is established, the extraction unit 107 proceeds to step ST1107.

  Next, in step ST1107, the extraction unit 107 cuts out a partial plane from the projection plane area and adds it to the partial plane set. The size of the partial plane is w and h determined in step ST1104, and the center of the horizontal position is u at this point. As shown in FIG. 17, the rectangular position 1702 is set so that the point 1701 that hits the projection plane area for the first time after the search is started from the bottom of the rectangle becomes the bottom of the partial plane area, as shown in FIG. This rectangle becomes a partial plane.

  Next, in step ST1108, the extraction unit 107 determines whether the variable u has reached the right end.

  Next, in step ST1109, the extraction part 107 increments the value of u, when not having reached, and returns to step ST1104.

  If it has reached the end, it outputs a partial plane set and ends.

  When one or more partial planes are output in the process so far, the process proceeds from step ST310 to step ST311. In step ST311, the output unit 108 adds the elements of the partial plane set output in step ST309 to the candidate area set. Finally, the output unit 108 outputs the target object candidate area obtained as a result of the processing so far. The output unit 108 includes the position and size of the candidate area in the image as the minimum information to be output. The parallax / distance information of the area is combined, and the candidate area is either the selection unit 103 or the extraction unit 107. It is also possible to add information indicating whether it was generated by.

  According to the present embodiment, by the above processing procedure, even in a complicated target space that cannot be detected as an object candidate only by the disparity distribution information, it is more stable by using the luminance value information together. Candidate areas can be detected.

  The present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

  For example, in the said embodiment, the target object to detect is demonstrated as a pedestrian. However, candidate areas can be detected by the same process even if other objects such as bicycles other than pedestrians, motorcycles (and people riding on them), and automobiles are different.

1 is a block diagram of an image processing apparatus according to an embodiment of the present invention. It is a figure which shows an example of a structure of a distance distribution acquisition part. It is a figure which shows the flowchart of a target object candidate area | region detection process. It is explanatory drawing of the corresponding point search in a stereo image process. It is a figure which shows an example of the object space ahead of a vehicle. It is a figure which shows the division | segmentation for every horizontal line of parallax distribution information. It is a figure which shows the flowchart of a line segment division | segmentation process. It is a figure which shows the flowchart of a plane division process. It is a figure which shows an example of a plane division | segmentation process result. It is explanatory drawing of the information obtained by simplifying a plane. It is a figure which shows the flowchart of the extraction process of a partial plane. It is a figure which shows the coefficient of the Sobel filter for vertical edge extraction. It is a figure which shows the coefficient of the Sobel filter for horizontal edge extraction. It is a figure which shows the coefficient of the Laplacian filter for edge extraction. It is a figure which shows an example of a pedestrian's edge shape model. It is explanatory drawing of edge amount evaluation of two area | regions. It is explanatory drawing of the determination method of a partial plane cutout position.

Explanation of symbols

101 distance distribution acquisition unit 102 plane calculation unit 103 selection unit 104 imaging unit 105 image capture unit 106 luminance distribution acquisition unit 107 extraction unit 108 output unit

Claims (7)

An image processing apparatus that detects an object in a target space from an image captured by an imaging unit,
A distance distribution acquisition unit that acquires distance distribution information from the imaging unit to a plurality of three-dimensional objects existing in the target space;
A luminance distribution acquisition unit for acquiring luminance distribution information from the image;
A plane calculation unit that calculates, from the distance distribution information, a projection plane region obtained by projecting the plurality of three-dimensional objects on the two-dimensional imaging surface of the target space viewed from the imaging unit;
Each size of each projection plane area is converted into a conversion size that is a size in a predetermined real space, and a ratio between each of the conversion sizes and a predetermined object size based on the object is calculated. A selection unit that calculates each of the projection plane regions that are calculated and determined that the ratio is within a predetermined range, as candidate regions in which the object exists;
The candidate area is extracted from the projection area of the projection plane area determined to be larger than the certain range based on the edge feature amount in the luminance distribution information corresponding to the projection plane area. An extractor;
An output unit that outputs the candidate areas selected and extracted by the selection unit and the extraction unit, respectively;
An image processing apparatus comprising: The extraction unit includes:
Obtaining the edge in the projection plane region from the luminance distribution information;
A window obtained by converting the object size into a size in the image is set while sequentially shifting within the projection plane area,
Calculate the edge feature amount in each window,
The image processing apparatus according to claim 1, wherein the window whose edge feature amount is higher than a threshold is cut out as the candidate region.   The image processing apparatus according to claim 1, wherein the ratio calculation unit compares at least one of a width and a height of the projection plane area with the object size. The plane calculation unit
The distance distribution information is divided for each horizontal line,
Dividing the parallax of each horizontal line into a set of line segments;
Find the similarity of the set of line segments,
Based on each similarity, cluster the line segment set to form a cluster set;
The image processing apparatus according to claim 1, wherein each cluster set is set in the projection plane area.   The image processing apparatus according to claim 1, wherein the output unit outputs the cluster sets having similarities with the same label information added thereto. An image processing method for detecting an object in a target space from an image captured by an imaging unit,
A distance distribution acquisition step of acquiring distance distribution information from the imaging unit to a plurality of three-dimensional objects existing in the target space;
A luminance distribution acquisition step of acquiring luminance distribution information from the image;
A plane calculation step of calculating, from the distance distribution information, a projection plane area obtained by projecting the plurality of three-dimensional objects on the two-dimensional imaging surface of the target space viewed from the imaging unit;
Each size of each projection plane area is converted into a conversion size that is a size in a predetermined real space, and a ratio between each of the conversion sizes and a predetermined object size based on the object is calculated. A selection step of calculating each of the projection plane areas, each of which is calculated and determined that the ratio is within a predetermined range, as a candidate area where the object exists;
The candidate area is extracted from the projection area of the projection plane area determined to be larger than the certain range based on the edge feature amount in the luminance distribution information corresponding to the projection plane area. An extraction step;
An output step for outputting the selected and the extracted candidate regions, respectively;
An image processing method comprising: An image processing program for detecting an object in a target space from an image captured by an imaging unit,
A distance distribution acquisition function for acquiring distance distribution information from the imaging unit to a plurality of three-dimensional objects existing in the target space;
A luminance distribution acquisition function for acquiring luminance distribution information from the image;
A plane calculation function for calculating, from the distance distribution information, a projection plane area obtained by projecting each of the plurality of three-dimensional objects on a two-dimensional imaging surface of the target space viewed from the imaging unit;
Each size of each projection plane area is converted into a conversion size that is a size in a predetermined real space, and a ratio between each of the conversion sizes and a predetermined object size based on the object is calculated. A selection function for calculating each of the projection plane regions that are calculated and determined that the ratio is within a predetermined range as a candidate region in which the object exists;
The candidate area is extracted from the projection area of the projection plane area determined to be larger than the certain range based on the edge feature amount in the luminance distribution information corresponding to the projection plane area. Extraction function;
An output function for outputting each of the selected and extracted candidate regions;
An image processing program for realizing a computer.

Images