JP2022037358A - Surrounding environment recognition apparatus - Google Patents

Abstract

To improve the accuracy of identifying a three-dimensional object by properly pairing a back face and side faces of the three-dimensional object.SOLUTION: A surrounding environment recognition apparatus includes: a position derivation unit which derives a three-dimensional position by block in a captured image; a grouping unit which groups blocks of which differences of three-dimensional positions fall within a predetermined range to specify three-dimensional object candidates; a pairing unit which projects the three-dimensional object candidates on a horizontal plane, discriminates a back face from side faces on the basis of angles with respect to a depth direction, and pairs the side faces and back face to specify a pair; and a pair prediction unit which predicts future positions of the pair. The grouping unit specifies three-dimensional object candidates (220h, 220i) on the basis of the future positions of the pair (232, 234) predicted in the past by the pair prediction unit.SELECTED DRAWING: Figure 11

Description

The present invention relates to an external environment recognition device that identifies a three-dimensional object existing in the traveling direction of the own vehicle.

Patent Document 1 includes a technology for detecting a preceding vehicle located in front of the own vehicle to reduce collision damage with the preceding vehicle, and a technology for following control so as to keep the distance between the preceding vehicle and the vehicle at a safe distance. It has been disclosed.

Japanese Patent No. 3349060

In order to reduce the damage caused by the collision between the own vehicle and the preceding vehicle and to realize the follow-up control to the preceding vehicle, the own vehicle first identifies a three-dimensional object existing in the traveling direction, and the three-dimensional object is the preceding vehicle. It is determined whether or not it is a specific object such as. In order to appropriately identify a three-dimensional object, the own vehicle groups, for example, blocks of distance images whose three-dimensional positions are adjacent to each other as the back surface of the three-dimensional object and the side surface of the three-dimensional object. Then, the own vehicle pairs the grouped back surface and the side surface, and identifies them as a pair representing the same three-dimensional object.

However, if noise is generated due to the mismatch of the relative distance in the distance image, the own vehicle cannot properly group the back surface and the side surface of the three-dimensional object, and as a result, it becomes difficult to stably identify the three-dimensional object.

In view of such problems, it is an object of the present invention to provide an external environment recognition device capable of improving the accuracy of specifying a three-dimensional object by appropriately grouping the back surface and the side surface of the three-dimensional object.

In order to solve the above problems, the vehicle exterior environment recognition device of the present invention has a position deriving unit that derives a three-dimensional position in block units in an captured image, and blocks having a difference in three-dimensional positions within a predetermined range. The grouping part that identifies the three-dimensional object candidate by grouping, the three-dimensional object candidate is projected on the horizontal plane, the back surface and the side surface are distinguished based on the angle with respect to the depth direction, and the back surface and the side surface are paired to specify the pair. A pairing unit and a pair prediction unit that predicts the future position of the pair are provided, and the grouping unit identifies a three-dimensional object candidate based on the future position of the pair predicted in the past by the pair prediction unit.

The grouping unit identifies the three-dimensional object candidate corresponding to the back surface based on the distance between the block and the predicted back line indicating the future position of the back surface of the pair predicted in the past in the pair prediction unit, and the pair prediction unit in the past. Candidates for three-dimensional objects corresponding to the sides may be identified based on the distance between the block and the predicted side line indicating the future position of the side of the predicted pair.

According to the present invention, it is possible to improve the accuracy of specifying a three-dimensional object by appropriately pairing the back surface and the side surface of the three-dimensional object.

FIG. 1 is a block diagram showing the connection relationship of the vehicle exterior environment recognition system. FIG. 2 is a functional block diagram showing a schematic function of the vehicle exterior environment recognition device. FIG. 3 is a flowchart showing the flow of the vehicle exterior environment recognition method. FIG. 4 is an explanatory diagram for explaining a luminance image and a distance image. FIG. 5 is an explanatory diagram for explaining the operation of the grouping unit. FIG. 6 is an explanatory diagram for explaining the operation of the pairing unit. FIG. 7 is an explanatory diagram for explaining the determination of whether or not pairing is possible by the pairing unit. FIG. 8 is an explanatory diagram for explaining the correction of the back surface and the side surface. FIG. 9 is an explanatory diagram for explaining the operation of the pair prediction unit. FIG. 10 is an explanatory diagram for explaining the operation of the grouping unit. FIG. 11 is an explanatory diagram for explaining the operation of the grouping unit.

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The dimensions, materials, and other specific numerical values shown in the embodiment are merely examples for facilitating the understanding of the invention, and do not limit the present invention unless otherwise specified. In the present specification and the drawings, elements having substantially the same function and configuration are designated by the same reference numerals to omit duplicate description, and elements not directly related to the present invention are not shown. do.

(External environment recognition system 100)
FIG. 1 is a block diagram showing a connection relationship of the vehicle exterior environment recognition system 100. The vehicle exterior environment recognition system 100 is provided in the own vehicle 1 and includes an image pickup device 110, a vehicle exterior environment recognition device 120, and a vehicle control device 130.

The image pickup device 110 is composed of an image pickup element such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor). The image pickup device 110 is arranged so as to be separated from each other in the substantially horizontal direction so that the optical axes of the two image pickup devices 110 are substantially parallel to each other on the traveling direction side of the own vehicle 1. The image pickup device 110 captures an image of the environment outside the vehicle in front of the own vehicle 1 and generates a luminance image (color image or monochrome image) including at least luminance information. The image pickup apparatus 110 continuously generates a luminance image of a three-dimensional object existing in the detection region in front of the own vehicle 1 every frame (60 fps) of, for example, 1/60 second. Here, the three-dimensional objects recognized by the vehicle exterior environment recognition device 120 are not only independently existing objects such as bicycles, pedestrians, vehicles, traffic lights, road signs, guardrails, and buildings, but also the back and sides of the vehicle and the wheels of the bicycle. Etc., including those that can be identified as part of it. Here, the back surface of the vehicle indicates a surface facing the own vehicle 1 in front of the own vehicle 1, and does not indicate a rear surface of the vehicle itself.

The vehicle exterior environment recognition device 120 acquires a luminance image from each of the two image pickup devices 110, and generates a distance image using so-called pattern matching. The vehicle exterior environment recognition device 120 groups blocks that are located above the road surface, have the same color value, and have three-dimensional position information adjacent to each other as a three-dimensional object, based on a luminance image and a distance image. Then, the vehicle exterior environment recognition device 120 identifies which specific object the three-dimensional object is. For example, the vehicle exterior environment recognition device 120 identifies the three-dimensional object as the preceding vehicle. Further, when the preceding vehicle is specified in this way, the vehicle outside environment recognition device 120 performs reduction control of collision damage between the own vehicle 1 and the preceding vehicle, and also performs follow-up control for causing the own vehicle 1 to follow the preceding vehicle. The specific operation of the vehicle exterior environment recognition device 120 will be described in detail later.

The vehicle control device 130 is composed of an ECU (Electronic Control Unit) or the like, receives a driver's operation input through a steering wheel 132, an accelerator pedal 134, and a brake pedal 136, and transmits the operation input to the steering mechanism 142, the drive mechanism 144, and the braking mechanism 146. By doing so, the own vehicle 1 is controlled. Further, the vehicle control device 130 controls the steering mechanism 142, the drive mechanism 144, and the braking mechanism 146 in accordance with the instructions of the vehicle exterior environment recognition device 120.

(External environment recognition device 120)
FIG. 2 is a functional block diagram showing a schematic function of the vehicle exterior environment recognition device 120. As shown in FIG. 2, the vehicle exterior environment recognition device 120 includes an I / F unit 150, a data holding unit 152, and a central control unit 154.

The I / F unit 150 is an interface for bidirectional information exchange with the image pickup device 110 and the vehicle control device 130. The data holding unit 152 is composed of a RAM, a flash memory, an HDD, and the like, and holds various information necessary for processing of each of the following functional units.

The central control unit 154 is composed of a semiconductor integrated circuit including a central processing unit (CPU), a ROM in which a program or the like is stored, a RAM as a work area, and the like, and is an I / F unit 150 and a data holding unit through a system bus 156. It controls 152 and so on. Further, in the present embodiment, the central control unit 154 also functions as a position derivation unit 170, a grouping unit 172, a pairing unit 174, a pair prediction unit 176, and a three-dimensional object identification unit 178. Hereinafter, the method of recognizing the environment outside the vehicle will be described in detail with reference to the operation of each functional unit of the central control unit 154.

(Method of recognizing the environment outside the vehicle)
FIG. 3 is a flowchart showing the flow of the vehicle exterior environment recognition method. The vehicle exterior environment recognition device 120 executes the vehicle exterior environment recognition method at predetermined interruption times. In the vehicle exterior environment recognition method, the position derivation unit 170 derives a three-dimensional position in block units in the luminance image acquired from the image pickup apparatus 110 (position derivation process S200). The grouping unit 172 groups the blocks together to specify a three-dimensional object candidate (grouping process S202). The pairing unit 174 projects the three-dimensional object candidate onto the horizontal plane, distinguishes between the back surface and the side surface based on the angle with respect to the depth direction, and pairs the back surface and the side surface to specify the pair (pairing process S204). The pair prediction unit 176 predicts the future position of the pair (pair prediction process S206). The three-dimensional object specifying unit 178 specifies which specific object the three-dimensional object or other three-dimensional object constituting the pair corresponds to (three-dimensional object specifying process S208).

(Position derivation process S200)
4 (FIGS. 4A to 4C) are explanatory views for explaining a luminance image and a distance image. The position derivation unit 170 acquires a plurality of luminance images (here, 2) imaged at the same timing by the image pickup device 110 with different optical axes. Here, the position derivation unit 170 uses the first luminance image 212a captured by the image pickup device 110 located on the relatively right side of the own vehicle 1 shown in FIG. 4A and the own vehicle shown in FIG. 4B as the luminance image 212. It is assumed that the second luminance image 212b imaged by the image pickup apparatus 110 located on the relatively left side of 1 is acquired.

With reference to FIGS. 4A and 4B, it is understood that the image positions of the three-dimensional objects included in the images differ in the horizontal direction between the first luminance image 212a and the second luminance image 212b due to the difference in the imaging position of the imaging device 110. can. Here, horizontal indicates the horizontal direction of the screen of the captured image, and vertical indicates the vertical direction of the screen of the captured image.

The position deriving unit 170 determines the distance to be imaged as shown in FIG. 4C based on the first luminance image 212a shown in FIG. 4A and the second luminance image 212b shown in FIG. 4B acquired by the position deriving unit 170. Generates a identifiable distance image 214.

Specifically, the position derivation unit 170 uses so-called pattern matching to derive parallax and parallax information including an image position indicating a position in an image of an arbitrary block. Then, the position derivation unit 170 searches the block corresponding to the block arbitrarily extracted from one luminance image (here, the first luminance image 212a) from the other luminance image (here, the second luminance image 212b). Here, the block is represented by, for example, an array of horizontal 4 pixels × vertical 4 pixels. Further, pattern matching is a method of searching for a block corresponding to a block arbitrarily extracted from one luminance image from the other luminance image.

For example, SAD (Sum of Absolute Difference) that takes the difference in brightness, SSD (Sum of Squared intensity Difference) that squares the difference, and the brightness of each pixel are functions for evaluating the degree of matching between blocks in pattern matching. There is a method such as NCC (Normalized Cross Correlation) that takes the similarity of the variance value obtained by subtracting the average value from.

The position derivation unit 170 performs such a block-based parallax derivation process for all the blocks projected in the detection area of, for example, 600 pixels × 200 pixels. Here, the position derivation unit 170 processes the block as 4 pixels × 4 pixels, but may process it with an arbitrary number of pixels.

The position derivation unit 170 converts the parallax information for each block of the distance image 214 into a three-dimensional position in real space including the horizontal distance x, the height y, and the relative distance z by using the so-called stereo method. Here, the stereo method is a method of deriving the relative distance z of the block to the image pickup device 110 from the parallax in the distance image 214 of the block by using the triangulation method. At this time, the position deriving unit 170 is from the road surface of the block based on the relative distance z of the block and the detected distance on the distance image 214 between the point on the road surface at the same relative distance z as the block and the block. The height y of is derived. Then, the position deriving unit 170 associates the derived three-dimensional position with the distance image 214 again. Since various known techniques can be applied to the process of deriving the relative distance z and the process of specifying the three-dimensional position, the description thereof will be omitted here.

(Grouping process S202)
The grouping unit 172 groups blocks that are located above the road surface, have the same color value, and have a difference in three-dimensional positions in the distance image 214 within a predetermined range, and identifies a three-dimensional object candidate. Specifically, the grouping unit 172 connects blocks in the distance image 214 in which the difference in the horizontal distance x, the difference in the height y, and the difference in the relative distance z are within a predetermined range (for example, 0.1 m). Grouping is performed on the assumption that it corresponds to the same specific object. In this way, a virtual block group is generated.

5 (FIGS. 5A to 5C) are explanatory views for explaining the operation of the grouping unit 172. Here, it is assumed that the position derivation unit 170 generates, for example, the distance image 214 as shown in FIG. 5A. The grouping unit 172 groups the blocks from the distance image 214. In this way, a plurality of grouped blocks are extracted as shown in FIG. 5B. In FIG. 5B, the grouping unit 172 forms a three-dimensional object having an outline including all of the grouped blocks, for example, a horizontal line and a vertical line, or a rectangular frame (face) composed of a line extending in the depth direction and a vertical line. Candidate 220 (220a, 220b, 220c, 220d, 220e) is used.

The three-dimensional object candidates 220a, 220b, 220c, 220d, 220e on the distance image 214 in FIG. 5B are represented by a two-dimensional horizontal plane represented by the horizontal distance x and the relative distance z, and the three-dimensional object candidates 220a, 220b in FIG. 5C. , 220c, 220d, 220e.

(Pairing process S204)
In FIG. 5C, for example, the three-dimensional object candidate 220a and the three-dimensional object candidate 220b are different three-dimensional object candidates. However, the three-dimensional object candidate 220a constitutes the back surface of the preceding vehicle, and the three-dimensional object candidate 220b constitutes the side surface of the preceding vehicle. Therefore, the three-dimensional object candidate 220a and the three-dimensional object candidate 220b should be originally recognized as a pair constituting the same three-dimensional object. Similarly, the three-dimensional object candidate 220d and the three-dimensional object candidate 220e should be recognized as the same three-dimensional object. Therefore, the pairing unit 174 pairs the three-dimensional object candidate 220 that should be the back surface and the side surface of the same three-dimensional object.

FIG. 6 is an explanatory diagram for explaining the operation of the pairing unit 174. FIG. 6 is a projection of the three-dimensional object candidate 220 onto a horizontal plane, and is represented only by the horizontal distance x and the relative distance z.

The pairing unit 174 projects the three-dimensional object candidate 220 onto a horizontal plane, distinguishes between the back surface and the side surface based on the angle with respect to the depth direction, and if the relationship between the back surface and the side surface satisfies a predetermined condition, the back surface and the side surface are paired. Ring to identify the pair.

Specifically, the pairing unit 174 first derives the angle of the approximate straight line of the three-dimensional object candidate 220 with respect to the axis of the relative distance corresponding to the depth direction. If the absolute value of the derived angle is 45 degrees or more and less than 135 degrees, the pairing unit 174 uses the three-dimensional object candidate 220 as the back surface. On the other hand, if the absolute value of the derived angle is 0 degrees or more and less than 45 degrees or 135 degrees or more and 180 degrees or less, the pairing unit 174 uses the three-dimensional object candidate 220 as a side surface.

Here, for example, as a result of grouping by the grouping unit 172, it is assumed that the three-dimensional object candidates 220f and 220g are derived as shown in FIG. 6A. In such an example, as shown in FIG. 6B, the absolute value of the angle of the approximate straight line of the three-dimensional object candidate 220f is 45 degrees or more and less than 135 degrees, so that the pairing unit 174 determines that the three-dimensional object candidate 220f is the back surface. do. On the other hand, since the absolute value of the angle of the approximate straight line of the three-dimensional object candidate 220 g is 0 degrees or more and less than 45 degrees, the pairing unit 174 determines that the three-dimensional object candidate 220 g is a side surface. In this way, the grouping portion 172 can roughly distinguish between the back surface and the side surface.

The pairing unit 174 determines whether or not all combinations of the back surface and the side surface thus distinguished can be paired in a round-robin manner. Therefore, since the pairing unit 174 repeats the pairing determination with all the side surfaces with respect to the back surface by the number of the back surface, the number of processing is the number of the back surface × the number of the side surface. At this time, in the pairing portion 174, the distance between the back surface and the side surface is within an appropriate distance (for example, 2 m) as a part of the vehicle, the speed between the back surface and the side surface is stable, and the back surface and the side surface are respectively. If the length of is equal to or greater than a predetermined value (for example, 1 m) and the center position thereof is within the detection area, pairing may be executed. By adding such a precondition for pairing, the pairing unit 174 can appropriately limit the target of pairing and can reduce the processing load.

FIG. 7 (FIGS. 7A to 7D) is an explanatory diagram for explaining the determination of whether or not pairing is possible by the pairing unit 174. The pairing unit 174 determines whether or not pairing is possible depending on whether or not the relationship between the back surface 222 and the side surface 224 satisfies a predetermined condition. As a result, if it is determined that a predetermined condition is satisfied, the pairing unit 174 identifies the back surface 222 and the side surface 224 of the same three-dimensional object as a pair. Here, the back surface 222 corresponding to the three-dimensional object candidate 220f and the side surface 224 corresponding to the three-dimensional object candidate 220g are listed, and whether or not the pairing is possible is determined. Further, as predetermined conditions, four items of "distance between end points", "velocity vector distance", "intersection distance", and "angle formed" will be described. It should be noted that it is not always necessary to adopt all four items as predetermined conditions, and one or a plurality of items may be adopted as appropriate.

First, as shown in FIG. 7A, the pairing unit 174 identifies the end points of the back surface 222 and the side surface 224 that are closer to each other in the horizontal plane, and derives the distance between the specified end points. Then, if the distance between the end points is less than, for example, 1 m, the pairing unit 174 determines that the condition of "distance between end points" is satisfied.

Next, the pairing unit 174 derives the velocity vector of the back surface 222 and the velocity vector of the side surface 224 in the horizontal plane as shown in FIG. 7B. Then, the pairing unit 174 derives the difference vector of the velocity vectors, that is, the distance between the two velocity vectors in the state where the start points of the two velocity vectors are combined. Then, if the velocity vector distance y is less than, for example, 10 km / h, the pairing unit 174 determines that the condition of the "velocity vector distance" is satisfied.

Subsequently, the pairing unit 174 derives the distance between the intersection of the approximate straight line of the back surface 222 and the approximate straight line of the side surface 224 and the end point on the intersection side of the back surface 222 in the horizontal plane as shown in FIG. 7C. Then, if the intersection distance is less than, for example, 1 m, the pairing unit 174 determines that the condition of "intersection distance" is satisfied.

Next, the pairing unit 174 derives the angle formed by the approximate straight line of the back surface 222 and the approximate straight line of the side surface 224 in the horizontal plane as shown in FIG. 7D. Then, the pairing unit 174 determines that the condition of "angle formed" is satisfied if the angle v formed thereof is close to 90 degrees, for example, in the range of 70 to 110 degrees.

The pairing unit 174 pairs the back surface 222 and the side surface 224 when the four conditions of "distance between end points", "velocity vector distance", "intersection distance", and "angle formed" are satisfied.

Subsequently, the pairing unit 174 corrects the paired back surface 222 and side surface 224 by using the pair of the back surface 222 and the side surface 224 generated in the previous frame.

FIG. 8 (FIG. 8A, 8B) is an explanatory diagram for explaining the correction of the back surface 222 and the side surface 224. FIG. 8 shows the back surface 222 and the side surface 224, as in FIG. 7. The pairing unit 174 first specifies an actually measured value regarding an intersection and an end point with respect to the back surface 222 and the side surface 224 paired with the frame.

Specifically, as shown in FIG. 8A, the pairing unit 174 uses the measured value of the intersection of the approximate straight line of the back surface 222 and the approximate straight line of the side surface 224 as the measured intersection, and sets the measured value of the end point on the intersection side of the back surface 222 as the measured intersection. The measured value of the end point opposite to the intersection side of the back surface 222 is used as the measured back end point, and the measured value of the end point opposite to the intersection side of the side surface 224 is used as the measured side end point. The pairing unit 174 does not use the actual intersection of the back surface 222 and the side surface 224 as it is in deriving the measured value corresponding to the intersection point, but the intersection point and the end point of the back surface 222 on the intersection side, that is, the measured intersection side. Use the midpoint with the end point. This is because the specific accuracy of the back surface 222 is higher than that of the side surface 224, and the end point of the back surface 222 is more reliable.

Here, the intermediate point is obtained by averaging the horizontal distance and the relative distance of each point. The intermediate points are not necessarily averaged, and may be weighted based on their reliability. For example, when the reliability of the end point of the back surface 222 is high, the correction intersection may be obtained with a point closer to the end point on the actually measured intersection side than the actually measured intersection as an intermediate point.

Next, the pairing unit 174 corrects the measured value and derives the correction intersection point, the correction back end point, and the correction side end point. Specifically, the pairing unit 174 uses the intermediate point between the predicted intersection and the intermediate point between the measured intersection and the measured intersection side end point as the corrected intersection, and the intermediate point between the predicted back end point and the actually measured back end point as the corrected back end point. The intermediate point between the predicted side surface end point and the measured side surface end point is defined as the corrected side surface end point. Here, the predicted intersection point, the predicted back end point, and the predicted side end point are those predicted by the pair prediction unit 176 in the previous frame, the intersection point, the back end point, and the side end point in the frame. When the predicted intersection, the predicted back end point, and the predicted side end point are not derived in the previous frame, the pairing unit 174 uses the actually measured intersection, the actually measured back end point, and the actually measured side end point as they are as the predicted intersection and the predicted back end point. , And used as the predicted side end points.

In this way, the pairing unit 174 identifies the correction intersection point, the correction back end point, and the correction side end point as shown in FIG. 8B. In the vehicle exterior environment recognition device 120, the line connecting the corrected intersection point and the corrected back surface end point is the corrected back surface 222, and the line connecting the corrected intersection point and the corrected side surface end point is the corrected side surface 224. The vehicle exterior environment recognition device 120 uses the corrected intersection point, the corrected rear end point, the predicted side end point, the back surface 222, and the side surface 224 to control the reduction of collision damage between the own vehicle 1 and the preceding vehicle, and also controls the own vehicle 1. Follow-up control is performed to follow the preceding vehicle.

(Pair prediction processing S206)
The pair prediction unit 176 predicts the future position of the pair of the back surface 222 and the side surface 224 paired by the pairing unit 174, in this case, the position in the next frame.

FIG. 9 is an explanatory diagram for explaining the operation of the pair prediction unit 176. FIG. 9 is represented only by the horizontal distance x and the relative distance z. The pair prediction unit 176 uses the derived correction intersections, correction back end points, and correction side end points, and the correction intersections, correction back end points, and correction side end points for a plurality of times derived in the past, for example, using a Kalman filter or the like. The next predicted intersection point, predicted back end point, and predicted side end point are derived. At this time, the pair prediction unit 176 predicts the next predicted intersection point, the predicted back end point, and the predicted side end point in consideration of the relative speed of the back surface and the side surface, the angular velocity around the Y axis, and the ego motion.

(Three-dimensional object identification process S208)
The three-dimensional object specifying unit 178 specifies which specific object corresponds to the three-dimensional object or other three-dimensional object composed of the pair of the back surface 222 and the side surface 224 paired by the pairing unit 174. For example, in the three-dimensional object identification unit 178, the three-dimensional object composed of a pair of the back surface 222 and the side surface 224 has a size, shape, and relative speed that are typical of a vehicle, and a brake lamp or a high mount is provided at a predetermined position behind. If it has a light emitting source such as a stop lamp, the three-dimensional object is specified as the preceding vehicle.

As described with reference to FIG. 9, the pair prediction unit 176 predicts the back surface 222 and the side surface 224 in the next frame. Then, as described with reference to FIG. 5, the grouping section 172 groups adjacent blocks together, and as described with reference to FIGS. 6 and 7, the pairing section 174 groups the grouped three-dimensional object candidates 220 with each other. Pair. Then, as described with reference to FIG. 8, the pairing unit 174 corrects the positions of the back surface 222 and the side surface 224 by using the predicted back surface 222 and the side surface 224. As described above, the pairing unit 174 can appropriately correct the positions of the back surface 222 and the side surface 224 by using the back surface 222 and the side surface 224 predicted by the pair prediction unit 176.

However, even if the pairing in the pairing unit 174 is properly performed, the specific accuracy of the three-dimensional object is not improved unless the grouping in the grouping unit 172 in the previous stage is properly performed.

FIG. 10 is an explanatory diagram for explaining the operation of the grouping unit 172. FIG. 10 is represented only by the horizontal distance x and the relative distance z. For example, in the distance image 214, it is assumed that the block as shown in FIG. 10 is extracted. The grouping unit 172 groups blocks whose three-dimensional position differences in the distance image 214 are within a predetermined range. However, if noise is generated due to the mismatch of the relative distance in the distance image 214, the grouping unit 172 may not be able to properly group the back surface 222 and the side surface 224.

For example, in the example of FIG. 10, it is assumed that the grouping unit 172 identifies three three-dimensional object candidates 220h, 220i, and 220j. Here, the three-dimensional object candidate 220h corresponds to the back surface 222, and the three-dimensional object candidate 220i corresponds to the side surface 224. On the other hand, the three-dimensional object candidate 220j is noise and should not be originally a three-dimensional object candidate. Further, depending on the position of the block, the grouping unit 172 may specify the three-dimensional object candidate 220j including such noise. Then, the pairing unit 174 cannot properly pair the back surface 222 and the side surface 224, and it becomes difficult for the vehicle exterior environment recognition device 120 to stably identify these as a three-dimensional object.

Therefore, the grouping unit 172 identifies the three-dimensional object candidate 220 by using the back surface 222 and the side surface 224 in the next frame predicted by the pair prediction unit 176 in the previous frame.

FIG. 11 (FIGS. 11A and 11B) is an explanatory diagram for explaining the operation of the grouping unit 172. The grouping unit 172 preferentially groups the blocks close to the back surface 222 and the side surface 224 predicted by the pair prediction unit 176.

Specifically, as shown in FIG. 11A, the grouping unit 172 generates a predicted back line 232 corresponding to the back surface 222 connecting the predicted intersection point predicted by the pair prediction unit 176 and the predicted back end point. Next, the grouping unit 172 groups the blocks whose distances from the predicted back line 232 are within the predetermined range indicated by the broken line in the figure and whose three-dimensional position difference is within the predetermined range. .. In this way, the grouping unit 172 can identify the three-dimensional object candidate 220h corresponding to the back surface 222. The distance from the block to the predicted back line 232 is a vertical line drawn from the block to the predicted back line 232, and indicates the distance between the intersection of the perpendicular line and the predicted back line 232 and the block.

Similarly, as shown in FIG. 11B, the grouping unit 172 generates a predicted side surface line 234 corresponding to the side surface 224 connecting the predicted intersection point predicted by the pair prediction unit 176 and the predicted side surface end point. Next, the grouping unit 172 groups the blocks whose distances from the predicted side line 234 are within the predetermined range indicated by the broken line in the figure and whose three-dimensional position difference is within the predetermined range. .. In this way, the grouping unit 172 can identify the three-dimensional object candidate 220i corresponding to the side surface 224.

Here, the grouping unit 172 targets blocks whose distance from the predicted side line 234 is within a predetermined range, but the grouping unit 172 is not limited to this case, and various statistical methods can be used. For example, the grouping unit 172 can use the least squares distance and the M estimation to group only the blocks whose variance is within a predetermined range.

Further, the grouping portion 172 may treat the block located at the intersection of the back surface 222 and the side surface 224 as a block belonging to either the back surface 222 or the side surface 224, or on either the back surface 222 or the side surface 224. It may be treated as a block to which only belongs.

With this configuration, even if noise is generated due to the mismatch of the relative distance in the distance image 214, the grouping unit 172 can appropriately identify the three-dimensional object candidate 220h corresponding to the back surface 222 and the three-dimensional object candidate 220i corresponding to the side surface 224. .. As a result, the pairing unit 174 can appropriately pair the grouped three-dimensional object candidates 220h and 220i, and can improve the identification accuracy of the three-dimensional object.

Further, here, the pair prediction unit 176 predicts the position by the pair of the back surface 222 and the side surface 224, but the grouping unit 172 independently executes the grouping of the back surface 222 and the grouping of the side surface 224. .. Therefore, the grouping unit 172 can appropriately identify the back surface 222 and the side surface 224 independently and appropriately.

Further, a program for making a computer function as an external environment recognition device 120, and a storage medium such as a computer-readable flexible disk, a magneto-optical disk, a ROM, a CD, a DVD, and a BD on which the program is recorded are also provided. Here, the program refers to a data processing means described in any language or description method.

Although the preferred embodiment of the present invention has been described above with reference to the accompanying drawings, it goes without saying that the present invention is not limited to such an embodiment. It is clear that a person skilled in the art can come up with various modifications or modifications within the scope of the claims, and it is understood that these also naturally belong to the technical scope of the present invention. Will be done.

For example, in the above-described embodiment, the preceding vehicle is located on the right side of the screen from the own vehicle 1, and the grouping unit 172 specifies the left side surface 224 of the preceding vehicle. However, not limited to this case, when the preceding vehicle is located on the left side of the screen with respect to the own vehicle 1, the grouping unit 172 will specify the side surface 224 on the right side of the preceding vehicle.

Further, in the above-described embodiment, it is specified whether the three-dimensional object candidate 220 is the back surface 222 or the side surface 224, but until the pairing portion 174 is specified, the three-dimensional object candidate 220 is the back surface 222 or the side surface 224. It may not be possible to specify clearly. In this case, the grouping unit 172 may process the two three-dimensional object candidates 220 that can be a pair as the left surface and the right surface, respectively, or may process them as the first surface and the second surface, respectively.

It should be noted that each step of the vehicle exterior environment recognition method of the present specification does not necessarily have to be processed in chronological order according to the order described as a flowchart, and may include parallel processing or processing by a subroutine.

INDUSTRIAL APPLICABILITY The present invention can be used as an external environment recognition device for identifying a three-dimensional object existing in the traveling direction of the own vehicle.

110 Imaging device 120 External environment recognition device 170 Position derivation unit 172 Grouping unit 174 Pairing unit 176 Pair prediction unit 222 Back surface 224 Side surface 232 Prediction back line 234 Prediction side line

Claims (2)

A position derivation unit that derives the three-dimensional position of each block in the captured image,
A grouping unit that groups blocks whose three-dimensional position differences are within a predetermined range to specify a three-dimensional object candidate, and a grouping unit.
A pairing unit that projects the three-dimensional object candidate onto a horizontal plane, distinguishes the back surface and the side surface based on the angle with respect to the depth direction, and pairs the back surface and the side surface to specify the pair.
A pair prediction unit that predicts the future position of the pair,
Equipped with
The grouping unit is an external environment recognition device that identifies the three-dimensional object candidate based on the future position of the pair predicted in the past by the pair prediction unit. The grouping section is
The pair predictor identifies the three-dimensional object candidate corresponding to the back surface based on the distance between the block and the predicted back line indicating the future position of the back surface of the pair predicted in the past.
The external environment according to claim 1, wherein the three-dimensional object candidate corresponding to the side surface is specified based on the distance between the block and the predicted side surface line indicating the future position of the side surface of the pair predicted in the past by the pair prediction unit. Recognition device.

Images