JP7071102B2 - External environment recognition device - Google Patents

Description

The present invention relates to an external environment recognition device that identifies a three-dimensional object in front of the own vehicle.

Conventionally, there is known a technique for identifying a three-dimensional object such as a vehicle, a two-wheeled vehicle, or a pedestrian located in front of the own vehicle based on an image captured by an image pickup device (for example, Patent Documents 1 and 2).

JP-A-2015-195018 Japanese Unexamined Patent Publication No. 2010-097541

When the above image pickup device captures the motorcycle from the front or the rear, it is difficult to distinguish the characteristic parts such as the wheels of the motorcycle. Therefore, even with the techniques of Patent Documents 1 and 2, it may be difficult to distinguish between a two-wheeled vehicle and a pedestrian. Therefore, it is desired to develop a technique for improving the accuracy of distinguishing between a two-wheeled vehicle and a pedestrian.

In view of such problems, it is an object of the present invention to provide an external environment recognition device capable of improving the discrimination accuracy between a two-wheeled vehicle and a pedestrian.

In order to solve the above problems, in the vehicle exterior environment recognition device of the present invention, a computer arranges a candidate identification unit for specifying a partial image of a pedestrian or a two-wheeled vehicle in an image and the partial image in the horizontal direction of the image. It functions as a representative specifying unit that specifies the representative value of the distance for each of the plurality of vertical regions, and a determination unit that determines whether the partial image is a pedestrian or a two-wheeled vehicle based on the variation of the representative values in the plurality of vertical regions.

When the index value of the variation of the representative value exceeds a predetermined threshold value, the determination unit may determine that the partial image is a two-wheeled vehicle.

The computer may further function as a compensator to reduce the estimated distance to the motorcycle than it was before it was determined to be a motorcycle.

According to the present invention, it is possible to improve the discrimination accuracy between a two-wheeled vehicle and a pedestrian.

It is a block diagram which showed the connection relation of the outside environment recognition system. It is explanatory drawing for demonstrating a luminance image and a distance image. It is a functional block diagram which showed the schematic function of the vehicle exterior environment recognition device. It is a flowchart which shows the flow of the vehicle outside environment recognition processing. It is a figure for demonstrating the difference between a pedestrian and a two-wheeled vehicle.

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The dimensions, materials, and other specific numerical values shown in the embodiment are merely examples for facilitating the understanding of the invention, and do not limit the present invention unless otherwise specified. In the present specification and the drawings, elements having substantially the same function and configuration are designated by the same reference numerals to omit duplicate explanations, and elements not directly related to the present invention are not shown. do.

(External environment recognition system 100)
FIG. 1 is a block diagram showing a connection relationship of the vehicle exterior environment recognition system 100. The vehicle exterior environment recognition system 100 includes an image pickup device 110, a vehicle exterior environment recognition device 120, and a vehicle control unit (ECU) 130.

The image pickup device 110 is configured to include an image pickup element such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor), captures an image of the outside environment in front of the own vehicle 1, and includes at least luminance information. It is possible to generate a luminance image (color image or monochrome image). Further, the image pickup device 110 is arranged so as to be separated from each other in the substantially horizontal direction so that the optical axes of the two image pickup devices 110 are substantially parallel to each other on the traveling direction side of the own vehicle 1. The image pickup apparatus 110 continuously generates a luminance image of a three-dimensional object existing in the detection region in front of the own vehicle 1 every frame (60 fps) of, for example, 1/60 second. Here, if the three-dimensional object recognized by the image pickup device 110 is only an independently existing object such as a two-wheeled vehicle (bicycle or the like), a pedestrian (person), a vehicle, a traffic light, a road (travel path), a road sign, a guardrail, or a building. It also includes objects that can be identified as part of it, such as the head and shoulders of pedestrians.

Further, the outside environment recognition device 120 acquires a luminance image from each of the two image pickup devices 110, and blocks corresponding to a block (for example, an arrangement of 4 horizontal pixels × 4 vertical pixels) arbitrarily extracted from one of the luminance images. Using so-called pattern matching, which is searched from the other luminance image, disparity information including disparity and a screen position indicating the position of an arbitrary block in the screen is derived. Here, horizontal indicates the horizontal direction of the screen of the captured image, and vertical indicates the vertical direction of the screen of the captured image. As this pattern matching, it is conceivable to compare the luminance (Y) between a pair of images in an arbitrary block unit. For example, SAD (Sum of Absolute Difference) that takes the difference in brightness, SSD (Sum of Squared intensity Difference) that uses the difference squared, and NCC that takes the similarity of the dispersion value obtained by subtracting the average value from the brightness of each pixel. There are methods such as (Normalized Cross Correlation). The vehicle exterior environment recognition device 120 performs such parallax derivation processing in block units for all blocks projected in the detection area (for example, 600 pixels × 200 pixels). Here, the block is 4 pixels × 4 pixels, but the number of pixels in the block can be set arbitrarily.

However, in the vehicle exterior environment recognition device 120, although the parallax can be derived for each block, which is a detection resolution unit, it is not possible to recognize what kind of three-dimensional object the block is a part of. Therefore, the parallax information is independently derived not in units of three-dimensional objects but in units of detection resolution (for example, in units of blocks) in the detection region. Here, the image associated with the parallax information derived in this way is referred to as a distance image to distinguish it from the above-mentioned luminance image.

FIG. 2 is an explanatory diagram for explaining the luminance image 126 and the distance image 128. For example, it is assumed that the luminance image 126 as shown in FIG. 2A is generated for the detection region 124 through the two image pickup devices 110. However, here, for ease of understanding, only one of the two luminance images 126 is schematically shown. The vehicle exterior environment recognition device 120 obtains the parallax for each block from such a luminance image 126, and forms a distance image (image) 128 as shown in FIG. 2 (b). Each block in the distance image 128 is associated with the parallax of that block. In FIG. 2, for convenience of explanation, the block from which the parallax is derived is represented by black dots.

Further, the vehicle exterior environment recognition device 120 is three-dimensional in a real space including a luminance value (color value) based on the luminance image 126 and a relative distance (distance) to the own vehicle 1 calculated based on the distance image 128. Using the position information of, groups blocks with the same color value and similar three-dimensional position information as a three-dimensional object, and specify what the three-dimensional object is in the detection area in front of the own vehicle 1 (for example, a pedestrian). do. Further, when the vehicle exterior environment recognition device 120 identifies a three-dimensional object in this way, the own vehicle 1 can avoid a collision with the three-dimensional object (collision avoidance control) and keep the distance between the vehicle and the preceding vehicle at a safe distance. Control (cruise control). The relative distance is obtained by converting the parallax information for each block in the distance image 128 into three-dimensional position information using the so-called stereo method. Here, the stereo method is a method of deriving the relative distance of the stereoscopic object to the image pickup device 110 from the parallax of the stereoscopic object by using the triangulation method.

Returning to FIG. 1, the vehicle control device 130 receives the driver's operation input through the steering wheel 132, the accelerator pedal 134, and the brake pedal 136, and transmits the operation input to the steering mechanism 142, the drive mechanism 144, and the braking mechanism 146. Control the vehicle 1. Further, the vehicle control device 130 controls the steering mechanism 142, the drive mechanism 144, and the braking mechanism 146 in accordance with the instructions of the vehicle exterior environment recognition device 120.

Hereinafter, the configuration of the vehicle exterior environment recognition device 120 will be described in detail. Here, the specific processing of a three-dimensional object (for example, a pedestrian and a two-wheeled vehicle) in the detection region in front of the own vehicle 1, which is characteristic of the present embodiment, will be described in detail, and a configuration unrelated to the characteristics of the present embodiment will be described. Omit.

(External environment recognition device 120)
FIG. 3 is a functional block diagram showing a schematic function of the vehicle exterior environment recognition device 120. As shown in FIG. 3, the vehicle exterior environment recognition device 120 includes an I / F unit 150, a data holding unit 152, and a central control unit 154.

The I / F unit 150 is an interface for bidirectional information exchange with the image pickup device 110 and the vehicle control device 130. The data holding unit 152 is composed of a RAM, a flash memory, an HDD, and the like, and holds various information necessary for processing of each of the following functional units.

The central control unit 154 is composed of a semiconductor integrated circuit including a central processing unit (CPU), a ROM in which a program or the like is stored, a RAM as a work area, and the like, and is an I / F unit 150 and a data holding unit through a system bus 156. It controls 152 and so on. Further, in the present embodiment, the central control unit 154 also functions as a three-dimensional object region identification unit 160, a candidate identification unit 162, a representative identification unit 164, a candidate determination unit 166, and a distance correction unit 168. Hereinafter, the vehicle exterior environment recognition process for recognizing pedestrians and motorcycles, which is characteristic of the present embodiment, will be described in detail with reference to the operation of each functional unit of the central control unit 154.

(Outside environment recognition processing)
FIG. 4 is a flowchart showing the flow of the vehicle exterior environment recognition process. In the vehicle exterior environment recognition process, first, the three-dimensional object area specifying unit 160 executes the three-dimensional object area specifying process (S200) for specifying the three-dimensional object area including (photographed) the three-dimensional object in the luminance image 126 and the distance image 128. To. Then, the candidate identification unit 162 executes a candidate identification process (S202) for specifying a three-dimensional object region (partial image) including a three-dimensional object of a pedestrian or a two-wheeled vehicle. When a three-dimensional object region including a three-dimensional object of a pedestrian or a two-wheeled vehicle is specified, the representative identification unit 164 performs a representative value identification process (S204) for specifying a representative value described later, and then a candidate determination unit (determination unit). 166, a candidate determination process (S206) for determining whether or not the three-dimensional object is a two-wheeled vehicle is executed, and finally, a correction process (S208) in which the distance correction unit (correction unit) 168 corrects the relative distance is performed. Will be executed. The vehicle exterior environment recognition process is repeatedly executed for each frame in which the luminance image 126 and the distance image 128 are acquired.

(Three-dimensional object area identification process S200)
As described above, the three-dimensional object region specifying unit 160 identifies a three-dimensional object by grouping in the detection regions of the luminance image 126 and the distance image 128. The three-dimensional object region specifying unit 160 extracts a three-dimensional object region (partial image) including (photographed) the specified three-dimensional object from the luminance image 126 and the distance image 128.

(Candidate identification process S202)
Subsequently, the candidate specifying unit 162 identifies what the specified three-dimensional object is based on the extracted three-dimensional object region. For example, as described above, the three-dimensional object includes types such as two-wheeled vehicles, pedestrians (people), vehicles, traffic lights, roads (travel paths), road signs, guardrails, buildings, and pedestrian heads and shoulders. Will be. Setting conditions are set in advance for each type, and the specified three-dimensional object is determined to be the corresponding type when the three-dimensional object region meets the setting conditions.

Here, it may be difficult to distinguish between pedestrians and motorcycles. For example, it is difficult to identify a two-wheeled vehicle on which a person is riding facing the front or the back of the image pickup device 110. The wheels, which are characteristic of motorcycles, become, for example, black rod-shaped objects, and it is difficult to identify the contour. Moreover, the upper body of a person located above the two-wheeled vehicle is indistinguishable from that of a pedestrian. Here, the two-wheeled vehicle means, for example, a motorcycle having one front wheel and one rear wheel (pair).

Therefore, the candidate identification unit 162 does not identify whether the three-dimensional object is a pedestrian or a two-wheeled vehicle, but specifies it as a type of "pedestrian or two-wheeled vehicle". The candidate specifying unit 162 identifies the "pedestrian or two-wheeled vehicle" by, for example, the contour shape, size, moving speed, or the like of a three-dimensional object.

(Representative value specifying process S204)
FIG. 5 is a diagram for explaining the difference between a pedestrian and a two-wheeled vehicle. In FIG. 5, two regions A and B of the distance image 128 are extracted and shown. A partial image 300 including a pedestrian is located in the area A, and a partial image 302 including a two-wheeled vehicle is located in the area B. Further, images 304 and 306 of representative values of relative distances are shown below the regions A and B, respectively.

As shown in FIG. 5, in the partial image 302, the two-wheeled vehicle faces the rear surface, and as described above, the rear wheel 308 is difficult to identify. Therefore, the candidate identification unit 162 determines that the three-dimensional object is a "pedestrian or a two-wheeled vehicle" for both the partial images 300 and 302.

However, it may be possible to identify whether the three-dimensional object is a pedestrian or a two-wheeled vehicle. For example, if the moving speed is equal to or higher than the threshold value and cannot be a pedestrian, it is specified as a two-wheeled vehicle. In such a case, the candidate identification unit 162 identifies whether it is a pedestrian or a two-wheeled vehicle without performing the following processing of the representative identification unit 164 and the candidate determination unit 166.

Subsequently, the representative identification unit 164 sets a representative value of the relative distance for each of the plurality of vertically long vertical regions S extending in the upper and lower ends of the image (screen) arranged in the horizontal direction of the partial images 300 and 302. Identify. Specifically, the distance image 128 is divided into a plurality of vertical regions S divided in the horizontal direction (horizontal direction) of the distance image 128.

For any vertical region S, a frequency distribution of relative distances of all blocks contained within the vertical region S is generated. In the frequency distribution, for example, the number of blocks having relative distances belonging to each of the distance divisions divided by predetermined distances is shown as the frequency. Here, when comparing the same three-dimensional objects, the closer the relative distance is, the larger the area occupied by the three-dimensional objects in the distance image 128. Therefore, for example, the center value of the distance division having the highest frequency is taken as the representative value of the relative distance of the vertical region S. As a result, the relative distance of the three-dimensional object close to the own vehicle 1 is likely to be used as the representative value. Further, even if there is noise in a block having a short relative distance, the influence on the representative value is avoided if the frequency is not high.

Here, the case where the center value of the distance division having the highest frequency is the representative value has been described. However, for example, among the distance categories in which the frequency is equal to or higher than a predetermined value, the center value of the distance category closest to the own vehicle 1 may be the representative value. Further, the representative value may represent the relative distance of the blocks in the vertical region S, and may be, for example, the average value of the relative distances in the vertical region S.

Representative values are similarly specified for all vertical regions S. In the image diagrams 304 and 306, the representative values of the relative distances of the vertical regions S are shown by black squares. In the images 304 and 306, it is shown that the relative distance is longer toward the upper side in the figure.

The representative value of the relative distance is used for the above grouping. For example, the adjacent vertical regions S are compared with each other, and a vertical region group in which the vertical regions S having close representative values are grouped is generated. Within this vertical region group, blocks with a relative distance close to the representative value as a base point, and blocks located in the vicinity within a predetermined range such as relative distance, horizontal distance, height, etc. are grouped as one solid object. Be made.

Therefore, the representative specifying unit 164 specifies the representative value of the vertical region S before the grouping process of the three-dimensional object region specifying unit 160. The three-dimensional object region specifying unit 160 performs the grouping process as described above by using the representative value specified by the representative specifying unit 164.

However, the representative value does not have to be used for the grouping process by the three-dimensional object region specifying unit 160. In this case, the representative specifying unit 164 may, for example, divide only the partial images 300 and 302 into the vertical region S and specify the representative value of the vertical region S. At this time, the representative value may be, for example, the average value of the relative distances in the blocks in the lower predetermined range among the partial images 300 and 302.

(Candidate determination process S206)
The candidate determination unit 166 determines whether the partial images 300 and 302 are pedestrians or motorcycles based on the variation of the representative values in the plurality of vertical regions. As shown in image FIG. 304, the representative value of the relative distance of the vertical region S including the partial image 300 of the pedestrian has a small variation. On the other hand, as shown in image FIG. 306, the representative value of the relative distance of the vertical region S including the partial image 302 of the motorcycle has a large variation. This is because the relative distance of the rear wheels 308 of the two-wheeled vehicle is smaller than that of the person riding the two-wheeled vehicle.

The candidate determination unit 166 uses this characteristic to discriminate between a pedestrian and a two-wheeled vehicle. When the index value of the variation of the representative value exceeds a predetermined threshold value, the candidate determination unit 166 determines that the partial image 302 is a two-wheeled vehicle. Here, the index value of variation may be, for example, a statistical index such as variance or standard deviation, or may simply be the difference between the maximum value and the minimum value of the representative value. In any case, it suffices to show the degree of variation in the representative value.

(Correction processing S208)
As described above, the relative distance to a three-dimensional object such as a pedestrian or a two-wheeled vehicle is estimated from the representative value of the relative distance. However, it is difficult to identify the contours of the wheels of a two-wheeled vehicle (rear wheels 308 in the case of the rear surface and front wheels in the case of the front surface). Further, in general, the wheels of a two-wheeled vehicle have a narrow width, so that it is difficult to obtain accurate parallax information. As a result, the relative distance to the two-wheeled vehicle in the partial image 302 is originally the relative distance to the end of the rear wheel 308 (the end on the one side of the own vehicle), for example, to the back of the person riding the two-wheeled vehicle. Will be the relative distance of. In this way, the relative distance is derived larger than it actually is.

That is, in the image FIG. 306, the representative value of the vertical region S including the rear wheel 308 is a value farther from the own vehicle 1 than the actual end portion of the rear wheel 308. This hinders the control of the own vehicle 1 to maintain an appropriate distance from the two-wheeled vehicle.

Therefore, when the partial image 302 is determined to be a motorcycle, the distance correction unit 168 corrects the estimated value (estimated distance) of the relative distance to the motorcycle corresponding to the partial image 302. That is, the distance correction unit 168 reduces the estimated distance to the motorcycle from the value before the candidate determination unit 166 determines that the motorcycle is a motorcycle (for example, the value derived by the representative identification unit 164). Specifically, the distance correction unit 168 makes a correction by subtracting a predetermined value (for example, 70 cm or the like) from the estimated distance to the motorcycle. The size of the wheels of a two-wheeled vehicle is generally limited, and the predetermined value is set on the safe side so as to correspond to the larger wheel.

As described above, in the present embodiment, the candidate determination unit 166 makes it possible to appropriately distinguish between a pedestrian and a two-wheeled vehicle. Further, for a two-wheeled vehicle, by reducing the estimated distance, an appropriate distance between the own vehicle 1 and the two-wheeled vehicle can be maintained.

In addition, a program that causes the computer to function as the vehicle exterior environment recognition device 120, and a storage medium such as a computer-readable flexible disk, magneto-optical disk, ROM, CD, DVD, or BD that records the program are also provided. Here, the program refers to a data processing means described in any language or description method.

Although the preferred embodiment of the present invention has been described above with reference to the accompanying drawings, it goes without saying that the present invention is not limited to such an embodiment. It is clear that a person skilled in the art can come up with various modifications or modifications within the scope of the claims, and it is understood that these also naturally belong to the technical scope of the present invention. Will be done.

For example, in the above-described embodiment, the candidate determination unit 166 has described a case where the partial image 302 determines that the partial image 302 is a motorcycle when the index value of the variation of the representative value exceeds a predetermined threshold value. However, if the determination is made based on the variation in the representative value, the comparison process between the index value and the threshold value is not essential.

Further, in the above-described embodiment, the case where the vehicle exterior environment recognition device 120 also functions as the distance correction unit 168 has been described. However, the distance correction unit 168 is not an essential configuration.

Further, in the above-described embodiment, the case where the vehicle exterior environment recognition device 120 also functions as the candidate identification unit 162 and the representative identification unit 164 has been described. However, if the vehicle exterior environment recognition device 120 functions as the candidate determination unit 166 and the distance correction unit 168, it does not have to function as the candidate identification unit 162 and the representative identification unit 164. In this case, the candidate determination unit 166 may determine at least whether or not the partial images 300 and 302 are motorcycles. The determination logic of the two-wheeled vehicle is not limited to the above-mentioned one, and any logic may be used.

It should be noted that each step of the vehicle exterior environment recognition process of the present specification does not necessarily have to be processed in chronological order according to the order described as a flowchart, and may include parallel or subroutine processing.

The present invention can be used as an outside environment recognition device for identifying a three-dimensional object in front of the own vehicle.

120 External environment recognition device 128 Distance image (image)
160 Three-dimensional object area identification part 162 Candidate identification part 164 Representative identification part 166 Candidate judgment part (judgment part)
168 Distance correction unit (correction unit)
300, 302 partial image

Claims (3)

The computer
A candidate identification part that identifies a partial image that is a pedestrian or a motorcycle in the image,
With respect to the partial image, a representative specifying portion for specifying a representative value of a distance for each of a plurality of vertical regions arranged in the horizontal direction of the image,
An external environment recognition device that functions as a determination unit for determining whether a partial image is a pedestrian or a two-wheeled vehicle based on variations in representative values in a plurality of vertical regions. The vehicle exterior environment recognition device according to claim 1, wherein the determination unit determines that the partial image is the motorcycle when the index value of the variation of the representative value exceeds a predetermined threshold value. The computer
The vehicle exterior environment recognition device according to claim 1 or 2, which further functions as a correction unit that further reduces the estimated distance to the two-wheeled vehicle from before it is determined to be the two-wheeled vehicle.

Images