JP2021128671A - Three-dimensional object detection apparatus, in-vehicle system, and three-dimensional object detection method - Google Patents

Abstract

To improve accuracy in detecting a three-dimensional object.SOLUTION: A three-dimensional object detection apparatus generates a mask image 90 for masking a difference image G between a first perspective image F1 and a second perspective image F2 captured in a same imaging position O, excluding a three-dimensional object candidate area, specifies a near ground line L1 of an object on the basis of a mask difference image obtained by masking the difference image G with the mask image 90, determines an end point of the three-dimensional object on the basis of the mask difference image Gm, specifies a width of the object on the basis of a distance between a non-masking area boundary N in the mask image 90 and the end point V of the three-dimensional object, specifies a far ground line L2 of the three-dimensional object on the basis of the width of the object and the near ground line L1, and identifies a location of the three-dimensional object in the difference image G on the basis of the near ground line L1 and the far ground line L2.SELECTED DRAWING: Figure 2

Description

The present invention relates to a three-dimensional object detection device, an in-vehicle system, and a three-dimensional object detection method.

A technique for detecting a three-dimensional object such as another vehicle around a vehicle based on the difference between bird's-eye view images (also called bird's-eye view images) at different timings is known (see, for example, Patent Documents 1 and 2). As in Patent Document 1, such a technique is applied to a parking support system that detects a three-dimensional object such as another vehicle in the vicinity as an obstacle and issues an alarm when the vehicle is parked.

Japanese Unexamined Patent Publication No. 2008-227646 International Publication No. 2014/017521

Shadows of structures such as buildings, signs, and traffic lights often appear on the road surface, and these shadows are observed by relative movement from moving vehicles. When such a shadow exists between the moving vehicle and the three-dimensional object around it, there is a problem that the detection accuracy of the three-dimensional object deteriorates.

An object of the present invention is to provide a three-dimensional object detection device, an in-vehicle system, and a three-dimensional object detection method capable of improving the detection accuracy of a three-dimensional object existing around a moving vehicle.

The present invention includes a bird's-eye view conversion processing unit that converts, respectively, a first bird's-eye view image and a second bird's-eye view image taken by a camera at different timings while the vehicle is running, and a bird's-eye view conversion processing unit. Masks other than the difference image generation unit that generates the difference image of the first bird's-eye view image and the second bird's-eye view image in which the shooting positions are aligned, and the candidate three-dimensional object candidate region in which the three-dimensional object is reflected in the difference image. A mask difference image generation unit that generates a mask image and masks the difference image with the mask image to generate a mask difference image, and a near ground line of a three-dimensional object in the difference image are specified based on the mask difference image. The end point of the three-dimensional object is obtained based on the near ground line specific portion and the mask difference image, and the distance between the non-masking area boundary, which is the boundary of the non-masking area in the mask image, and the end point of the three-dimensional object is set. Based on the width specifying portion that specifies the width of the three-dimensional object, the width of the three-dimensional object, and the distant ground line that specifies the distant ground line of the three-dimensional object in the difference image based on the width of the three-dimensional object and the nearby ground line. Provided is a three-dimensional object detecting device including a unit, a position specifying unit for specifying a position of a three-dimensional object in the difference image based on the near ground line and the far ground line.

According to the present invention, it is possible to improve the detection accuracy of three-dimensional objects existing around a moving vehicle.

It is a figure which shows the structure of the in-vehicle system which concerns on embodiment of this invention. It is a figure which shows the functional structure of a camera ECU. It is a figure which shows the positional relationship of the vehicle and another vehicle in the same embodiment. It is a flowchart of a three-dimensional object detection process. It is a figure for demonstrating the difference image generation operation, (A) shows an example of a 1st bird's-eye view image and 2nd bird's-eye view image, and (B) shows an example of a difference image. It is a flowchart of a mask image generation process. It is a figure for demonstrating the collapse of a vertical contour line, (A) shows an example of a photographed image, and (B) shows an example of a bird's-eye view image. It is a figure which shows the label image schematically. It is a figure which shows typically the lookup table. It is a figure for demonstrating the generation of a difference histogram, (A) shows an example of a difference image, and (B) shows an example of a difference histogram. It is a figure for demonstrating the generation of an edge intensity histogram, (A) shows an example of an edge image, (B) shows an example of an edge intensity histogram Rb. It is a figure for demonstrating the generation of a mask image, (A) shows an example of a label image, and (B) shows an example of a mask image. It is explanatory drawing of the generation operation of a mask difference image. It is a flowchart of the neighborhood grounding line identification processing. It is a figure for demonstrating the generation of a mask difference histogram, (A) shows an example of a mask difference image, (B) shows an example of a mask difference histogram obtained from the mask difference image of (A). It is explanatory drawing of the neighborhood grounding line specific operation. It is a flowchart of a vehicle width identification process. It is explanatory drawing of various parameters set in a mask image in a vehicle width specifying process. It is explanatory drawing of the identification operation of the end point of another vehicle based on the difference mask image, (A) shows an example of the difference mask image, (B) shows an example of the difference mask image for end point identification, (C) is an end point. An example of the specific difference histogram is shown, and (D) shows an example of the end points of other vehicles in the difference mask image. It is a figure which shows the example of the camera image, the bird's-eye view image, and the mask image about a plurality of other vehicles having different vehicle widths. It is explanatory drawing of the vehicle width specific condition. It is a figure which shows the relationship between the near grounding line, the distant grounding line, and the vertical contour line in the bird's-eye view image. It is a schematic explanatory drawing of the three-dimensional object area identification processing. It is a flowchart of a three-dimensional object area identification process. It is a figure for demonstrating the generation of the difference neighborhood histogram, (A) shows an example of a difference image, (B) shows an example of a difference neighborhood histogram. It is a figure for demonstrating the generation of the edge intensity neighborhood histogram, (A) shows an example of an edge image, (B) shows an example of an edge intensity neighborhood histogram Rbn. It is a figure for demonstrating the generation of the difference distance histogram, (A) shows an example of a difference image, (B) shows an example of a difference distance histogram. It is a figure for demonstrating the generation of the edge intensity distant histogram, (A) shows an example of an edge image, and (B) shows an example of an edge intensity distant histogram. It is explanatory drawing of the mask label image for the vicinity and the mask label image for the distance. It is explanatory drawing of the operation of specifying the other vehicle area when the three-dimensional object is detected only in the vicinity area. It is explanatory drawing of the grouping operation of each intersection in the neighborhood grounding line. It is explanatory drawing of the determination operation of the final single other vehicle area. It is explanatory drawing of the vehicle width specifying process which concerns on the modification of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram showing a configuration of an in-vehicle system 1 according to the present embodiment.
The in-vehicle system 1 is a system mounted on the vehicle 2, and includes a photographing unit 3, a camera ECU 6, a vehicle control unit 8, an HMI unit 9, and a CAN 10. The CAN 10 is a kind of in-vehicle network that connects the camera ECU 6, the vehicle control unit 8, and the HMI unit 9.

The photographing unit 3 outputs the camera image 5 obtained by photographing the entire circumference (range of 360 degrees) of the vehicle 2 to the camera ECU 6. The shooting unit 3 includes a front camera that shoots the front of the vehicle 2, a rear camera that shoots the rear DB (FIG. 3), a right side camera that shoots the right side, and a left side camera that shoots the left side. A plurality of cameras 4 are provided, and a camera image 5 is output from each camera 4. In the present embodiment, it is assumed that the vehicle 2 is traveling forward, and the front of the vehicle 2 and the traveling direction B coincide with each other.

The camera ECU 6 is a device having a function of controlling the shooting operation of the shooting unit 3 and a function of performing appropriate signal processing on each camera image 5. The camera ECU 6 of the present embodiment detects the position of the other vehicle A (FIG. 3), which is a three-dimensional object running in parallel with the vehicle 2, for each camera image 5 of each camera 4, and transmits the position information of the other vehicle A through the CAN 10. It also functions as another vehicle detection device (three-dimensional object detection device) that transmits to the vehicle control unit 8. Details of the camera ECU 6 will be described later.

The vehicle control unit 8 is a unit including a processor that executes various controls related to the running of the vehicle 2, and has a function of controlling each part of the vehicle 2 such as a steering mechanism and a drive mechanism for this control. Further, the vehicle control unit 8 includes one or a plurality of sensors that detect various vehicle information (at least traveling speed) required for these controls.

Further, the vehicle control unit 8 has a function of driving and controlling the vehicle 2 instead of the driver's driving operation (so-called automatic driving control function), and when the vehicle 2 is traveling, the position of a three-dimensional object existing around the vehicle 2. Are sequentially acquired, and the operation is controlled so as to secure an appropriate distance to the three-dimensional object. For example, when the vehicle 2 can approach another vehicle A, which is an example of a three-dimensional object, such as when changing lanes, merging, or diverging, the vehicle control unit 8 acquires the position of the other vehicle A and said that. Based on the position of the other vehicle A, the operation is controlled so as to secure an appropriate inter-vehicle distance from the other vehicle A.

The vehicle control unit 8 may include a driving support function that supports the driving operation of the driver instead of the automatic driving control function or in combination with the automatic driving control function. The driving support function sequentially acquires the positions of three-dimensional objects existing around the vehicle 2 when the vehicle 2 is running, guides the three-dimensional objects to the driver, and issues various warnings based on the positions of the three-dimensional objects. It supports the driver's driving operation by notifying the driver.

The HMI unit 9 includes an input device and an output device that serve as a user interface, and the output device includes a display device 9A that displays various information and a speaker that outputs various sounds. The display device 9A displays a camera image 5 of each camera 4, an image of a bird's-eye view of the vehicle 2 from above, and information about the other vehicle A detected by the camera ECU 6 (position and size of the other vehicle A). , Etc.) are also displayed in an appropriate manner.

The camera ECU 6 described above includes a CPU 12 as an example of a processor, a memory 14 such as a ROM or RAM for storing various information such as a program 13, and a CANI / F16 as a communication circuit module that communicates with a vehicle control unit 8 through a CAN 10. It is equipped with a so-called computer equipped with. The camera ECU 6 of the present embodiment functions as the other vehicle detection device described above by executing the program stored in the memory 14 by the CPU 12.

FIG. 2 is a diagram showing a functional configuration of the camera ECU 6.
The camera ECU 6 of the present embodiment acquires the first shot image and the second shot image shot at different timings from the camera image 5 shot by the same camera 4, and these first shot image and second shot image. The position of the other vehicle A reflected in the camera image 5 of the camera 4 is detected based on the difference image G which is the difference between the first bird's-eye view image F1 and the second bird's-eye view image F2 obtained by converting the bird's-eye view. The camera ECU 6 detects the position of the other vehicle A with respect to the entire circumference of the vehicle 2 by performing such detection for each camera image 5 of each camera 4.
For such a detection operation, the camera ECU 6 includes a vehicle information acquisition unit 20, a preprocessing unit 22, and a three-dimensional object position specifying unit 24, as shown in FIG.

The vehicle information acquisition unit 20 acquires vehicle information from the vehicle control unit 8. This vehicle information includes at least the traveling speed of the vehicle 2.

The pre-processing unit 22 executes processing for obtaining the difference image G from the camera image 5, and includes the camera image acquisition unit 30, the brightness conversion processing unit 32, the bird's-eye view conversion processing unit 34, and the difference image generation unit. 36 and.

By controlling the shooting unit 3, the camera image acquisition unit 30 continues shooting by the shooting unit 3 for a predetermined time or longer, and acquires the camera image 5 obtained by the shooting.
The luminance conversion processing unit 32 converts each frame (captured image) constituting the camera image 5 into a luminance image. Each frame corresponds to a captured image which is a still image, and the luminance image is an image obtained by converting each pixel value of the captured image into the luminance value of the pixel.
The bird's-eye view conversion processing unit 34 converts each luminance image into a bird's-eye view image. The bird's-eye view image is an image viewed directly below from a virtual viewpoint set above the vehicle 2. The bird's-eye view conversion processing unit 34 generates a bird's-eye view image by projecting a luminance image (also called a viewpoint conversion).
The difference image generation unit 36 includes two first shot images having different shooting timings and a first bird's-eye view image F1 obtained from the second shot images from the bird's-eye view images sequentially generated by the bird's-eye view conversion processing unit 34. And the second bird's-eye view image F2 (see FIG. 5A) are extracted to generate a difference image G between the two. The difference image G converts the pixel value (brightness value) of each pixel of the first captured image E1 into the difference from the pixel value (brightness value) of the pixel of the second captured image E2 corresponding to the pixel, and determines a predetermined value. It is an image in which the pixel value of each pixel is binarized by a threshold value. At the time of generating the difference image G, the difference image generation unit 36 aligns the shooting positions O of the first bird's-eye view image F1 and the second bird's-eye view image F2 with the first bird's-eye view image F1 as a reference, and the pixels of each pixel of both. The difference image G is generated by calculating the difference amount of the values.

The three-dimensional object position specifying unit 24 specifies the position of another vehicle A, which is an example of a three-dimensional object, based on the difference image G. A width specifying unit 52, a distant ground line specifying unit 53, and a position specifying unit 54 are provided.

The mask difference image generation unit 50 generates a mask image 90 that masks the remaining area excluding the other vehicle candidate area 60 in the difference image G, and masks the difference image G with the mask image 90 to obtain another vehicle candidate. A mask difference image Gm (see FIG. 13) in which the areas other than the region 60 are masked is generated.
The other vehicle candidate region 60 is a region with a high probability that the other vehicle A is reflected in the difference image G, and is a candidate region of the other vehicle region H. The other vehicle area H is an area (three-dimensional object area) in which the other vehicle A is determined to be reflected in the difference image G.

The neighborhood grounding line specifying unit 51 identifies the neighborhood grounding line L1 in the difference image G based on the mask difference image Gm.
Here, the ground contact line refers to a line in which another vehicle A existing in another lane adjacent to the lane in which the vehicle 2 is traveling is in contact with the ground, and contour lines 63 on both the left and right sides of the other vehicle A in a plan view (FIG. Corresponds to 3).
In the present embodiment, as shown in FIG. 3, the line of the contour lines 63 on both the left and right sides of the other vehicle A, which is closer to the vehicle 2, is referred to as the near ground line L1, and the line far from the vehicle 2 is the far ground. It is called line L2.
By setting the near ground line L1 and the far ground line L2 in the difference image G, the position of the other vehicle region H in the lateral direction Ch with respect to the vehicle 2 is specified in the difference image G. .. The lateral direction Ch refers to a direction perpendicular to the traveling direction B of the vehicle 2.

The vehicle width specifying unit 52 specifies the vehicle width Vw (FIG. 3) of the other vehicle A based on the mask image 90 (FIG. 13).
The distant ground line specifying unit 53 identifies the distant ground line L2 in the difference image G based on the near ground line L1 and the vehicle width Vw.

The position specifying unit 54 identifies the single other vehicle area K in the difference image G based on the near ground line L1, the far ground line L2, the front end VF of the other vehicle A, and the rear end VB (FIG. 3). The single other vehicle area K is an area occupied (displayed) by one (single) other vehicle A in the difference image G.
More specifically, when a plurality of other vehicles A are traveling in a column, one region including the plurality of other vehicles A may be specified as the other vehicle region H. The single other vehicle area K corresponds to an area in which the other vehicle area H is divided for each other vehicle A, and the position specifying unit 54 finally replaces the other vehicle area H with the single other vehicle area K. To specify.
Further, the position specifying unit 54 identifies the position of the other vehicle A in the real space based on the position of the single other vehicle area K in the difference image G, and sequentially transmits the position to the vehicle control unit 8. A known or well-known appropriate method can be used for the conversion from the position of the single other vehicle region K to the position of the other vehicle A in the real space.

Further, the position specifying unit 54 transmits data for displaying the position information of the other vehicle A in the real space to the HMI unit 9, and causes the display device 9A to display the information. As such a display mode, for example, there may be a mode in which the contour line of the single other vehicle region K is superimposed on the image of the vehicle 2 viewed from above. In the present embodiment, the vehicle width Vw of the other vehicle A is not a preset default value, but an actually measured value specified by the vehicle width specifying unit 52 based on the mask image 90. The position (the area occupied by the other vehicle A) can be displayed more accurately.

Next, the detection operation of the other vehicle A by the camera ECU 6 will be described.
In this operation description, a case where the camera ECU 6 detects another vehicle A reflected in the camera image 5 of the rear camera will be illustrated. Further, in this operation description, as shown in FIG. 3, other lanes 70R and 70L are adjacent to each other on both sides of the lane 70 in which the vehicle 2 is traveling, and the other vehicle A is a vehicle in each lane 70R and 70L. It is assumed that the vehicle is running in the same direction as 2 (that is, running in parallel). Further, it is assumed that all the other vehicles A are located in the rear DB behind the vehicle 2 and are located within the range of the angle of view α of the rear camera. In FIG. 3, the area shown by the rectangular line indicates the detection area 72 that detects the presence of the other vehicle A in the automatic driving control and the driving support by the vehicle control unit 8.

FIG. 4 is a flowchart of the three-dimensional object detection process.
The three-dimensional object detection process is continuously and repeatedly executed by the camera ECU 6 for each camera image 5 of each camera 4 in order to detect the presence of another vehicle A in the vicinity while the vehicle 2 is traveling.

In the three-dimensional object detection process, first, the vehicle information acquisition unit 20 acquires vehicle information (including at least the traveling speed) (step Sa1), and the camera image acquisition unit 30 acquires the camera image 5 (step Sa2).
Next, the luminance conversion processing unit 32 sequentially converts each frame (captured image) of the camera image 5 into a luminance image (step Sa3), and the bird's-eye view conversion processing unit 34 sequentially converts the bird's-eye view image F based on each luminance image. Generate (step Sa4).

Next, the difference image generation unit 36 receives two first shot images having different shooting timings and a first bird's-eye view obtained from the second shot images from the bird's-eye view images sequentially generated by the bird's-eye view conversion processing unit 34. The image F1 and the second bird's-eye view image F2 are extracted, and the difference image G between the two is generated (step Sa5).

5A and 5B are diagrams for explaining a difference image generation operation, FIG. 5A shows an example of a first bird's-eye view image F1 and a second bird's-eye view image F2, and FIG. 5B is an example of a difference image G. Is shown.
In the following, the most recently captured image will be referred to as the first captured image, and the captured image captured before that will be referred to as the second captured image.

Since the respective shooting positions O shift in the traveling direction B in the first shot image and the second shot image, they are also shot in the first bird's-eye view image F1 and the second bird's-eye view image F2 as shown in FIG. 5 (A). At the position O, a shift DE due to the movement of the vehicle 2 occurs. In order to correct the deviation DE of the shooting position O and generate the difference image G, the difference image generation unit 36 sets the shooting position O of both the first bird's-eye view image F1 and the second bird's-eye view image F2 (FIG. In 5 (A), the difference image G between the two is generated in a state where the first bird's-eye view image F1) is aligned with the reference.

Specifically, the difference image generation unit 36 calculates the mileage of the vehicle 2 based on the traveling speed of the vehicle 2 and the time difference ta of the imaging timings of the first captured image and the second captured image. Then, the difference image generation unit 36 travels on each pixel of either the first bird's-eye view image F1 or the second bird's-eye view image F2 (in FIG. 5A, the second bird's-eye view image F2 having the earlier shooting timing). It is shifted along the traveling direction B by the number of pixels according to the distance. As a result, the shooting positions O of both are aligned with respect to the other of the first bird's-eye view image F1 and the second bird's-eye view image F2.

When an arbitrary stationary object such as a white line 74, which is a kind of road marking, is reflected in each of the first bird's-eye view image F1 and the second bird's-eye view image F2, the shooting positions O of both are aligned, so that FIG. 5 (A) ), The positions where the stationary objects are reflected are aligned.
On the other hand, when another vehicle A, which is a moving object, is reflected in each of the first bird's-eye view image F1 and the second bird's-eye view image F2, the shooting positions O of both are aligned, which is shown in FIG. 5 (A). As described above, the positions of the other vehicles A reflected in each of them are displaced.
As a result, in the difference image G showing the difference between the two, as shown in FIG. 5 (B), the pixel value (difference amount) of the region where the stationary object of the road marking (white line 74, etc.) is reflected becomes small. The pixel value (difference amount) in the area where the other vehicle A is reflected is relatively large. Therefore, by appropriately setting the threshold value of the brightness value used for binarizing the difference image G, the image excluding the stationary object can be obtained as the difference image G. Then, in the difference image G, the other vehicle region H in which the other vehicle A is reflected can be specified based on the region of the high luminance value.

However, as shown in FIG. 3 above, when a shadow 76 is generated between the vehicle 2 and the other vehicle A by an arbitrary moving object such as the vehicle 2 or the other vehicle A, the shadow 76 is generated in the difference image G. The pixel value in the region corresponding to 76 also increases. Therefore, when the position specifying unit 54 simply extracts the region having a large pixel value in the difference image G as the other vehicle region H, the region corresponding to the shadow 76 is included in the other vehicle region H, and the other vehicle region H The accuracy of

Therefore, in the present embodiment, before the position specifying unit 54 identifies the other vehicle area H based on the difference image G, as shown in FIG. 4 above, the mask difference image generation unit 50 first sets the difference image G in the difference image G. A mask difference image Gm that masks the remaining area excluding the other vehicle candidate area 60, that is, the area corresponding to noise such as the shadow 76 is generated (step Sa6: mask difference image generation process). Then, the neighborhood grounding line specifying unit 51 identifies the neighborhood grounding line L1 which is one of the lines dividing the other vehicle region H based on the mask difference image Gm (step Sa7: neighborhood grounding line specifying process). .. Thereby, the near ground line L1 can be specified with high accuracy.

FIG. 6 is a flowchart of the above-mentioned mask image generation process.
In the mask image generation process, first, the mask difference image generation unit 50 generates the mask image 90 (step Sb1). The mask image 90 is an image that masks the masking region 62 in the difference image G. The masking area 62 is a residual area excluding the other vehicle candidate area 60 in the difference image G. The mask difference image generation unit 50 identifies the other vehicle candidate region 60 based on the vertical contour line P of the other vehicle A shown in the difference image G.

7A and 7B are views for explaining the collapse of the vertical contour line P. FIG. 7A shows an example of a captured image M, and FIG. 7B shows an example of a bird's-eye view image F.
As shown in FIG. 7A, the vertical contour line P includes the contour line 63 of the other vehicle A shown in the photographed image M (frame of the camera image 5) and the vehicle body parts (doors, etc.) of the other vehicle A. ), The outline of the pattern drawn on the other vehicle A, and the like, which extends in the vertical direction (perpendicular to the ground) in the real space. Along with the projective transformation (viewpoint transformation) of the captured image M, the vertical contour line P of the captured image M is the camera 4 (rear camera in this operation description) in the bird's-eye view image F as shown in FIG. 7 (B). It is converted into radiation Q extending radially from the imaging position O (so-called collapse of the vertical contour line P). That is, in the bird's-eye view image F, the region defined by the radiation Q including the vertical contour line P in the line segment (the region shown by the hatching in FIG. 7B) indicates the region where the other vehicle A exists. The area becomes another vehicle candidate area 60. The direction of the radiation Q including the vertical contour line P in the line segment is also called the direction in which the three-dimensional object collapses due to the projective transformation (viewpoint transformation).

In the difference image G, which is the difference between the two bird's-eye view images F, the vertical contour line P is a line segment of the radiation Q as in the bird's-eye view image F. In the difference image G, each pixel constituting the radiation Q including the vertical contour line P has a pixel value (difference amount) larger than that of the other pixels. Based on this, the mask difference image generation unit 50 identifies another vehicle candidate region 60 based on each pixel value of the difference image G. Further, the mask difference image generation unit 50 efficiently identifies the other vehicle candidate area 60 by using the label image 91 and the look-up table 92.

FIG. 8 is a diagram schematically showing the label image 91.
The label image 91 is an image of a plurality of radiation Qs extending radially from the photographing position O at equal intervals and each of which is identified by a label number, and each radiation Q includes a vertical contour line P in the difference image G. However, it is a candidate for radiation Q. In the present embodiment, the label image 91 includes 100 radiation Qs whose label numbers are "1" to "100".
The label image 91 has a number of pixels corresponding to the difference image G, and as shown in FIG. 8, each pixel constituting the same radiation Q has a label number (“1” to “100”) of the radiation Q. Either) is associated.

FIG. 9 is a diagram schematically showing a look-up table 92.
The look-up table 92 specifies a pixel value of either "255" corresponding to non-masking (white) or "0" corresponding to masking (black) for each pixel of the label image 91. By setting the pixel value of each pixel of the label image 91 based on the designation of the lookup table 92, a mask image 90 in which each pixel is in the non-masking state (white) or the masking state (black) is obtained. Be done.

As shown in FIG. 9, in the lookup table 92, the pixel value of the radiation Q is specified for each label number of the radiation Q. This pixel value is determined based on the pixel value for each radiation Q in the difference image G.

In the mask image generation process (FIG. 6: step Sb1), the mask difference image generation unit 50 first sets all the pixel values of the label image 91 to "255" (non-masking state) or "0" (masking state). ) To initialize (step Sb1A).
Next, the mask difference image generation unit 50 creates the look-up table 92 based on the pixel values of each radiation Q in the difference image G (step Sb1B). Specifically, the mask difference image generation unit 50 uses a flag (“0” or “255”) indicating the brightness value of each pixel for each radiation Q in the lookup table 92 based on the difference histogram Ra and the edge intensity histogram Rb. ") Is decided.

10A and 10B are diagrams for explaining the generation of the difference histogram Ra, FIG. 10A shows an example of the difference image G, and FIG. 10B is from the difference image G shown in FIG. 10A. An example of the obtained difference histogram Ra is shown.
As shown in FIG. 10B, the difference histogram Ra has the label number as the horizontal axis, and the difference image G shown in FIG. 10A is a value obtained by accumulating the presence or absence of pixel values for each radiation Q (hereinafter, “radiation”). It is a graph with the vertical axis as the "cumulative value of the amount of difference in direction"). Since the cumulative value of the radiation direction difference increases when the radiation Q includes the vertical contour line P, in this difference histogram Ra, each radiation whose radiation direction difference cumulative value exceeds a predetermined first threshold Th1. By specifying Q, it is possible to specify the range Ua of the radiation Q that is the other vehicle candidate region 60.
Further, by specifying the radiation Q including the vertical contour line P based on the cumulative value of the radiation direction difference amount for each radiation Q, image processing such as contour extraction processing is applied to the difference image G to perform the vertical direction contour line P. Radiation Q can be identified at high speed and with high accuracy as compared with the method of detecting.

11A and 11B are views for explaining the generation of the edge intensity histogram Rb, FIG. 11A shows an example of the edge image E, and FIG. 11B is obtained from the edge image E of FIG. 11A. An example of the edge intensity histogram Rb is shown.
As shown in FIG. 11 (B), the edge intensity histogram Rb is a value obtained by accumulating the presence or absence of pixel values for each radiation Q in the edge image E shown in FIG. 11 (A) with the label number as the horizontal axis (hereinafter, "" It is a graph with the vertical axis as the "cumulative value of edge intensity in the radiation direction").
The edge image E is a bird's-eye view image of the first bird's-eye view image F1 and the second bird's-eye view image F2, whichever has the later shooting timing (that is, the nearest one) (the first bird's-eye view image F1 in the present embodiment). It is an image which extracted the contour component of the object (including the pattern of the object) reflected in. In such an edge image E, the mask difference image generation unit 50 sets the pixel value of each pixel having a large brightness difference (a predetermined value or more) from the surrounding pixels in the bird's-eye view image to a value corresponding to the brightness difference ( It is generated by converting to intensity value). Therefore, the edge intensity histogram Rb is a graph showing the magnitude of the edge component of the three-dimensional object included in the radiation Q for each label of the radiation Q.

When the mask difference image generation unit 50 creates the lookup table 92 in step Sb1B of FIG. 6, the cumulative value of the radiation direction difference amount exceeds a predetermined first threshold value Th1 in the difference histogram Ra, and the edge strength In the histogram Rb, the radiation Q whose cumulative value of the radiation direction edge intensity exceeds a predetermined second threshold value Th2 is specified. Then, the mask difference image generation unit 50 sets the pixel values of the “non-masking state” for these radiation Qs in the lookup table 92, and sets the pixel values of the “masking state” for the other radiation Qs.
Next, the mask difference image generation unit 50 generates the mask image 90 by setting each pixel value of the label image 91 based on the lookup table 92 (step Sb1C).

12A and 12B are views for explaining the generation of the mask image 90, FIG. 12A shows an example of the label image 91, and FIG. 12B shows an example of the mask image 90.
By applying the look-up table 92 to the label image 91 shown in FIG. 12 (A), as shown in FIG. 12 (B), the region corresponding to the other vehicle candidate region 60 is the non-masked region in the non-masked state. A mask image 90 is obtained in which the number is 64 and the area other than the non-masking area 64 is the masking area 62.

Then, in the subsequent step Sb2, the mask difference image generation unit 50 superimposes the mask image 90 on the difference image G, so that the mask difference in which the area other than the other vehicle candidate area 60 is masked is masked as shown in FIG. Generate an image Gm.

Then, when the mask difference image Gm is generated, as described above, the neighborhood grounding line specifying unit 51 identifies the neighborhood grounding line L1 based on the mask difference image Gm (FIG. 4: Step Sa7: neighborhood grounding line identification). process).

FIG. 14 is a flowchart of the near ground line identification process.
The neighborhood ground line identification unit 51 first generates a mask difference histogram Rc in order to obtain the neighborhood ground line L1 (step Sc1).

FIG. 15 is a diagram for explaining the generation of the mask difference histogram Rc, FIG. 15 (A) shows an example of the mask difference image Gm, and FIG. 15 (B) is from the mask difference image Gm of FIG. 15 (A). An example of the obtained mask difference histogram Rc is shown.
As shown in FIG. 15B, the mask difference histogram Rc has the horizontal direction Ch position (hereinafter, referred to as “horizontal position”) perpendicular to the traveling direction B of the vehicle 2 as the horizontal axis, and is shown in FIG. The lateral direction of the mask difference image Gm shown in (A) is divided into strip-shaped small areas at predetermined intervals, and the presence or absence of pixel values is accumulated along the traveling direction B for each area (hereinafter, "traveling direction difference amount"). It is a graph with the vertical axis of "cumulative value"). In the mask difference image Gm, since the area other than the other vehicle candidate region 60 is masked, the near ground line L1 of the other vehicle A can be specified by the distribution of the cumulative value of the difference amount in the traveling direction in the lateral direction Ch.

Specifically, as shown in FIG. 14 above, the proximity grounding line specifying unit 51 sets a third threshold value Th3 of the cumulative value of the difference amount in the traveling direction in which the other vehicle A is considered to exist at the lateral position thereof (). Step Sc2). An intermediate value (= (Ave + Min) / 2) between the average value Ave of the cumulative value of the difference amount in the traveling direction and the minimum value Min of the accumulated value of the difference amount in the traveling direction is set in the third threshold value Th3.

Next, in the mask difference histogram Rc, the near-grounding line specifying unit 51 is based on the range Uc of the lateral position in which the cumulative value of the difference amount in the traveling direction continuously exceeds the third threshold value Th3 for a predetermined number or more. The nearby ground line L1 is specified.
Specifically, as shown in FIG. 16, the neighborhood grounding line specifying unit 51 sets the determination points X at equal intervals at i (i is an integer of 1 or more) on the horizontal axis of the mask difference histogram Rc. Each determination point X may correspond to an interval (column of the graph) on the horizontal axis of the mask difference histogram Rc.
Then, as shown in FIG. 14 above, the neighborhood grounding line specifying unit 51 determines in order from the determination point X closest to the shooting position O whether or not the predetermined neighborhood grounding line determination condition is satisfied (step Sc3). If it is not satisfied (step Sc3: No), the next determination point X is determined (step Sc4). Further, when the neighborhood grounding line determination condition is satisfied (step Sc3: Yes), the neighborhood grounding line specifying unit 51 identifies the determination point X as the position of the neighborhood grounding line L1 (step Sc5).

In the above-mentioned neighborhood grounding line determination condition, the cumulative value of the traveling direction difference amount of the determination point X is the third threshold value Th3 or less, and the traveling direction difference amount is accumulated at all of the predetermined number of determination points X from the next determination point X. The condition is that the value is equal to or greater than the third threshold value Th3.
By determining the neighboring ground line determination conditions in order from the determination point X closest to the shooting position O, as shown in FIG. 16, the cumulative value of all the traveling direction difference amounts of the predetermined number of determination points X is the third threshold value Th3. For the range Uc exceeding the above range, the determination point X immediately before the shooting position O is obtained, and this determination point X is specified as the near ground line L1. As a result, the near ground line L1 is not set at the position where the other vehicle A has entered (the range exceeding the third threshold value Th3), and the near ground line L1 is set at a more accurate position.

As described above, the position of the near ground line L1 in the difference image G is specified not based on the difference image G but based on the mask difference image Gm in which noise such as the shadow 76 is masked, so that the specified position is very high. Will be accurate.

When the nearby ground line L1 is specified, the vehicle width specifying unit 52 executes the vehicle width specifying process as shown in FIG. 4 above (step Sa8). In this vehicle width specifying process, the vehicle width specifying unit 52 specifies the vehicle width Vw of the other vehicle A based on the mask image 90.

FIG. 17 is a flowchart of the vehicle width specifying process, and FIG. 18 is an explanatory diagram of various parameters set on the mask image 90 in the vehicle width specifying process.
In the vehicle width specifying process, the vehicle width specifying unit 52 first determines whether or not the position of the other vehicle A is within the vehicle width specifying condition range (step Se1). The vehicle width specifying condition is a condition indicating whether or not the vehicle width Vw of the other vehicle A shown in the difference image G can be specified from the mask image 90. The vehicle width specifying conditions will be described later.

When the position of the other vehicle A is not within the vehicle width specifying condition range (step Se1: No), the vehicle width specifying unit 52 ends the process as it is, and when the position of the other vehicle A is within the vehicle width specifying condition range (step Se1: No). Step Se1: Yes), the following process is executed in order to specify the vehicle width Vw of the other vehicle A.

That is, the vehicle width specifying unit 52 specifies the position of the end point V of the other vehicle A in the mask image 90 (step Se2). The end point V of the other vehicle A is the front end VF or the rear end VB of the other vehicle A shown in the camera image 5. In this operation example, since the other vehicle A is reflected in the camera image 5 of the rear camera, the tip VF of the other vehicle A is reflected in the camera image 5 and the difference image G obtained from the camera image 5. Therefore, in step Se2, as shown in FIG. 18, the position of the tip VF is specified as the end point V of the other vehicle A.

In the present embodiment, the vehicle width specifying unit 52 specifies the position of the end point V of the other vehicle A in the mask image 90 based on the mask difference image Gm.
Specifically, as shown in FIG. 19A, the vehicle width specifying unit 52 superimposes the near ground line L1 on the mask difference image Gm, and in the mask difference image Gm, the vehicle 2 is more than the near ground line L1. By further masking the region on the side close to the photographing position O, the mask difference image Gmt for specifying the end point shown in FIG. 19B is generated.
Next, as shown in FIG. 19C, the vehicle width specifying unit 52 generates an end point specifying difference histogram Rgmt from the end point specifying mask difference image Gmt. The end point identification difference histogram Rgmt divides the vertical direction of the end point identification mask difference image Gmt (in the present embodiment, the direction of the rear DB of the vehicle 2) into strip-shaped small areas at predetermined intervals, and specifies the end points for each area. This is a graph in which the presence / absence of pixel values is accumulated in the lateral direction Ch of the mask difference image Gmt. Hereinafter, the cumulative value of this pixel value is referred to as the cumulative value of the lateral difference amount.

In the end point identification mask difference image Gmt, in addition to the area other than the other vehicle candidate area 60, the area closer to the shooting position O of the vehicle 2 than the nearby ground line L1 is masked, so that the cumulative value of the lateral difference amount is accumulated. From the distribution of, the position of the end of the other vehicle A on the side closer to the photographing position O of the vehicle 2 (in the present embodiment, the tip VF of the other vehicle A) can be specified.

Specifically, the vehicle width specifying unit 52 is similar to the method in which the neighborhood grounding line specifying unit 51 identifies the neighborhood grounding line L1 based on the mask difference histogram Rc, and the vehicle width specifying unit 52 is another vehicle based on the difference histogram Rgmt for specifying the end point. Identify the near-end of A.
That is, the vehicle width specifying unit 52 scans each section of the end point specifying difference histogram Rgmt in order from the determination point X closest to the shooting position O. Then, as shown in FIG. 19B, when the cumulative value of the lateral difference amount of each section continuously exceeds the end point determination threshold value Thgmt for a predetermined number of times in the end point identification difference histogram Rgmt, the vehicle As shown in FIG. 19C, the width specifying unit 52 specifies the first section of the continuous section (that is, the section of the continuous section closest to the shooting position O) Kgmt. Then, in the mask difference image Gm, the vehicle width specifying unit 52 superimposes the end line Lgmt extending in the lateral direction Ch through the position corresponding to the section Pgmt on the mask difference image Gm. This end line Lgmt indicates the position of the end portion (tip VF) of the other vehicle A in the mask difference image Gm. Therefore, the vehicle width specifying unit 52 specifies the end point V in the mask difference image Gm by obtaining the intersection of the end line Lgmt and the nearby ground line L1.
Then, the vehicle width specifying unit 52 specifies in the mask image 90 the position corresponding to the position of the end point V of the other vehicle A in the mask difference image Gm as the position of the end point V in the mask image 90.

Next, as shown in FIG. 18, the vehicle width specifying unit 52 sets a horizontal line Lc extending in the lateral direction Ch through the end point V on the mask image 90 (FIG. 17: step Se3). Then, in the mask image 90, the vehicle width specifying portion 52 scans along the horizontal line Lc from the end point V as the starting point toward the far direction Chf (FIG. 18) away from the vehicle 2, and the horizontal line Lc and the non-masking region boundary N The intersection Vm with is specified (Fig. 17: Step Se4).

Here, the non-masking region boundary N is a boundary between the non-masking region 64 and the masking region 62, and corresponds to the edge of the non-masking region 64. The non-masking area 64 corresponds to an area other than the masking area 62 in the difference image G, that is, another vehicle candidate area 60 which is a candidate in which the other vehicle A is reflected.
Further, the horizontal line Lc indicates a line extending in the vehicle width direction of the other vehicle A shown in the difference image G, and the intersection Vm between the horizontal line Lc and the non-masking region boundary N is the tip VF of the other vehicle A or the rear. The end VB is shown.

Therefore, the end point V specified by step Se2 and the intersection point Vm specified by step Se4 correspond to both ends in the vehicle width direction on the tip side of the other vehicle A. Therefore, the vehicle width specifying unit 52 specifies the vehicle width Vw in the real space by converting the distance between the end point V and the intersection Vm in the mask image 90 into the distance in the real space (step Se5).

FIG. 20 is a diagram showing examples of a camera image 5, a bird's-eye view image F, and a mask image 90 for a plurality of other vehicles A having different vehicle widths Vw. In the figure, as an example of the other vehicle A, an ordinary vehicle (ordinary four-wheeled vehicle), a large bus (large passenger vehicle), and a motorcycle (two-wheeled vehicle) are shown.
As shown in the figure, the vehicle width Vw is appropriately specified for each of the other vehicles A by the vehicle width specifying process.

By the way, in this vehicle width specifying process, the vehicle width Vw is specified based on the end point V on the front end side or the rear end side of the other vehicle A in the mask image 90, so that the front end side or the rear end side is the bird's-eye view image F. And, if it is not reflected in the camera image 5, the vehicle width Vw cannot be specified. In this case, the execution of the vehicle width specifying process is useless.
Therefore, in the present embodiment, the condition that the front end side or the rear end side of the other vehicle A is reflected in the camera image 5 is set in advance in the vehicle width specifying condition, and the vehicle is set in step Sa1 at the start of the vehicle width specifying process. By determining the satisfaction of the width specific condition, unnecessary execution of processing is prevented.

The condition that the front end side or the rear end side of the other vehicle A is reflected in the camera image 5 can be defined by the range in which the other vehicle A is located within the entire 360-degree circumference centered on the vehicle 2. For example, as shown in FIG. 21, the other vehicle A is substantially lateral to the vehicle 2 so that the front end side or the rear end side of the other vehicle A is not reflected in the camera image 5 (overhead image F in the example of FIG. 21). The predetermined range β of is set, and the remaining range excluding this predetermined range β is set in the vehicle width specifying condition.
Then, in step Se1, the vehicle width specifying unit 52 determines whether or not the other vehicle A is located in the remaining range excluding the predetermined range β, and only when the other vehicle A is located within the range, the step The process after Se2 is executed. As a result, it is possible to prevent the vehicle width specifying process from being unnecessarily performed. The vehicle width specifying unit 52 may specify the position of the other vehicle A with respect to the vehicle 2 based on, for example, the camera image 5 of each camera 4 or the bird's-eye view image F, or another object detection such as sonar. It may be specified by using the detection result of the sensor.

When the vehicle width Vw is specified, as shown in FIG. 4 above, the distant ground line specifying unit 53 specifies the distant ground line L2 in the difference image G (step Sa9). Specifically, the distant ground line specifying unit 53 specifies, in the difference image G, a line parallel to the nearby ground line L1 as the distant ground line L2, separated from the nearby ground line L1 by the vehicle width Vw.
Since the position of the distant ground line L2 is obtained by using the vehicle width Vw obtained from the mask image 90, it can be obtained more accurately than when the vehicle width Vw is set to a fixed value appropriately determined in advance.

Next, the position specifying unit 54 specifies the other vehicle area H in the difference image G (step Sa10: three-dimensional object area specifying process).
The other vehicle area H includes the near ground line L1 and the far ground line L2 specified in each of the near ground line identification process (step Sa7) and the far ground line identification process (step Sa9), and the tip VF of the other vehicle A. , And the trailing edge VB.

As described above, the front end VF and the rear end VB of the other vehicle A can be obtained from the intersection of the near ground line L1 and the far ground line L2 and the vertical contour line P.
That is, as shown in FIG. 22, in the bird's-eye view image F, at the intersections of the vertical contour line P1 near the rear end of the other vehicle A, the vertical contour line P2 near the tip end, and the near ground line L1. The position near the rear end L1VB and the position near the tip L1VF on the near ground line L1 can be obtained. Similarly, the position on the far side of the rear end of the far ground line L2 is determined by the intersection of the vertical contour line P1 near the rear end and the vertical contour line P2 near the tip of the other vehicle A and the distant ground line L2. L2VB and L2VF at the tip far side position can be obtained.
The rear end near side position L1VB and the rear end far side position L2VB indicate both ends in the vehicle width direction on the rear end side of the other vehicle A, and the tip near side position L1VF and the tip far side position L2VF indicate the other vehicle. Both ends in the vehicle width direction on the tip side of A are shown.

However, in the bird's-eye view image F, as described above, since the other vehicle A collapses in the direction of the radiation Q, the other vehicle extends longer in the traveling direction B than it actually is due to the influence of the roof portion Ar of the other vehicle A and the like. The region H will be detected, and an error will occur in the position of the other vehicle A.

In the three-dimensional object area specifying process of the present embodiment, in order to eliminate such an error, the position specifying unit 54 specifies the other vehicle area H as follows.
That is, as shown in FIG. 23, the position specifying unit 54 obtains the intersection LV with the radiation Q including the vertical contour line P of the other vehicle A for each of the near ground line L1 and the far ground line L2. Next, the position specifying unit 54 identifies the provisional first other vehicle region H1 from the intersection LV on the near ground line L1 and identifies the provisional second other vehicle region H2 from the intersection LV on the distant ground line L2. Then, the position specifying unit 54 identifies the front end VF and the rear end VB of the other vehicle A based on the area where the provisional first other vehicle area H1 and the provisional second other vehicle area H2 overlap. Then, an accurate other vehicle region H is specified by the near ground line L1, the far ground line L2, the front end VF, and the rear end VB.
Here, if the accuracy of the distant ground line L2 is poor, the provisional first other vehicle area H1 and the provisional second other vehicle area H2 may not overlap, and the other vehicle area H may not be specified. On the other hand, in the present embodiment, the distant ground line L2 is accurately specified by using the near ground line L1 specified based on the mask difference image Gm and the vehicle width Vw specified based on the mask image 90. Therefore, the other vehicle area H can be reliably specified.

Hereinafter, such a three-dimensional object region identification process will be described in more detail.

FIG. 24 is a flowchart of the three-dimensional object region identification process.
In order to identify the provisional first other vehicle region H1 and the provisional second other vehicle region H2, the position specifying unit 54 first first specifies the difference neighborhood histogram Ran, the edge intensity neighborhood histogram Rbn, the difference distant histogram Raf, and the edge. Intensity distant histogram Rbf and each are generated (step Sd1).

FIG. 25 is a diagram for explaining the generation of the difference neighborhood histogram Ran, FIG. 25 (A) shows an example of the difference image G, and FIG. 25 (B) is obtained from the difference image G of FIG. 25 (A). An example of the difference neighborhood histogram Ran is shown. Further, FIG. 26 is a diagram for explaining the generation of the edge intensity neighborhood histogram Rbn, FIG. 26 (A) shows an example of the edge image E, and FIG. 26 (B) is from the edge image E of FIG. 26 (A). An example of the obtained edge strength neighborhood histogram Rbn is shown.
As shown in FIG. 25 (B), the difference neighborhood histogram Ran is obtained by obtaining the above-mentioned difference histogram Ra for the neighborhood region Jn in the difference image G shown in FIG. 25 (A). Further, as shown in FIG. 26 (B), the edge intensity neighborhood histogram Rbn is obtained by obtaining the above-mentioned edge intensity histogram Rb for the neighborhood region Jn in the edge image E shown in FIG. 26 (A).
The neighborhood region Jn is a region sandwiched between the near ground line L1 and the far ground line L2 in the difference image G.

27 is a diagram for explaining the generation of the difference distance histogram Raf, FIG. 27 (A) shows an example of the difference image G, and FIG. 27 (B) is obtained from the difference image G of FIG. 27 (A). An example of the difference distant histogram Raf is shown. Further, FIG. 28 is a diagram for explaining the generation of the edge intensity distant histogram Rbf, FIG. 28 (A) shows an example of the edge image E, and FIG. 28 (B) is from the edge image E of FIG. 28 (A). An example of the obtained edge intensity distant histogram Rbf is shown.
As shown in FIG. 27 (B), the difference distant histogram Raf is obtained by obtaining the above-mentioned difference histogram Ra for the distant region Jf in the difference image G shown in FIG. 27 (A). Further, as shown in FIG. 28 (B), the edge intensity distant histogram Rbf is obtained by obtaining the above-mentioned edge intensity histogram Rb for the distant region Jf in the edge image E shown in FIG. 28 (A).
The distant region Jf is a region in the difference image G that is farther than the near ground line L1 when viewed from the shooting position O.

The neighborhood mask label image 91n is used to generate the difference neighborhood histogram Ran and the edge intensity neighborhood histogram Rbn, and the distance mask label image 91f is used to generate the difference distance histogram Raf and the edge intensity distance histogram Rbf.

FIG. 29 is an explanatory diagram of the near mask label image 91n and the distant mask label image 91f.
The neighborhood mask label image 91n is an image in which the pixel values other than the neighborhood region Jn are invalidated in the difference image G so as to be excluded from the cumulative value count. As shown in FIG. 29, the neighborhood mask label image 91n is obtained by superimposing the neighborhood mask image 90n that masks areas other than the neighborhood region Jn in the difference image G on the label image 91 described above.
When the position specifying unit 54 obtains the difference histogram Ra and the edge intensity histogram Rb using the neighborhood mask label image 91n, only the pixel values of the neighborhood region Jn are added to the cumulative values, and the difference neighborhood histogram Ran and the difference neighborhood histogram Ran and The edge strength neighborhood histogram Rbn is obtained.

The distant mask label image 91f is an image in which the pixel values other than the distant region Jf are invalidated in the difference image G so as to be excluded from the cumulative value count, and the distant mask label image 91f masks the distant region other than the distant region Jf in the difference image G. It is obtained by superimposing the area mask image 90f on the label image 91.
When the position specifying unit 54 obtains the difference histogram Ra and the edge intensity histogram Rb using the distant mask label image 91f, only the pixel values of the distant region Jf are added to the cumulative values, and the difference distant histogram Raf and the difference distant histogram Raf and The edge strength distant histogram Rbf is obtained.

Returning to FIG. 24 above, the position specifying unit 54 then determines a three-dimensional object in the neighborhood region Jn based on the difference neighborhood histogram Ran and the edge strength neighborhood histogram Rbn (step Sd2).
Specifically, in the position specifying unit 54, the cumulative value of the radiation direction difference amount in the difference neighborhood histogram Ran is equal to or higher than the fourth threshold value Th4 (FIG. 25), and the radiation direction edge intensity cumulative value is the third in the edge intensity neighborhood histogram Rbn. Specify the radiation Q with a label number that is equal to or higher than the 5 threshold Th5 (FIG. 26).
Then, as shown in FIG. 30, the position specifying unit 54 identifies the intersection LV of each of the specified radiations Q and the nearby ground line L1 in the difference image G. These intersection points LV are performed only for a predetermined detection region set in the difference image G. This detection area is, for example, the detection area 72 (FIGS. 3 and 5).
By these intersection points LV, the provisional first other vehicle region H1 when the three-dimensional object is detected only in the vicinity region Jn is specified. The vehicle width Vw specified in the vehicle width specifying process (FIG. 4: step Sa9) is used as the width of the provisional first other vehicle region H1 in the lateral direction Ch.

Returning to FIG. 24 above, the position specifying unit 54 determines a three-dimensional object in the distant region Jf based on the difference distant histogram Raf and the edge intensity distant histogram Rbf (step Sd3). As a result, the provisional second other vehicle region H2 when the three-dimensional object is detected only in the distant region Jf is specified.

Here, as described above, when a plurality of other vehicles A are traveling in tandem, the provisional first other vehicle area H1 and the provisional second other vehicle area H2 may include two or more other vehicles A. There is sex.
Therefore, the position specifying unit 54 groups each intersection LV on the near ground line L1 and each intersection LV on the far ground line L2 for each other vehicle A as follows (step Sd4, step Sd4, Sd5).

Taking the intersection LV as the proximity ground line L1 as an example, as shown in FIG. 31, the position specifying unit 54 searches for each intersection LV on the proximity ground line L1 in order from the shooting position O, and two adjacent intersections. When the distance W between the LVs is equal to or less than the predetermined sixth threshold Th6, the two intersection points LV are classified into the same group 97, and when the distance W exceeds the predetermined sixth threshold Th6, the shooting position O The intersection LV farther from is classified into a new group 97. As a result, the group 97 is divided into places where the distance between the intersection LVs is wider than the sixth threshold Th6, that is, between the two other vehicles A, and the intersection LVs are grouped for each other vehicle A.
Then, the position specifying unit 54 identifies the single other vehicle area K1 by the intersection LV belonging to the group 97 for each group 97, so that the provisional first other vehicle area H1 is set for each single other vehicle A. It will be divided.

Returning to FIG. 24 above, the position specifying unit 54 finally determines the single other vehicle area K1 specified for the near ground line L1 and the single other vehicle area K2 specified for the far ground line L2. Single other vehicle area K is determined (step Sd6).
That is, as shown in FIG. 32, among the front end VF and the rear end VB of each of the other vehicle A of the single other vehicle area K1 and the single other vehicle area K2, the single other vehicle area K1 and the single other vehicle area K1 and the single other vehicle area K1. Those in the range where both of the other vehicle areas K2 overlap are specified as the front end VF and the rear end VB of the final single other vehicle area K.
Then, by using these front end VF and rear end VB, the near ground line L1 and the far ground line L2, the error caused by the collapse due to the projective transformation is eliminated, and the accurate position of the rectangular single other vehicle area K is determined. It will be identified.

According to the above-described embodiment, the following effects are obtained.

The camera ECU 6 (three-dimensional object detection device) of the present embodiment is a candidate when another vehicle A is shown in the difference image G of the first bird's-eye view image F1 and the second bird's-eye view image F2 in which the shooting positions O of each other are aligned. A mask difference image Gm that masks areas other than the other vehicle candidate area 60 is generated, and the position of the other vehicle A in the difference image G is specified based on the mask difference image Gm.
As a result, as compared with the case where the position of the other vehicle A is specified based on the difference image G, even when the shadow 76 exists between the running vehicle 2 and the other vehicle A around it, the other vehicle A The position can be specified accurately.
Therefore, in the vehicle control unit 8, more accurate automatic driving control can be realized in a scene where the vehicle approaches another vehicle at the time of lane change, merging, or divergence, based on the accurate position of the other vehicle A.
In addition to this, the camera ECU 6 obtains the end point V of the other vehicle A based on the mask difference image Gm, and based on the distance between the end point V of the other vehicle A and the non-masking region boundary N in the mask image 90, the other vehicle A vehicle width specifying unit 52 for specifying the vehicle width Vw of A is provided. As a result, the position of the other vehicle A can be accurately specified as compared with the case where the camera ECU 6 uses a fixed value appropriately determined in advance for the vehicle width Vw to specify the position of the other vehicle A.
Further, since the vehicle width specifying unit 52 specifies the vehicle width Vw based on the mask image 90 used to generate the mask difference image Gm, the vehicle width Vw can be efficiently specified using the existing mask image 90.

The vehicle width specifying unit 52 of the present embodiment specifies an intersection Vm between the horizontal line Lc extending in the vehicle width direction of the other vehicle A from the end point V of the other vehicle A and the non-masking region boundary N, and the intersection Vm and the other vehicle. The vehicle width Vw is specified based on the distance from the end point V of A.
As a result, the vehicle width Vw of the other vehicle A can be accurately obtained from the mask image 90.

The vehicle width specifying unit 52 of the present embodiment identifies the intersection Vm by scanning the mask image 90 along the horizontal line Lc extending from the end point V of the other vehicle A in the vehicle width direction of the other vehicle A.
Thereby, the intersection Vm can be specified relatively easily.

The position specifying unit 54 of the present embodiment is at the near ground line L1 based on the intersection LV of the radiation Q including the vertical contour line P of the other vehicle A and each of the near ground line L1 and the far ground line L2. The provisional first other vehicle area H1 in which the other vehicle A is located and the provisional second other vehicle area H2 in which the other vehicle A is located are specified on the distant ground line L2, and these provisional first other vehicle areas H1 and the provisional first 2 The other vehicle area H in the difference image G is specified based on the range in which the other vehicle area H2 overlaps.
Thereby, the influence of the collapse due to the projective transformation can be eliminated, and the other vehicle region H can be accurately specified.
If the accuracy of the distant ground line L2 is poor, the provisional first other vehicle area H1 and the provisional second other vehicle area H2 may not overlap, and the other vehicle area H may not be specified. On the other hand, in the present embodiment, the distant ground line L2 is accurately specified by using the near ground line L1 specified based on the mask difference image Gm and the vehicle width Vw specified based on the mask image 90. Therefore, the other vehicle area H can be reliably specified.

The near ground line specifying unit 51 of the present embodiment is a difference image based on the position on the horizontal axis in which the cumulative value of the difference amount in the traveling direction exceeds the third threshold Th3 in the mask difference histogram Rc generated based on the mask difference image Gm. The near ground line L1 of the other vehicle A in G is specified. As a result, the near ground line L1 is accurately identified using the mask difference image Gm.

The above-described embodiment is merely an example of one aspect of the present invention, and can be arbitrarily modified and applied without departing from the spirit of the present invention.

(Modification example)
The following modifications can be made to the vehicle width specifying process (FIG. 17) described above.
That is, in the mask image 90, the vehicle width specifying unit 52 scans along the horizontal line Lc in the distant direction Chf starting from the end point V, and identifies the intersection Vm between the horizontal line Lc and the non-masking region boundary N (FIG. 17 :). Step Se4).

However, when a part on the side surface of the vehicle body, an outline of a pattern, a shadow of the vehicle 2 or the like is reflected in the camera image 5, the mask image 90 masks the reflected portion as shown in FIG. 33, for example. Alternatively, a plurality of masking regions 62A (one in the illustrated example) are generated. Since the masking area 62A is generated in the non-masking area 64 (corresponding to the other vehicle candidate area 60 of the difference image G) for one vehicle, the non-masking area 64 is divided into a plurality of (two in the illustrated example). .. Therefore, in this case, there are as many intersections Vm1, Vm2, ... As the number of non-masking regions 64 in which the horizontal line Lc starts from the end point V and crosses in the distant direction Chf. In such a case, when the vehicle width Vw1 is obtained based on the intersection Vm1 first detected when the vehicle width specifying unit 52 scans along the horizontal line Lc, the vehicle width Vw1 shorter than the actual vehicle width Vw is obtained. It ends up.

Therefore, in the vehicle width specifying process, when the vehicle width specifying portion 52 scans along the horizontal line Lc in the distant direction Chf starting from the end point V (step Se4), the vehicle width specifying portion 52 uses the edge 90F of the mask image 90. You may scan to and identify all the intersections Vm1, Vm2, ... Existing between the end point V and the edge 90F. In this case, the vehicle width specifying unit 52 identifies a value that is appropriate as the vehicle width Vw of the other vehicle A from the vehicle widths Vw1 and Vw2 obtained from the intersections Vm1 and Vm2, respectively.

Instead of the configuration in which the vehicle width specifying portion 52 specifies all the intersections Vm1, Vm2, ... Existing between the end point V and the edge 90F, the following may be used.
That is, the masking region 62A generated by the contour line included in the vehicle 2 and the shadow of the vehicle 2 tends to be relatively narrow. Therefore, a predetermined width for distinguishing the masking area 62A from the other masking areas 62 is set in advance in the vehicle width specifying portion 52.
Then, each time the intersection Vm is detected during scanning along the horizontal line Lc, the vehicle width specifying unit 52 detects the width 62Aw (FIG. 33) along the horizontal line Lc of the masking region 62A to which the intersection Vm belongs. The width 62Aw is compared with the predetermined width. Based on this comparison result, the vehicle width specifying unit 52 determines whether or not the intersection Vm belongs to the masking region 62A caused by the shadow of the vehicle 2, and the intersection Vm does not belong to the masking region 62A. In this case, the vehicle width Vw is specified based on the intersection Vm.

(Other variants)
In the above-described embodiment, the camera ECU 6 functions as a three-dimensional object detection device, but the present invention is not limited to this, and any device included in the in-vehicle system 1 may function as a three-dimensional object detection device.

Further, in the above-described embodiment, the functional blocks shown in FIG. 2 are schematic views showing the components of the camera ECU 6 classified according to the main processing contents in order to facilitate understanding of the present invention. The components of can be further classified into more components according to the processing content. It can also be categorized so that one component performs more processing.

Further, in the above-described embodiment, the directions such as horizontal and vertical, and various shapes, unless otherwise specified, the directions around them and similar shapes (so-called so-called) as long as they have the same effect and effect. Equal range) is included.

1 In-vehicle system 2 Vehicle 4 Imaging unit 6 Camera ECU (three-dimensional object detection device, computer)
9A Display device 24 Three-dimensional object position identification unit 30 Camera image acquisition unit 34 Bird's-eye view conversion processing unit 36 Difference image generation unit 50 Mask difference image generation unit 51 Near ground line identification unit 52 Vehicle width specification unit (width specification unit)
53 Remote ground line identification part 54 Position identification part 60 Other vehicle candidate area 62, 62A Masking area 64 Non-masking area 90 Mask image A Other vehicle (three-dimensional object)
B Travel direction Ch Lateral direction Chf Far direction E Edge image F Bird's-eye view image G Difference image Gm Mask difference image H Other vehicle area H1 Temporary first other vehicle area (first area)
H2 provisional second other vehicle area (second area)
L1 Near ground line L2 Far ground line Lc Horizontal line M Captured image N Masking area boundary O Shooting position P Vertical contour line Rc Mask difference histogram Th3 Third threshold V End point Vm, Vm1, Vm2 Intersection point Vw, Vw1, Vw2 Vehicle width

Claims (7)

A bird's-eye view conversion processing unit that converts each of the first and second shot images taken by the camera at different timings while the vehicle is running into the first bird's-eye view image and the second bird's-eye view image.
A difference image generation unit that generates a difference image of the first bird's-eye view image and the second bird's-eye view image in which the shooting positions are aligned with each other.
A mask difference image generation unit that generates a mask image that masks a region other than the candidate three-dimensional object candidate region in which a three-dimensional object is reflected in the difference image, masks the difference image with the mask image, and generates a mask difference image.
Based on the mask difference image, a near-grounding line specifying portion that specifies the near-grounding line of a three-dimensional object in the difference image,
The end points of the three-dimensional object are obtained based on the mask difference image, and the width of the three-dimensional object is based on the distance between the non-masking region boundary, which is the boundary of the non-masking region in the mask image, and the end points of the three-dimensional object. Width specification part to specify, and
A distant grounding line specifying portion that specifies a distant grounding line of the three-dimensional object in the difference image based on the width of the three-dimensional object and the nearby grounding line.
A position specifying unit that specifies the position of a three-dimensional object in the difference image based on the near ground line and the far ground line.
A three-dimensional object detection device characterized by comprising. The width specifying part is
The intersection of the line extending from the end point of the three-dimensional object in the width direction of the three-dimensional object and the boundary of the non-masking region is specified, and the width of the three-dimensional object is based on the distance between the intersection and the end point of the three-dimensional object. The three-dimensional object detection device according to claim 1, wherein the three-dimensional object detection device is characterized. The width specifying part is
The three-dimensional object detection device according to claim 2, wherein the mask image is scanned along a line extending from the end point of the three-dimensional object in the width direction of the three-dimensional object to identify the intersection. The position specifying part is
In the difference image, the three-dimensional object extends from the imaging position and includes the vertical contour line of the three-dimensional object, and the three-dimensional object at the near-grounding line based on the intersection of each of the near-grounding line and the distant grounding line. The first region in which the three-dimensional object is located and the second region in which the three-dimensional object is located are specified in the distant ground line, and the three-dimensional object is found in the difference image based on the range in which the first region and the second region overlap. The three-dimensional object detection device according to any one of claims 1 to 3, wherein the three-dimensional object region in which the image is reflected is specified. The near ground line identification part is
In the mask difference histogram in which the horizontal direction perpendicular to the traveling direction of the vehicle is the horizontal axis and the pixel value of each pixel of the mask difference image is the cumulative value of the traveling direction difference amount accumulated along the traveling direction as the vertical axis. The three-dimensional object detection device according to any one of claims 1 to 4, wherein the near ground line is specified based on the position of the horizontal axis in which the cumulative value of the difference amount in the traveling direction exceeds the third threshold value. The three-dimensional object detection device according to any one of claims 1 to 5.
A display device that displays the position of another vehicle specified by the three-dimensional object detection device, and
An in-vehicle system characterized by being equipped with. The computer
A step of converting the first and second shot images taken by the camera at different timings while the vehicle is running into the first bird's-eye view image and the second bird's-eye view image, respectively.
A step of generating a difference image of the first bird's-eye view image and the second bird's-eye view image in which the shooting positions are aligned with each other, and
A step of generating a mask image that masks a region other than a candidate three-dimensional object candidate region in which a three-dimensional object is reflected in the difference image, masking the difference image with the mask image, and generating a mask difference image.
Based on the mask difference image, a step of identifying a grounding line in the vicinity of a three-dimensional object in the difference image, and
The end points of the three-dimensional object are obtained based on the mask difference image, and the width of the three-dimensional object is based on the distance between the non-masking region boundary, which is the boundary of the non-masking region in the mask image, and the end points of the three-dimensional object. And the steps to identify
A step of identifying a distant grounding line of the three-dimensional object in the difference image based on the width of the three-dimensional object and the near grounding line.
A step of identifying the position of a three-dimensional object in the difference image based on the near ground line and the far ground line, and
A method for detecting a three-dimensional object, which comprises.

Images