JP4731392B2 - In-vehicle peripheral status presentation device - Google Patents

Description

  The present invention relates to an in-vehicle peripheral situation presentation device that is installed in a car and presents a situation around the automobile to a user.

  As a technology for presenting the situation around the vehicle mounted on the vehicle to the user, a camera that captures the rear from the rear of the vehicle, a camera that captures the rear from the left side of the vehicle, and the rear from the right side of the vehicle There is known a technique for photographing the back of an automobile with three cameras and cameras and displaying the images photographed by the three cameras in combination (for example, Patent Document 1).

In addition, a technique for performing image recognition processing on an image obtained by photographing the front of the automobile with a camera to identify the other vehicle in front of the automobile and presenting information on the identified other vehicle is also known (for example, Patent Document 2). .
JP 2003-81014 A Special table 2005-509984

  When photographing the periphery of a car with a camera, it is inevitable that there will be a range (a blind spot) where the camera cannot shoot the situation because it is behind another car. On the other hand, the size of such a blind spot can be reduced by providing a plurality of cameras that photograph the same region from different directions as in the technique of Patent Document 1.

  However, when presenting the situation around the vehicle by displaying images taken with the camera, displaying images taken with multiple cameras at all times is the same if no blind spots have occurred. This is not efficient from the viewpoint of effective use of the display area. In addition, when presenting the situation around the vehicle by presenting information such as other vehicles identified by performing image recognition processing on the image captured by the camera, all images captured by a plurality of cameras are displayed. On the other hand, when the image recognition process is always performed, the image recognition process is performed on an image having the same content when no blind spot is generated, and thus the efficiency of the image recognition process is lacking.

  Therefore, an object of the present invention is to present a situation around the host vehicle more efficiently using a plurality of cameras while suppressing the generation of blind spots.

  In order to achieve the above object, the present invention provides a first on-vehicle peripheral situation presentation device that is mounted on an automobile and presents the situation around the automobile to a user. Photographed by the first camera, a second camera that photographs a second photographing area that is an area including at least a part of the first photographing area, from a different direction from the first camera. The situation information presenting means for presenting the situation of the first imaging area, which is represented by an image, to the user and other objects in the first imaging area are behind the other cameras, and cannot be taken by the first camera. A blind spot area calculating means for calculating a blind spot area, which is an area where the blind spot area calculated by the blind spot area calculating means is overlapped with the second imaging area. In case, Serial second camera is represented by the photographed image, in which to perform the presentation of the status of the overlap region.

  According to such an in-vehicle peripheral situation presentation device, only when a blind spot occurs in the first camera, the image captured by the first camera is complemented with the image captured by the second camera, and the first camera Present the surrounding situation of the shooting area. Therefore, the situation of the first imaging region is more efficient than the case where the situation is presented using the image always captured by the first camera and the image captured by the second camera while suppressing the occurrence of blind spots. Can be presented.

  Here, in such an in-vehicle surrounding state presentation device, the first camera may take an image of the rear of the vehicle from the rear of the vehicle, and the second camera may take an image of the direction including the rear of the vehicle from the side of the vehicle. . In this case, the on-vehicle peripheral situation presentation device is provided with a third camera that captures a third imaging area that is an area including at least a part of the first imaging area, and the second camera The direction including the rear of the automobile is taken from the left side of the automobile, the third camera is taken of the direction including the rear of the automobile from the right side of the automobile, and the blind spot area calculating means in the situation information presenting means When an area where the blind spot area calculated by the above and the third imaging area overlap is generated, the situation of the overlapping area represented by the image captured by the third camera is presented. Also good.

By doing in this way, the situation behind the automobile can be efficiently presented while suppressing the occurrence of blind spots.
In each of the on-vehicle surrounding state presentation devices as described above, the blind spot area calculation means includes an area that cannot be captured by the first camera because it is behind other objects in the first imaging area. The area may be calculated as the blind spot area only when it has a spread of a predetermined level or more.

  By doing this, the situation is presented using only the image photographed by the first camera, except when the area that cannot be photographed by the first camera has a practical extent. As a result, the process for presenting the situation around the vehicle can be performed more efficiently.

  As described above, according to the present invention, it is possible to efficiently present a situation around the own vehicle while suppressing the generation of blind spots using a plurality of cameras.

Hereinafter, embodiments of the present invention will be described.
FIG. 1 a shows the configuration of the surrounding situation presentation device according to the present embodiment.
This peripheral situation presentation device is a device mounted on an automobile, and as shown in the figure, a radar device 1, a rear camera 2, a left rear camera 3, a right rear camera 4, a rear image extraction unit 5, a peripheral situation presentation unit 6, A display device 7 and an audio output device 8 are provided.
Here, as shown in FIG. 1b, the radar apparatus 1 and the rear camera 2 are installed in the rear part of the automobile. The radar apparatus 1 detects an object existing behind the host vehicle, and measures a relative position to the detected object. The rear camera 2 captures the back of the vehicle. Next, the left rear camera 3 is installed, for example, on the left side mirror of the automobile, and images the rear from the left side of the automobile. Further, the right rear camera 4 is installed, for example, on a right side mirror of an automobile and photographs the rear from the right side of the automobile.

Next, the rear image extraction unit 5 performs the rear image extraction process shown below.
FIG. 2 shows the procedure of the rear image extraction process.
As shown in the figure, in this process, first, it is determined whether or not there is an object such as another vehicle that has a distance from the vehicle in the front-rear direction of the vehicle within a predetermined distance ThL behind the vehicle. (Step 202). This determination is made when the radar apparatus 1 detects an object behind the host vehicle having a relative position in which the relative distance in the front-rear direction of the host vehicle is within the predetermined distance ThL. Then, it is performed by determining. However, this determination is performed by performing an image recognition process on the image behind the host vehicle captured by the rear camera 2 to detect an object behind the host vehicle, and the position in the image where the object is reflected, According to the size of the image in which the object is reflected, the relative position of the object with respect to the own vehicle is obtained, and if the relative distance in the front-rear direction of the own vehicle obtained from the obtained relative position is within the predetermined distance ThL, It may be performed by determining that such an object exists.

  Then, when it is determined that there is no object such as another vehicle in which the distance in the front-rear direction of the own vehicle from the own vehicle is within the predetermined distance ThL behind the own vehicle, the rear camera 2 takes an image. The image is extracted as an effective rear image for the entire shooting area of the rear camera 2 and is output to the surrounding state presentation unit 6 (step 208), and the process returns to step 202. Here, the imaging area of the rear camera 2 is determined by the direction of the rear camera 2 facing the rear camera 2 (usually the optical axis direction of the objective lens of the rear camera 2) and the angle of view of the rear camera 2. An area within a range photographed by the rear camera 2 when the rear of the vehicle is only the horizontal ground.

  On the other hand, when there is an object such as another vehicle whose distance in the front-rear direction of the own vehicle from the own vehicle is within the predetermined distance ThL behind the own vehicle (step 202), the object is hidden behind the object. Then, the angle range occupied by the rear concealed object in the angle of view of the rear camera 2 in the horizontal direction is calculated as the blind angle range (step 204). Here, the blind spot angle range is obtained, for example, by image recognition processing using the relative position of the rear concealed object obtained in advance, for example, the size of the image of the rear concealed object in the image captured by the rear camera image. Then, the horizontal angle range occupied by the rear concealed object in the angle of view of the rear camera 2 is determined from the horizontal size of the image of the rear concealed object as the blind spot angle range.

  If the blind spot angle range is obtained in this way, it is next checked whether or not the obtained blind spot angle range is larger than a predetermined predetermined size ThA (step 206). Then, the image captured by the rear camera 2 is extracted as a valid rear image for the entire photographing region of the rear camera 2, and is output to the peripheral state presentation unit 6 (step 208), and the process returns to step 202.

  On the other hand, if the obtained blind angle angle range is larger than a predetermined predetermined size ThA (step 206), the image taken by the rear camera 2 is excluded from the photographing area of the rear camera 2. The region is extracted as an effective rear image and is output to the surrounding situation presentation unit 6 (step 210). Here, the blind spot area means that the angle formed with respect to the rear camera 2 is within the previously determined blind spot angle range, and is behind the rear camera 2 from the relative position of the rear concealed object obtained earlier. Refers to the region.

  Next, it is checked whether or not there is an overlapping portion between the photographing region and the blind spot region of the left rear camera 3 (step 212). If there is an overlapping portion, the photographed image of the left rear camera 3 is converted into the left rear camera. 3 is extracted as an effective rear image with respect to the overlapping portion of the imaging region 3 and the blind spot region, and is output to the surrounding state presentation unit 6 (step 216). Further, it is checked whether or not there is an overlapping portion between the photographing region and the blind spot region of the right rear camera 4 (step 214). If there is an overlapping portion, the photographed image of the right rear camera 4 is photographed by the right rear camera 4. The overlapping portion of the region and the blind spot region is extracted as an effective rear image and is output to the surrounding state presentation unit 6 (step 218). Then, the process returns to step 202.

  The imaging area of the left rear camera 3 is the left when the periphery of the vehicle is only a horizontal ground determined by the direction of the left rear camera 3 and the angle of view of the left rear camera 3 behind the vehicle. An area within a range to be photographed by the rear camera 3 is said. In addition, the shooting area of the right rear camera 4 is determined when the area around the vehicle is only a horizontal ground, which is determined from the direction in which the right rear camera 4 faces and the angle of view of the right rear camera 4 behind the vehicle. An area within the range to be photographed by the rear camera 4 is said.

The rear image extraction process performed by the rear image extraction unit 5 has been described above.
Hereinafter, a processing example of such rear image extraction processing will be described.
Now, in the arrangement relationship shown in FIG. 3 a 1, when another vehicle 302 and a motorcycle 303 exist behind the own vehicle 301 equipped with the rear camera 2 that captures the rear at an angle of view 311, Since both the motorcycles 303 are far behind the vehicle by a predetermined distance ThL (step 202), the image shown in FIG. 3a2 captured by the rear camera 2 is an effective rear image for the entire rear camera photographing region. Extracted and output to the circumferential strong table presenting unit (step 208).

  Next, in the arrangement relationship shown in FIG. 3b1, when another vehicle 302 and a motorcycle 303 exist behind the own vehicle 301 equipped with the rear camera 2 that captures the rear at an angle of view 311, the vehicle 302 Is larger than the predetermined distance ThL behind the host vehicle, but since the motorcycle 303 exists at a position within the predetermined distance ThL behind the host vehicle (step 202), the blind angle range by the motorcycle 303 is calculated (step 202). Step 204). In the case of the arrangement of FIG. 3b1, this blind spot angle range is θ312 in the figure, and this θ312 is smaller than a predetermined size ThA (step 206). The image shown in 3b2 is extracted as a valid rear image for the entire rear camera photographing region, and is output to the circumferential strong table presenting unit (step 208).

  Here, as described above, when the object does not exist within the predetermined distance ThL behind the host vehicle or when the viewing angle range by the object existing behind the host vehicle is smaller than the predetermined size ThA, the rear camera 2 is By extracting the captured image as an effective rear image for all of the rear camera photographing regions, the blind spot region of the rear camera 2 caused by other objects is reduced to such an extent that there is no practical problem in presenting the surrounding situation. Or it will be far away.

  Next, in the arrangement relationship shown in FIG. 4a, when a bus 402, another automobile 403, and a motorcycle 404 exist behind the own vehicle 401 provided with the rear camera 2 that captures the rear at an angle of view 411. The automobile 403 and the motorcycle 404 are separated from the vehicle rearward by a predetermined distance ThL, but the bus 402 exists at a position within the predetermined distance ThL behind the vehicle (step 202). Is calculated (step 204). In the case of FIG. 4a, this blind spot angle range becomes θ412 in the figure, and this θ412 is larger than a predetermined size ThA (step 206).

  Therefore, in this case, the image shown in FIG. 4c2 taken by the rear camera 2 is extracted as an effective rear image for the area 421 excluding the blind spot area by the bus 402 from the rear camera photographing area shown in FIG. It is output to the situation presentation unit 6 (step 210). Also, the image shown in FIG. 4c1 taken by the left rear camera 3 having the angle of view 413 is extracted as an effective rear image for the overlapping portion 422 of the left rear camera photographing region shown in FIG. (Steps 212 and 216), the information is output to the surrounding situation presentation unit 6. Also, the image shown in FIG. 4c3 taken by the right rear camera 4 having the angle of view 414 is extracted as an effective rear image for the overlapping portion 423 of the left rear camera photographing region shown in FIG. (Steps 214 and 218), and is output to the surrounding situation presentation unit 6.

The processing example of the rear image extraction processing has been described above.
Now, referring back to FIG. 1, the surrounding situation presentation unit 6 performs processing for presenting the surrounding situation based on the rear image output from the rear image extraction unit 5.
The processing of the surrounding situation is performed as follows, for example.
In other words, during the period in which only the image captured by the rear camera 2 is output as the rear image by the rear image extraction unit 5, only the image 501 captured by the rear camera 2 is displayed on the display device 7 as shown in FIG. .
Then, during the period in which a plurality of rear images are output by the rear image extraction unit 5, the plurality of output rear images are displayed in an arrangement according to the positional relationship of the regions where each image is effective. 5a2 shows an image 511 taken by the left rear camera 3, an image 512 taken by the rear camera 2, and an image 513 taken by the right rear camera 4 output in the situation shown in FIG. The example arrange | positioned according to the positional relationship of right and left is shown. Here, in this example, for each image, only a portion of the image obtained by photographing an effective area of the image is displayed. In the case of the situation in FIG. 4, the portion where the region where the image is valid in the image captured by the left rear camera 3 is the portion 431 in FIG. 4 c 1, and the portion in the image captured by the right rear camera 4 A portion where an image is effective is photographed as a portion 432 in FIG. 4c3.

  However, during a period in which a plurality of rear images are output by the rear image extraction unit 5, instead of the display as illustrated in FIG. 5a2, as illustrated in FIG. The image 521 observed from the viewpoint may be combined and displayed. That is, FIG. 5a3 shows a portion in which an image is effective in the image captured by the left rear camera 3, and a portion in which an image is effective in the image captured by the right rear camera 4. Extraction is performed, and viewpoint conversion processing is performed to convert the image captured from the same viewpoint as that of the rear camera 2. In addition, the image converted in this way is visually recognized when the rear concealed object of the image captured by the rear camera 2 is reflected in the portion of the image that is translucent and seen through the image. As shown, the images are combined in an arrangement according to the area reflected in each image. By doing in this way, the blind spot area by the back concealment object was image | photographed with the left back camera 3 and the right back camera 4 by the expression that the back concealment object became translucent and the object behind it was seen through. Part can be presented.

Further, the processing for presenting the surrounding situation may be performed as follows.
That is, for each rear image output from the rear image extraction unit 5, the object detection image recognition process for the region where the rear image is effective is performed using a portion of the rear image where the region where the rear image is effective is reflected. The object identified by the image recognition processing performed on each rear image is presented using the display device 7 and the audio output device 8.

More specifically, during the period in which only the image captured by the rear camera 2 is output as the rear image by the rear image extraction unit 5, image recognition processing is performed only on the image captured by the rear camera 2, and the rear camera Detection and identification of an object in the second imaging region are performed.
On the other hand, for example, in the situation shown in FIG. 4, when an image taken by the left rear camera 3, an image taken by the rear camera 2, and an image taken by the right rear camera 4 are output, the rear camera 2 Image recognition processing is performed on a portion of the image captured by the area where the area 421 excluding the blind spot area by the bus 402 from the rear camera imaging area shown in FIG. 4B is captured, and the object in the area 421 is detected and identified. . Further, image recognition processing is performed on a portion (431 in FIG. 4c1) where an overlapping portion 422 of the left rear camera photographing region shown in FIG. Then, the object in the area 422 is detected and identified. Further, an image recognition process is performed on a portion (432 in FIG. 4c3) where an overlapping portion 423 of the right rear camera photographing region shown in FIG. The object in the area 423 is detected and identified.

  Then, as shown in FIG. 5 b, an image in which marks 541, 542, 543 representing the identified objects are arranged according to the positional relationship between the object and the own vehicle on the graphic 531 is displayed on the display device 7. By displaying, each identified object is presented. Or, when the identified object is approaching the host vehicle exceeding a predetermined level, the voice output device 8 outputs a sound to that effect, thereby presenting information on the identified object, etc. Process.

The embodiment of the present invention has been described above.
Thus, according to the present embodiment, when a blind spot occurs in the rear camera 2, the image captured by the rear camera 2 is complemented with the image captured by the left rear camera 3 or the right rear camera 4, Present the surrounding situation. Therefore, the situation behind the automobile can be presented while suppressing the occurrence of blind spots.

  Further, in the present embodiment, since the above-described complementation using the images photographed by the left rear camera 3 and the right rear camera 4 is performed only for the blind spot area of the rear camera 2, the situation of the rear of the automobile can be efficiently performed. You can make a presentation. In addition, when a camera having a lower resolution or the like than the rear camera 2 is used as the left rear camera 3 or the right rear camera 4, the rear camera 2 presents a high quality image of the rear of the vehicle as much as possible. It will be possible to use this image.

  In addition, the above embodiment is applicable similarly also when showing the condition of directions other than the back of the own vehicle. For example, when presenting the situation in front of the host vehicle, the rear of the above embodiment may be replaced with the front and applied.

It is a block diagram which shows the structure of the periphery condition presentation apparatus which concerns on embodiment of this invention. It is a flowchart which shows the back image extraction process which concerns on embodiment of this invention. It is a figure which shows the process example of the back image extraction process which concerns on embodiment of this invention. It is a figure which shows the process example of the back image extraction process which concerns on embodiment of this invention. It is a figure which shows the example of presentation of the periphery condition which concerns on embodiment of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Radar apparatus, 2 ... Back camera, 3 ... Left back camera, 4 ... Right back camera, 5 ... Back image extraction part, 6 ... Peripheral condition presentation part, 7 ... Display apparatus, 8 ... Sound output device.

Claims (5)

An on-vehicle peripheral situation presentation device that is mounted on an automobile and presents the situation around the automobile to a user,
A first camera for photographing a first photographing region around the own vehicle;
A second camera that images a second imaging area that is an area including at least a part of the first imaging area from a different direction from the first camera;
Status information presenting means for presenting the status of the first imaging area to the user, represented by an image captured by the first camera;
A blind spot area calculating means for calculating a blind spot area that is an area that cannot be captured by the first camera because it is shaded by other objects in the first capturing area;
The situation information presenting means is represented by an image captured by the second camera when an area where the blind spot area calculated by the blind spot area calculating means and the second imaging area overlap is generated. A vehicle-mounted peripheral situation presentation device that presents a situation of an overlapping area. The on-vehicle peripheral situation presentation device according to claim 1,
The on-vehicle peripheral situation presenting apparatus characterized in that the first camera photographs the rear of the automobile from the rear of the automobile, and the second camera photographs a direction including the rear of the automobile from the automobile side. An in-vehicle peripheral situation presentation device according to claim 2,
The vehicle-mounted peripheral situation presentation device includes a third camera that captures a third imaging area that is an area including at least a part of the first imaging area.
The second camera captures the direction including the rear of the vehicle from the left side of the vehicle,
The third camera captures the direction including the rear of the vehicle from the right side of the vehicle,
The situation information presenting means is represented by an image photographed by the third camera when an area where the blind spot area calculated by the blind spot area calculating means and the third photographing area overlap is generated. A vehicle-mounted peripheral situation presentation device that presents a situation of an overlapping area. The vehicle-mounted peripheral situation presentation device according to claim 1, 2, or 3,
The blind spot area calculation means only determines the area when the area that cannot be captured by the first camera due to the shadow of another object in the first imaging area has a predetermined level or more. An on-vehicle peripheral situation presentation device characterized by calculating as a blind spot area. In a device mounted on an automobile, a surrounding situation presentation method for presenting a situation around the automobile to a user,
Photographing a first photographing area around the vehicle by a first camera mounted on the automobile;
Photographing a second imaging area that is an area including at least a part of the first imaging area from a direction different from that of the first camera by a second camera mounted on an automobile;
Calculating a blind spot area that is an area that cannot be imaged by the first camera because it is shaded by other objects in the first imaging area;
Presenting to the user the status of the first imaging area represented by an image captured by the first camera;
A step of presenting a situation of the overlapping area represented by an image captured by the second camera when an area where the calculated blind spot area and the second imaging area overlap occurs; A method for presenting a surrounding situation, comprising: