JP7304922B2 - Perimeter monitoring system - Google Patents

Description

TECHNICAL FIELD The present invention relates to technology for presenting obstacles around a vehicle to a user.

As a technique for presenting obstacles around a vehicle to a user, a photographed image of interest, which is an image of a narrow field of view, which is a partial region of an image photographed by a back camera that photographs the rear of the automobile, is displayed on a display device. There is known a technique of extracting an image of an obstacle included outside the narrow field of view of a captured image and displaying it in a composite manner on the displayed captured image of interest (for example, Patent Document 1).

Further, in this technique, the compositing position of the image of the obstacle on the photographed image of interest is set to the upper portion of the photographed image of interest in the vertical direction, and the direction in which the obstacle exists in the horizontal direction.

Japanese Patent No. 5495071

On the photographed image of interest, which is an image of a narrow-field region that is a partial region of the photographed image of the back camera that photographs the rear of the vehicle, an obstacle exists above the photographed image of interest in the vertical direction and in the horizontal direction. According to the technique of synthesizing and displaying an image of an obstacle at a position in the direction in which the image is being viewed, depending on how the narrow-field area is set, the image portion of the captured image of interest requires the user's attention due to the image of the obstacle ( For example, the statue of another vehicle) may be hidden.

Therefore, according to the present invention, an image portion of an obstacle extracted from an image captured by a camera that captures the surroundings of a vehicle is displayed on an image representing the surroundings generated from the captured image so as not to hide an image portion that requires the user's attention. It is an object of the present invention to provide a perimeter monitoring system that synthesizes and displays images.

In order to achieve the above object, the present invention provides a peripheral monitoring system mounted on a vehicle, a display, a camera that captures a range behind the vehicle including the rear bumper of the vehicle, and an image captured by the camera. surrounding image display means for displaying on the display a surrounding image representing the surroundings of the vehicle, generated from at least a part of a photographed image; and obstacles included in at least the photographed image but not included in the surrounding image. obstacle image extracting means for extracting the image portion of the image as an obstacle image; and obstacle image display means for placing a small window on the peripheral image displayed on the display and displaying the obstacle image in the small window. is provided. Here, the peripheral image includes the image portion of the rear bumper of the automobile at the bottom. Further, the obstacle image display means arranges the small window so as to at least partially overlap an image portion of the rear bumper under the peripheral image displayed on the display, and the obstacle image display means is arranged in the small window. Display an image.

According to such a perimeter monitoring system, the obstacle image is displayed by arranging the small window so as to at least partially overlap the image portion of the rear bumper below the perimeter image displayed on the display. The display of the obstacle image prevents the image portion of the surrounding image that requires the user's attention from being hidden.

Further, in order to achieve the above object, the present invention provides a surrounding monitoring system mounted on a vehicle, a display, a camera that captures the surroundings of the vehicle, and at least a part of the captured image that is the image captured by the camera. peripheral image display means for generating a bird's-eye view image showing the surroundings of the vehicle observed from above, and displaying a peripheral image obtained by synthesizing a figure representing the vehicle with the generated bird's-eye image on the display; and at least the photographed image. obstacle image extracting means for extracting as an obstacle image an image portion of an obstacle that is included in the peripheral image and is not included in the peripheral image; and obstacle image display means for displaying the obstacle image. Here, the obstacle image display means arranges the small window on the figure representing the automobile in the peripheral image displayed on the display, and displays the obstacle image in the small window.

According to such a surroundings monitoring system, the obstacle image is displayed by arranging a small window on the figure representing the automobile in the surrounding image displayed on the display. This prevents the image portion that requires attention from being hidden.

Here, in these surroundings monitoring systems, when the obstacle whose image portion is extracted as the obstacle image is located on the left side of the vehicle, the obstacle image display means sets the left direction as the direction of the obstacle. When the obstacle is located on the right side of the vehicle, the right direction is set as the obstacle direction, and when the obstacle direction is the left direction, the layout position of the small window on the peripheral image is set to the left side of the peripheral image. position, and when the direction of the obstacle is the right direction, the position may be set to the right side of the peripheral image.

The surroundings monitoring system described above may be configured such that the obstacle image extracting means extracts, as an obstacle image, only an image portion of an obstacle included in the photographed image but not included in the surrounding image. good.

As described above, according to the present invention, an image portion of an obstacle extracted from a captured image of a camera that captures the surroundings of a vehicle is displayed on an image representing the surroundings generated from the captured image. It is possible to provide a perimeter monitoring system that synthesizes and displays parts so as not to hide them.

1 is a block diagram showing the configuration of a perimeter monitoring system according to an embodiment of the present invention; FIG. It is a figure which shows the arrangement|positioning of the display and back camera of the surroundings monitoring system which concerns on embodiment of this invention. It is a figure which shows the display mode of the perimeter monitoring system which concerns on embodiment of this invention. It is a flow chart which shows obstacle information synthesis processing concerning an embodiment of the present invention. It is a figure which shows the example of a display of the perimeter monitoring system which concerns on embodiment of this invention. FIG. 5 is a diagram showing another display example of the perimeter monitoring system according to the embodiment of the present invention; FIG. 5 is a diagram showing another display example of the perimeter monitoring system according to the embodiment of the present invention;

Embodiments of the present invention will be described below.
FIG. 1 shows the configuration of a perimeter monitoring system according to this embodiment.
The perimeter monitoring system is a system installed in an automobile. As shown in the figure, the perimeter monitoring system includes a display 1, a synthetic image generator 2, an obstacle detector 3, an obstacle image generator 4, and a view conversion processor. 5, an image memory 6, a back camera 7 for photographing the rear of the automobile, a reverse detection sensor 8 for detecting whether the automobile is in a reverse state from the state of the transmission of the automobile, the position of the shift lever, etc., an input device 9, and a control unit. 10.

Here, such a perimeter monitoring system may be configured using a computer equipped with a CPU, a memory, and various peripheral devices. The object image generation unit 4, the view conversion processing unit 5, the control unit 10, etc. may be embodied by the CPU executing a predetermined computer program.

Here, as shown in FIG. 2a, the display 1 is installed in the dashboard at the central position in the left-right direction in front of the front seats in the vehicle.
The back camera 7 is a camera having a wide angle of view in the horizontal direction, such as a fisheye lens or a wide-angle lens, and as shown in FIG.

In such a configuration, the control unit 10 sets the reverse mode when the reverse detection sensor 8 detects that the vehicle is in the reverse state, and when the reverse detection sensor 8 no longer detects that the vehicle is in the reverse state. release the reverse mode.

During the period in which the reverse mode is set, the photographed image taken by the back camera 7 is stored in the image memory 6, and the obstacle detection section 3 performs image recognition processing on the photographed image stored in the image memory 6. is applied to detect an image area of an obstacle such as another vehicle in the photographed image. During the period in which the backward travel mode is set, the obstacle image generation unit 4 converts the image portion of the obstacle image in the captured image into an obstacle image area according to the area of the obstacle image detected by the obstacle detection unit 3. Extract as an image.

Next, while the backward mode is set, the view conversion processing unit 5 generates a rearward image, which is an image subjected to conversion processing according to the display mode.
Here, the control unit 10 sets any one of wide view, normal view, and top view as the display mode according to the user's operation via the input device 9 .
Then, when the display mode is the wide view, the view conversion processing unit 5 converts the captured image, which is the image captured by the back camera 7 in the range of the angle of view θ1 of the back camera 7 shown in FIG. , is generated as it is as a rear guidance image.

Further, when the display mode is normal view, the view conversion processing unit 5 extracts from the image captured by the back camera 7 an image portion obtained by capturing the range of angle θ2 within the angle of view θ1 of the back camera 7 shown in FIG. 3b1. 3b2 to generate an image as the rear image, the distortion of which is corrected by .

Further, when the display mode is the top view, the view conversion processing unit 5 extracts an image portion obtained by photographing the area A behind the vehicle shown in FIG. 3c2, which is converted into an image showing how the range of the area A is observed from above by viewpoint conversion processing, is generated as a rear image.

Here, as shown in FIGS. 3a2, 3b2, and 3c2, in the present embodiment, in each display mode, the rear bumper image 301 of the own vehicle appears at the bottom of the rearward image. In the view and top view, the range to be extracted from the captured image and included in the rear image is set.

Further, during the period in which the backward driving mode is set, the synthetic image generation unit 2 adds the rear image generated by the view conversion processing unit 5 to the extension line of the vehicle width or from the rear end of the vehicle shown in FIGS. A rear guide image is generated by synthesizing a guide figure 302 representing a point on the rear image which is a predetermined distance in the longitudinal direction of the automobile, and is displayed on the display 1.例文帳に追加

Further, during the period in which the backward driving mode is set, the synthetic image generation unit 2 performs obstacle information synthesizing processing in order to present obstacle information to the user.
The obstacle information synthesizing process performed by the synthetic image generator 2 will be described below.
FIG. 4 shows the procedure of this obstacle information synthesizing process.
As shown in the figure, the synthetic image generator 2 checks the display mode in the obstacle information synthesizing process (steps 402, 404, 406).
Then, when the display mode is wide view (step 406), first, if a detection frame, which will be described later, is being displayed, it is erased (step 412). Then, it is checked whether or not the obstacle detection section 3 has detected an obstacle (step 414).

On the other hand, when the obstacle detection unit 3 detects an obstacle (step 414), based on the area of the obstacle image in the photographed image detected by the obstacle detection unit 3, the rear guidance image Above, a detection frame, which is a frame surrounding the area of the image of the obstacle in the rear guidance image, is synthesized and displayed on the rear guidance image displayed on the display 1 (step 416). Then, the processing from step 402 is returned to.

According to such processing, when the display mode is the wide view and the image of an obstacle such as another vehicle is included in the captured image of the back camera 7, the display is displayed as shown in FIGS. 5a1 and 5a2. 1, a detection frame 501 surrounding the image of the obstacle in the rear guidance image is synthesized and displayed. The detection frame 501 calls the user's attention to the obstacle.

Returning to FIG. 4, when the display mode is normal view (step 404), first, if a small window is being displayed, it is erased (step 422). Then, it is checked whether or not there is an obstacle image area detected by the obstacle detection unit 3, where n% or more of the area is not included in the rear guidance image. (step 424). As n%, a value is set at which, if n% or more of the area is included in the rear guidance image, a rough part of the image of the obstacle appears in the rear guidance image. Specifically, for example, n% is set to a value that satisfies 80%≦n%≦100%.

Then, if there is no obstacle image region where n% or more of the region is not included in the rear guidance image (step 424), the processing from step 402 is returned to.
On the other hand, if n% or more of the area of the obstacle image is not included in the rear guidance image (step 424), a predetermined A rectangular image of the same size is generated as an obstacle image and displayed in a small window (child screen, wipe) of a predetermined size placed on the rear guide image (step 426). Here, the position on the rear guide image where the small window is arranged is the lower part in the vertical direction. The position on the rear guidance image where the small window is arranged is the position of the side edge in the left-right direction in which the image area of the obstacle exists, that is, the area of the image of the obstacle is the area on the left side of the vehicle. If there is, the position of the left edge on the rear guidance image is set, and if it is the area on the right side, the position of the right edge is set.

Then, the processing from step 402 is returned to.
According to such processing, when the display mode is the normal view, if the image captured by the rear camera 7 includes an obstacle whose approximate part is not shown in the rear guidance image, the As shown in 5b1, an obstacle image 502 is synthesized and displayed in a small window arranged on the rear guidance image. Here, FIG. 5b1 shows the case where the obstacle is on the right side of the vehicle, and in this case, as shown, the position of the small window in which the obstacle image 502 is displayed is the lower right corner of the rear guidance image. Become.

By displaying the obstacle image 502 in such a small window, the user is presented with the fact that the obstacle represented by the obstacle image 502 exists on the right side of the vehicle.
After that, as shown in FIG. 5b2, when the approximate portion of the obstacle appears in the rear guidance image, the display of the obstacle image 502 by the small window is erased. Further, when the image of the obstacle is no longer included in the captured image of the back camera 7, the display of the obstacle image 502 by the small window is also erased.

Note that FIG. 5b3 represents the display of the obstacle image 502 when the obstacle is on the left side of the car. As shown, in this case, the small window in which the obstacle image 502 is displayed is placed in the lower left corner of the rear guidance image, and the obstacle image 502 in the small window indicates the obstacle that the obstacle image 502 represents. is on the left side of the car.

Here, as described above, in this embodiment, when the display mode is the normal view, the image 301 of the rear bumper of the own vehicle appears in the small window displaying the obstacle image 502 below the rear guidance image. Therefore, the obstruction image 502 prevents the obstacle image 502 from hiding the image portion that requires the user's attention, such as the image portion of the obstacle in the rear guide image.

Returning to FIG. 4, next, when the display mode is the top view (step 402), if the small window is being displayed, it is deleted (step 432). The area of the image of the obstacle detected by the obstacle detection unit 3 and the area of the image of the obstacle that is wholly included outside the rearward guidance image, that is, even partially within the rearward guidance image. A check is made to see if there are areas of the obstacle image that are not included (step 434).

Then, if there is no obstacle image area that is entirely outside the rear guide image (step 434), the process returns to step 402 and beyond.
On the other hand, if there is an obstacle image area that is entirely outside the rear guidance image (step 434), a rectangular image containing the obstacle image corrected for distortion is extracted from the captured image. It is generated as an object image 502 and displayed in a small window (child screen, wipe) of a predetermined size arranged on the rear guidance image (step 436). Here, the position on the rear guide image where the small window is arranged is the center in the vertical direction. In addition, the position on the rear guidance image where the small window is arranged is the position of the side edge in the left-right direction where the area of the image of the obstacle exists.

Then, the processing from step 402 is returned to.
According to such processing, when an obstacle that does not appear in the rear guidance image is included in the image captured by the back camera 7 when the display mode is the top view, as shown in FIG. At the same time, an obstacle image 502 is displayed in a small window arranged on the rear guidance image. Here, FIG. 5c1 shows the case where the obstacle is on the right side of the automobile, and in this case, as shown in the figure, the position of the small window in which the obstacle image 502 is displayed is the vertical direction on the rear guidance image. , and the rightmost position in the left-right direction.

By displaying the obstacle image 502 in such a small window, the user is presented with the fact that the obstacle represented by the obstacle image 502 exists on the right side of the vehicle.
After that, as shown in FIG. 5c2, when the obstacle portion appears in the rear guidance image, the display of the obstacle image 502 by the small window is erased. Further, when the image of the obstacle is no longer included in the captured image of the back camera 7, the display of the obstacle image 502 by the small window is also erased.

Note that FIG. 5c3 represents the display of the obstacle image 502 when the obstacle is on the left side of the vehicle. As shown in the figure, in this case, the small window in which the obstacle image 502 is displayed is displayed at the center in the vertical direction and the left end position in the horizontal direction on the rear guide image. 502 indicates to the user that the obstacle represented by the obstacle image 502 is on the left side of the car.

Here, as described above, in this embodiment, when the display mode is the top view, the obstacle image 502 is displayed at the left and right ends of the rear guidance image at positions that do not overlap the course of the vehicle's backward travel. Therefore, the obstruction image 502 is prevented from hiding an image portion representing the state of the course in the rear guidance image, which requires the user's attention.

The obstacle information synthesizing process performed by the synthetic image generator 2 has been described above.
The embodiments of the present invention have been described above.
By the way, in the above-described embodiment, when the display mode is the normal view, a rectangular image including the image of the obstacle is displayed as the obstacle image 502 in the rectangular small window arranged at the lower left corner or the lower right corner of the rear guidance image. 6a and 6b, the shape of the small window and the obstacle image 502 is the shape of the portion where the small window of the image 301 of the rear bumper of the vehicle in the rear guidance image is arranged. It may be a shape that imitates.

Further, in the above embodiment, when the display mode is the top view, the position of the small window displaying the obstacle image 502 on the rear guidance image is set to the central position in the vertical direction. The position in the vertical direction may be changed according to the direction of the obstacle with respect to the own vehicle.

That is, for example, as shown in FIG. 7a1, when an obstacle 701 is directly behind the own vehicle 700, as shown in FIG. 7b1, and the obstacle 701 is diagonally behind the own vehicle 700 as shown in FIG. When the guide image is positioned at the center and the obstacle 701 is in the lateral direction of the own vehicle 700 as shown in FIG. may be positioned below the rear guidance image.

7a1, b1, and c1, the angle γ of the direction of the obstacle detected by the obstacle detection unit 3 with respect to the left-right direction of the vehicle 700 shown in FIGS. The vertical position of the small window displaying the obstacle image 502 changes from the lower end position to the upper end position of the rear guidance image as the angle changes. This can be done by positioning.

Further, in the above embodiment, when the display mode is the top view, as shown in FIGS. The display position of the object image 502 may be a position overlapping the figure 710 representing the rear part of the own vehicle.

Also in this case, as shown in FIGS. 7d1 and 7d2, when the direction of the obstacle is to the left of the own vehicle, the direction of the obstacle is positioned to the left of the graphic 710 representing the rear of the own vehicle. A small window displaying an obstacle image 502 is arranged at a rightward position on a graphic 710 representing the rear of the vehicle when the vehicle is to the right.

In the above embodiment, the back camera 7 is used to show the user the situation behind the automobile. The same applies to the case of presenting.

REFERENCE SIGNS LIST 1 display, 2 composite image generator, 3 obstacle detector, 4 obstacle image generator, 5 view conversion processor, 6 image memory, 7 back camera, 8 backward detection sensor, 9 Input device 10 Control unit 302 Guide figure 502 Obstacle image 701 Detection frame.

Claims (3)

A perimeter monitoring system mounted on an automobile,
a display;
a camera that captures an area behind the vehicle that includes the rear bumper of the vehicle;
a surrounding image display means for displaying on the display a surrounding image representing a state of the surroundings of the automobile in a range narrower than the range captured by the camera, which is generated from a captured image that is an image captured by the camera;
obstacle image extracting means for extracting, as an obstacle image, an image portion of an obstacle included in at least the photographed image but not included in the peripheral image;
an obstacle image display means for arranging a small window on the peripheral image displayed on the display and displaying the obstacle image in the small window;
The peripheral image includes an image portion of the rear bumper of the automobile at the bottom,
The obstacle image display means arranges the small window so as to at least partially overlap an image portion of the rear bumper below the peripheral image displayed on the display, and displays the obstacle image in the small window. A perimeter monitoring system characterized by displaying. A perimeter monitoring system according to claim 1, wherein
The obstacle image display means includes:
When the obstacle whose image portion is extracted as the obstacle image is located on the left side of the vehicle, the left direction is set as the obstacle direction, and when the obstacle is located on the right side of the vehicle, the right direction is set as the obstacle direction. As for the direction, the arrangement position of the small window on the peripheral image is set to the left on the peripheral image when the obstacle direction is the left direction, and to the right on the peripheral image when the obstacle direction is the right direction. A perimeter monitoring system, characterized in that it is located at a closer position. The perimeter monitoring system according to claim 1 or 2,
The surroundings monitoring system, wherein the obstacle image extracting means extracts only an image portion of the obstacle included in the photographed image but not included in the surrounding image as the obstacle image.

Images