JP2022042718A - Image processing system - Google Patents

Abstract

To provide an image processing system configured to improve safety by allowing a user to satisfactorily confirm an obstacle surrounding a vehicle.SOLUTION: A head-mounted display unit 10 to be worn by a user on a vehicle 3 comprising around view cameras 2a-2d includes: a camera 14 which images a field of view of the user; a head-mounted display 13; a display-side communication unit 15 which wirelessly communicates with the vehicle 3; and a control unit 11 which generates a display image to be displayed on the head-mounted display 13. The control unit 11 includes an image synthesis unit 17 which synthesizes a vehicle surroundings image generated by the around view cameras 2a-2d input via the display-side communication unit 15 with a field-of-view image captured by the camera 14 to generate a mixed reality image, and causes the head-mounted display 13 to display the mixed reality image.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing system that improves visibility around a vehicle.

In recent years, various monitoring devices such as an around view monitor have been developed in order to improve the safety of the vehicle. For example, as described in Patent Document 1, the around view monitor photographs the surroundings of the vehicle by a camera mounted on the vehicle, and displays the captured image on an image display device (monitor) provided in the vehicle interior. It is possible for the occupants of the vehicle, especially the driver, to see obstacles hidden behind the vehicle and invisible.

On the other hand, in recent years, MR technology has been developed in which a virtual image is superimposed on a real image taken by a camera or the like to generate and provide a composite image (mixed reality image). Further, a technique of providing a composite image by this MR technique to a occupant (user) who is on board the vehicle has also been proposed.
For example, in Patent Document 2, a tourist or the like boarding a tourist vehicle by superimposing a virtual image (for example, tourist information, advertisement) related to the landscape on a landscape taken by a camera mounted on the tourist vehicle. A technique has been proposed in which a head-mounted display (head-mounted display device) worn by a person is displayed and provided to the passenger.

Japanese Unexamined Patent Publication No. 2007-41791 Re-table WO2012 / 033095

Since the above-mentioned vehicle monitoring device can be checked after boarding the vehicle and turning on the vehicle power, when the driver is in a hurry, the vehicle will start running without sufficient confirmation. there is a possibility.
Therefore, as in Patent Document 2, a method of displaying an image of the surroundings of the vehicle on the head-mounted display can be considered, but in Patent Document 2, the passengers in the vehicle are based on the vehicle surrounding image and the virtual image. Since it provides information, there is a possibility that the vehicle will start running without sufficiently confirming obstacles.

The present invention has been made in consideration of such circumstances, and an object of the present invention is to provide an image processing system that allows a user to sufficiently confirm obstacles around a vehicle and improves safety. It is in.

In order to achieve the above object, the image processing system of the present invention is via a mobile terminal having an image pickup unit for taking an image, an image display unit for displaying an image, and a communication unit for communicating with a vehicle, and the communication unit. When the mobile terminal is located in the vicinity of the vehicle, it has an image compositing unit that generates a composite reality image in which the acquired vehicle surrounding image obtained by photographing the surroundings of the vehicle is superimposed on the captured image captured by the imaging unit. It is characterized by including a control unit for displaying the composite real image generated by the image synthesizing unit on the image display unit.

As a result, when the mobile terminal is located near the vehicle, a mixed reality image in which the surrounding image of the vehicle is superimposed on the image captured by the image pickup unit of the mobile terminal is displayed on the image display device, so that the user can use the image pickup unit. Both the captured image and the surrounding image of the vehicle can be visually recognized. Therefore, it is possible to confirm an obstacle at a position hidden behind the vehicle and not directly visible.
Preferably, the image synthesizing unit may generate the mixed reality image by superimposing the vehicle surrounding image on the predetermined position of the vehicle in the captured image.

As a result, when the mobile terminal is located near the vehicle, the user can visually recognize the image of the predetermined position of the vehicle displayed on the image display device, and at the same time, the surrounding image of the vehicle can be visually recognized.
Preferably, the predetermined position is a position near the outer wall surface of the vehicle or the outer wall surface of the door of the vehicle.

As a result, when the mobile terminal is located near the vehicle, the user visually recognizes an image of the position near the outer wall surface of the vehicle or the outer wall surface of the door displayed on the image display unit, thereby simultaneously displaying an image of the surroundings of the vehicle. It becomes possible to visually recognize.
Preferably, the image synthesizing unit transmits the vehicle in the captured image and superimposes the vehicle surrounding image on the ground at the position of the vehicle to generate the mixed reality image.

As a result, the vehicle surrounding image is displayed on the ground of the captured image displayed on the image display unit outside the vehicle, so that the position of an obstacle can be accurately visually recognized when there is an obstacle around the vehicle. can. In addition, the user carrying the mobile terminal can easily recognize the presence or absence of obstacles around the vehicle even before the user gets into the vehicle.
Preferably, the mobile terminal is a head-mounted display device worn on the head.

As a result, the user can easily attach a mobile terminal having an image pickup unit and an image display unit, and can confirm obstacles around the vehicle outside the vehicle.
It is preferable that the head-mounted display device is attached and the mixed reality image is displayed on the image display unit when the vehicle is located in the vicinity of the vehicle in a state where the vehicle is in the field of view.
As a result, when the user wearing the head-mounted display device is located near the vehicle with the vehicle in the field of view, the mixed reality image is displayed on the image display unit and the vehicle surrounding image is confirmed. Can be done.

According to the image processing system of the present invention, when a user carrying a mobile terminal having an image pickup unit and an image display unit is located near the vehicle, a mixed reality image including a vehicle surrounding image is displayed, so that the user can use the image processing system. It becomes possible to visually recognize the surroundings of the vehicle. Therefore, even when a user carrying a mobile terminal tries to get into the vehicle and start immediately, the time for checking the surroundings of the vehicle can be increased, and the safety can be improved.

It is a schematic block diagram of the display system for a vehicle using the head-mounted display unit of this embodiment. This is a first display example of an around view image displayed on the head-mounted display according to the present embodiment. This is a second display example of the around view image displayed on the head-mounted display according to the present embodiment. This is a third display example of the around view image displayed on the head-mounted display according to the present embodiment. This is a fourth display example of the around view image displayed on the head-mounted display according to the present embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a configuration diagram of a vehicle display system 1 using the head-mounted display unit 10 (head-mounted display device, mobile terminal, image processing system) according to the embodiment of the present invention.
The vehicle display system 1 includes an around view camera 2a, 2b, 2c, 2d, a head-mounted display unit 10, and a vehicle-side communication device 12.

Of the vehicle display system 1, the around view cameras 2a, 2b, 2c, and 2d and the vehicle-side communication device 12 are mounted on the vehicle 3.
For example, one around view camera 2a, 2b, 2c, and 2d are provided on the front, rear, left, and right sides of the vehicle, and the four around view cameras 2a, 2b, 2c, and 2d cover the entire circumference of the vehicle 3 in the front, rear, left, and right directions. To shoot.

The vehicle-side communication device 12 transmits the captured image of the surroundings of the vehicle input from the around view cameras 2a, 2b, 2c, and 2d. The vehicle-side communication device 12 may be capable of transmitting at least in the vehicle interior near the vehicle and around the vehicle, for example, within a range of several meters.
The head-mounted display unit 10 includes a head-mounted display 13 (image display unit), a camera 14 (imaging unit), a display-side communication device 15 (communication unit), and a control unit 11 (control unit), and is boarded in the vehicle 3. It is attached to the head of the occupant (user).

The head-mounted display 13 is provided on the front side of the eyeball of the user who wears the head-mounted display unit 10, and has a function of displaying an image and providing image information to the user.
The camera 14 is provided, for example, in the vicinity of the head-mounted display 13, and is arranged with the lens facing the front side of the user. Therefore, the camera 14 captures (photographs) the direction in which the head of the user wearing the head-mounted display unit 10 faces, that is, the front in the line-of-sight direction.

The display-side communication device 15 and the vehicle-side communication device 12 mutually perform wireless communication. The display-side communication device 15 receives the captured image signal transmitted from the vehicle-side communication device 12.
The control unit 11 is a control device for controlling the display of the head-mounted display 13, and includes an input / output device, a storage device (ROM, RAM, non-volatile RAM, etc.), a central processing unit (CPU), a timer, and the like. Consists of.

The control unit 11 requests the captured image around the vehicle input from the vehicle-side around view cameras 2a, 2b, 2c, and 2d from the display-side communication device 15 via the vehicle-side communication device 12, and also requests the display-side communication. The image captured by the around view cameras 2a, 2b, 2c, and 2d received by the machine 15 and the image captured by the camera 14 (image) are input, and the composite image (display image) signal to be displayed on the head mount display 13 is output. Then, the composite image based on the composite image signal is displayed on the head mount display 13.

The control unit 11 includes an around view monitor image generation unit 16 and an image composition unit 17.
The around view monitor image generation unit 16 generates an around view image 20 (surrounding image) from the captured images of the around view cameras 2a, 2b, 2c, and 2d input via the vehicle side communication device 12 and the display side communication device 15. do. As shown in FIG. 2, for example, the around view image 20 is an image viewed from above by combining the image 21 of the vehicle 3 and the images around it (front, back, left, and right). For the generation of the around view image 20 in the around view monitor image generation unit 16, for example, a known around view monitor system (overhead image display system) disclosed in Patent Document 1 may be used, so detailed description thereof will be omitted. do. If the vehicle 3 can generate the around view image 20 and transmit it from the vehicle side communication device 12, the around view image 20 may be input and used via the display side communication device 15.

The image synthesizing unit 17 inputs the captured image captured by the camera 14 of the head mount display unit 10 and also inputs the around view image generated by the around view monitor image generation unit 16 to input the vehicle 3 in the captured image of the camera 14. By superimposing the around view image on the predetermined position of the vehicle 3 according to the position of, a composite image (composite reality image) of the view image and the around view image is generated.

The composite image generated by the image compositing unit 17 is displayed on the head-mounted display 13.
The head-mounted display unit 10 is provided with a portable battery and a power switch (not shown), and by turning on the power switch, the head-mounted display 13, the camera 14, and the display side of the head-mounted display unit 10 are turned on. The communication device 15 and the control unit 11 can be operated.

On the other hand, the around view cameras 2a to 2d and the vehicle-side communication device 12 can be operated when the power of the vehicle 3 is turned on. Further, the head-mounted display unit 10 is provided with a position detection unit 18 for controlling the power supply of the around view cameras 2a to 2d. The position detection unit 18 has a function of detecting the relative position of the head-mounted display 13 with respect to the vehicle 3, and the head-mounted display unit 10 whose power is turned on is in the vicinity of the vehicle 3 in a predetermined range (for example, within a range of several meters). And when it is located in the vehicle interior, the power ON signal for turning on the around view cameras 2a to 2d is transmitted from the display side communication device 15. For example, the vehicle-side communication device 12 keeps the communication from the head-mounted display unit 10 always receivable, and when the power ON signal from the head-mounted display unit 10 is received, the around view cameras 2a to 2d are turned on. Control to be. Alternatively, the position detection unit 18 has a function of outputting a radio wave within a predetermined range when the power of the head-mounted display unit 10 is turned on, in addition to the display side communication device 15, and the vehicle 3 side receives this radio wave and is around. The view cameras 2a to 2d may be configured to be turned on. As a result, when the user wearing the head-mounted display unit 10 is located within a predetermined range around the vehicle 3 and in the vehicle interior, the around view cameras 2a to 2d are turned on, and the vehicle 3 by the around view cameras 2a to 2d is turned on. It is possible to shoot the surroundings.

Next, a display example of the around view image 20 in the head-mounted display unit 10 of the present embodiment will be described.
FIG. 2 is a first display example of the around view image 20. FIG. 3 is a second display example of the around view image 20. FIG. 4 is a third display example of the around view image 20. Regarding FIGS. 2 to 4, the user is located outside the vehicle 3, for example, from the right side of the vehicle (FIGS. 2 and 3), which is the driver's seat side, or the front right side in order for the driver to board the vehicle 3. (Fig. 4) shows the state of looking at the vehicle 3.

The vehicle 3 of the present embodiment is a five-door vehicle having two front and rear doors on the left and right sides of the vehicle and a total of five back doors on the rear side of the vehicle.
As shown in FIG. 2, the around view image 20 is displayed in the vicinity of the rear of the door handle 26 on the outer wall surface of the right front door 25 of the vehicle 3.
Alternatively, as shown in FIG. 3, the around view image 20 may be displayed in the vicinity of the front upper side of the right front door 25 of the vehicle 3.

The around view image 20 is displayed on the right side of the vehicle (driver's seat side), but may be similarly displayed not only on the right side of the vehicle but also on the left side of the vehicle. Further, it may be displayed in the vicinity of the open / close switch of the rear gate of the vehicle 3.
When the head-mounted display unit 10 is worn not only by the driver but also by other users, the around view image 20 may be displayed on all the head-mounted display units 10 of the users who visually recognize the vehicle 3. When the driver is specified in the head-mounted display unit 10, for example, when the head-mounted display unit 10 for the driver is set, only the head-mounted display unit 10 for the driver displays the around view image 20. It may be displayed.

Regarding the display of the around view image 20, it may be displayed at a predetermined display position (predetermined position) such as near the rear of the door handle 26 of the right front door 25 of the vehicle 3 at all times when the power of the vehicle 3 is turned on. When the door lock device of the vehicle 3 is unlocked, the display may be performed for a predetermined time.
The vehicle 3 is provided with a door locking / unlocking device that locks / unlocks the door locking device of the vehicle 3 according to the position of the user's line of sight by the head-mounted display unit 10.

As shown in FIG. 2, the door locking / unlocking device is displayed in front of the door handle 26 of the right front door 25 located on the side of the driver's seat of the vehicle 3 in the side image of the vehicle 3 displayed by the head mount display unit 10. The door locking / unlocking mark 30 is superimposed on the display. The door locking / unlocking mark 30 has, for example, a shape in which an arrow is aligned with a rectangular mark.
Then, the door locking / unlocking device regards the center of the image captured by the camera 14 of the head-mounted display unit 10 as the line-of-sight position, and the line-of-sight position is maintained in the rectangular shape of the door locking / unlocking mark 30 for a predetermined time. If so, the lock / unlock of the door lock device of the vehicle 3 is switched. When switching between locking and unlocking, it is advisable to blink the door locking / unlocking mark 30.

When the user has a function of detecting the line-of-sight position of the user wearing the head-mounted display unit 10 as in such a door locking / unlocking device, the time when the line-of-sight position of the user is located at the door locking / unlocking mark 30. The around view image 20 may be displayed for a predetermined time.
With the above configuration, the user wearing the head-mounted display unit 10 can obtain the same field of view as when directly visually observing because the image captured in front of the user captured by the camera 14 is displayed on the head-mounted display 13. Be done.

Further, since the captured image taken by the around view cameras 2a to 2d of the vehicle 3 is combined with the captured image displayed on the head mounted display 13 of the head mounted display unit 10 and displayed as the around view image 20, the head A user wearing the mount display unit 10 can visually recognize both a field view image and a vehicle surrounding image in the vicinity of the vehicle. Therefore, it is possible to confirm an obstacle at a position hidden behind the vehicle 3 so that it cannot be directly visually recognized.

Further, the vehicle 3 and the head-mounted display unit 10 are connected by wireless communication, and when the head-mounted display unit 10 is located within a predetermined range around the vehicle 3, the around view cameras 2a to 2d of the vehicle 3 are used. Since the surroundings can be photographed, the user wearing the head-mounted display unit 10 can visually recognize the image around the vehicle even outside the vehicle 3. Therefore, it is possible to check obstacles around the vehicle 3 before getting into the vehicle 3. As a result, even when the user, particularly the driver of the vehicle, gets into the vehicle 3 and tries to start immediately, the time for checking the surroundings of the vehicle 3 can be increased, and the safety is improved. be able to.

Further, since the around view image 20 is displayed near the position of the door handle 26 of the right front door 25 on the driver's seat side, when the driver visually recognizes the door handle 26 when trying to open the right front door 25 from the outside of the vehicle. The around view image 20 is displayed in the vicinity of the line-of-sight position. As a result, the around view image 20 can be easily visually recognized when the driver gets into the driver's seat of the vehicle 3.

In particular, in the vehicle 3 provided with the door locking / unlocking device by visual inspection, when the driver locks / unlocks the vehicle 3, the door locking / unlocking mark 30 in the vicinity of the door handle 26 on the driver's seat side is visually recognized. When unlocking the right front door 25 on the driver's side, the around view image 20 can be easily and relatively visually recognized for a relatively long time to sufficiently confirm obstacles.
As shown in FIG. 4, the around view image 20 may be displayed on the ground at the parking position of the vehicle 3. The around view image 20 is displayed according to the position and size of the vehicle 3. When the around view image 20 is displayed on the ground, the vehicle 3 in the field of view image is transmitted and displayed. It is preferable that the vehicle 3 is not completely transparent, but is semi-transparent to the extent that the outer shape of the vehicle 3 can be identified. In the head-mounted display unit 10 of the present embodiment, since the user's field of view image is captured by the camera 14 and displayed on the head-mounted display 13, it is possible to transmit and display the vehicle 3 as shown in FIG. ..

Then, since the around view image 20 is displayed on the ground at the position of the vehicle 3 through the vehicle 3, an obstacle hidden in the vehicle 3 can be visually recognized from the outside of the vehicle 3.
Further, since the around view image 20 is displayed according to the position and size of the vehicle 3, an obstacle of actual size can be visually recognized at the actual position, and the obstacle can be accurately recognized. ..

The present invention is not limited to the above embodiment.
For example, the display position and size of the around view image 20 may be appropriately set.
In the above embodiment, the around view image 20 is displayed in and around the vehicle 3 viewed from the outside of the vehicle, but further, as shown in FIG. 5, the around view image 20 may be displayed in the vehicle interior.

FIG. 5 is a fifth display example of the around view image 20, when a driver who is seated in the driver's seat 31 of the vehicle 3 with the head-mounted display unit 10 attached sees the left front door 32 of the vehicle 3 from the inside. This is a display example.
In a vehicle 3 such as an automobile, it is difficult for the driver to see the left side of the vehicle on the passenger seat 33 side because it is hidden behind the left front door 32.

Therefore, as shown in FIG. 5, by displaying the around view image 20 on the left front door 32, the side of the vehicle 3 including the left side of the vehicle can be visually recognized. Since the around view image 20 is displayed by the head-mounted display unit 10, the around view image 20 can be arbitrarily displayed at any position of the vehicle 3, so that the around view image 20 can be arbitrarily displayed at an appropriate position as needed. Can be displayed, and the visibility of the around view image 20 can be improved.

Alternatively, when there is a user equipped with the head-mounted display unit 10 in the vehicle interior, the vehicle 3 is semi-transparent and the vehicle 3 is adjusted to the position and size of the vehicle 3 as shown in the third display example shown in FIG. The around view image 20 may be displayed on the ground around the. In the head-mounted display unit 10 that transmits the vehicle 3 and displays the around view image 20 according to the size and position of the vehicle 3, when viewed from the vehicle interior, the left front door is displayed, for example, on the left side of the vehicle. The around view image 20 may be displayed only in a portion that is difficult to see directly due to 32 or the like.

Further, in the vehicle interior, for example, an icon showing an around view image is displayed on the upper part of the dashboard, and when a user wearing the head-mounted display unit 10 looks at this icon for a predetermined time or longer, the around view image is displayed at the position of this icon. 20 may be displayed. If the line of sight deviates from the position of the icon, the display of the around view image 20 may be stopped and the icon display may be restored.

Further, in the above embodiment, when the head-mounted display unit 10 is in the vicinity of the vehicle 3 in a state where the predetermined position of the vehicle 3 is in the user's field of view, the mixed reality in which the vehicle surrounding image is superimposed on the predetermined position. Although the image is displayed on the head-mounted display 13, a mixed reality image may be displayed when the head-mounted display unit 10 is in the vicinity of the vehicle 3 even if the predetermined position of the vehicle 3 is not in the user's field of view. ..

Further, the configuration of the head-mounted display unit 10 may be changed as appropriate.
The present invention can be widely applied to a unit including an image display unit, which is worn or carried by a user of a vehicle equipped with an around view camera.

1 Vehicle display system 3 Vehicle 10 Head-mounted display unit (head-mounted display device, mobile terminal, image processing system)
11 Control unit (control unit)
13 Head-mounted display (image display)
14 Camera (imaging unit)
15 Display side communication device (communication unit)
17 Image synthesis unit 18 Position detection unit

Claims (6)

A mobile terminal having an image pickup unit for taking an image, an image display unit for displaying an image, and a communication unit for communicating with a vehicle.
The mobile terminal has an image synthesizing unit that generates a mixed reality image in which an image of the surroundings of the vehicle acquired via the communication unit is superimposed on the image captured by the imaging unit, and the mobile terminal is a vehicle. A control unit that displays the mixed reality image generated by the image composition unit on the image display unit when it is located in the vicinity.
An image processing system characterized by being equipped with. The image processing system according to claim 1, wherein the image synthesizing unit superimposes the vehicle surrounding image on a predetermined position of the vehicle in the captured image to generate the mixed reality image. The image processing system according to claim 2, wherein the predetermined position is a position near the outer wall surface of the vehicle or the outer wall surface of the door of the vehicle. One of claims 1 to 3, wherein the image synthesizing unit transmits the vehicle in the captured image and superimposes the vehicle surrounding image on the ground at the position of the vehicle to generate the mixed reality image. The image processing system according to item 1. The image processing system according to any one of claims 1 to 4, wherein the mobile terminal is a head-mounted display device worn on the head. 5. The fifth aspect of the present invention is characterized in that the mixed reality image is displayed on the image display unit when the head-mounted display device is attached and the vehicle is located in the vicinity of the vehicle in a state where the vehicle is in the field of view. The image processing system described.

Images