JP4683272B2 - Vehicle external display method and display device - Google Patents

Description

  The present invention relates to a vehicle external display method and apparatus that are mounted on a vehicle or the like and can support driving by a driver.

As a device for assisting driving, a device that displays a turn signal on an instrument panel of an automobile or an external scene on a pillar has been proposed. Here, the turn signal and the external scene are displayed as virtual images.
For example, Patent Document 1 proposes various uses of this type of virtual image, and it is proposed to project an external scene on the inner surface of a pillar. The technique described in this document includes an imaging device that can capture an external scene, a viewpoint detection device that can detect a driver's viewpoint, and an image editing device. Image editing is to be performed.

In this document, paragraph [0051] states, “The image editing device 18 edits external information of a blind spot from the driver from the supplied image information of the front and rear of the vehicle and the viewpoint position signal. Then, the edit processing data is supplied to the holographic display device 19, and the holographic display device 19 displays the blind spot information 13 g on the real space at each pillar 31 position so as to be integrated with the visual field 23. Is described.
However, there is no specific description regarding how to detect the driver's viewpoint and how to edit the captured image captured by the camera.

JP 11-115546 A (paragraph 51, FIGS. 6, 7 and 8)

For example, when a scene outside the pillar is projected on the pillar in front of the driver, the projection state is good (an image that is faithful and naturally reflects the external scene and has no deviation). As long as it is, the effect can be exhibited.
Therefore, it becomes a problem when the image is distorted or the like, and when an unnecessary image is projected in the window around the pillar, unnatural feeling cannot be forbidden.
Furthermore, since the scene outside the pillar changes every moment in the driving state, it is required to project the outside scene almost without distortion and without deviation.

From this point of view, referring to the above-described prior art, there is no specific explanation regarding the editing process in the image editing apparatus, the processing process becomes complicated, and images other than the blind spot area are projected in a distorted and shifted state. There is a possibility that it will be.
Furthermore, although the pillar inner surface has a relatively complicated three-dimensional shape, no proposal has been made regarding this type of three-dimensional shape.

  An object of the present invention is to apply established image processing to the inner surface of a vehicle structure, for example, a pillar located in front of the driver, to give an unnatural impression in real time in an external scene that changes from moment to moment. Therefore, the present invention provides a vehicle external display method and apparatus that can perform projection without any problem and can perform driving support quickly and accurately.

In order to achieve the above object, the characteristic means of the vehicle external display method according to the present application is as follows:
An imaging step of obtaining an image of a scene outside the vehicle captured by the imaging means as a camera viewpoint image;
From the camera viewpoint image, performing a first image conversion step of obtaining a driver viewpoint image, which is an image viewed when viewed from the driver, as an image on a first virtual screen virtualized outside the vehicle ,
A blind spot area image extracting step of extracting an image of a blind spot area for the driver by the vehicle structure from the driver viewpoint image ;
Based on the positional relationship between the first virtual screen and the second virtual screen virtualized at the position of the vehicle structure in the vehicle , the image of the area that becomes the blind spot obtained in the blind spot area image extraction step is converted, A second image conversion step for obtaining an image on the second virtual screen;
The converted image obtained in the second image conversion step is an image that is free from distortion and displacement with respect to the real image of the scene outside the vehicle, according to the three-dimensional arrangement configuration and the three-dimensional inner surface shape of the vehicle structure. And an interior shape correction process for correcting the
The corrected image obtained in the interior shape correcting step is projected and displayed on the vehicle structure.

In this method, a predetermined first virtual screen is assumed outside the vehicle, and the display according to the present application is handled on the assumption that an external scene is reflected on the first virtual screen. As the first virtual screen of this type, a flat screen standing vertically in front of the car may be imagined, a cylindrical screen having an axis in the vertical direction around the car, or the car as the center. A spherical screen may be assumed. Furthermore, a virtual screen that is enlarged according to the shape of the vehicle body may be assumed outside. In addition, the second virtual screen is assumed at the position of the vehicle structure in the vehicle, and the display according to the present application is handled on the second virtual screen as an image of a blind spot area obtained in the blind spot area image extraction step. It will be.
Further, in this vehicle external display method, a camera viewpoint image is obtained by using an imaging unit, and this camera viewpoint image is converted into a driver viewpoint image in a first image conversion step . Furthermore, an image of the area that is a blind spot for the driver in the driver viewpoint image obtained in this way is extracted in a blind spot area image extraction step, and this image is subjected to a second image conversion step and an interior shape correction step. Then, it is projected and displayed on the vehicle structure. Therefore, since the image converted into the driver's viewpoint is displayed, a natural display can be performed. Further, it is possible to eliminate projection onto an extra part, and it is possible to prevent projection onto a person or an object outside the vehicle or another part of the vehicle. In addition, by executing the interior shape correction process, even if the three-dimensional arrangement configuration and the three-dimensional inner surface shape of the vehicle structure are relatively complicated, an appropriate driver viewpoint image is displayed corresponding to this. It becomes possible.

When executing the above external image display method,
A viewpoint detection process for detecting a driver's viewpoint is executed, and the first image conversion process and the blind spot area image extraction process are executed based on the driver's viewpoint detected in the viewpoint detection process. It is preferable to do.
By executing the viewpoint detection step, the viewpoint of the driver is detected, the image is converted, and the blind spot area is extracted, so that an image optimal for the movement of the viewpoint can be projected.

In order to achieve the above object, further characteristic means of the vehicle external display method according to the present application include:
An imaging step of obtaining an image of a scene outside the vehicle captured by the imaging means as a camera viewpoint image, and a viewpoint detection step of detecting the driver's viewpoint,
Based on the driver's viewpoint obtained in the viewpoint detection step,
From the camera viewpoint image , a driver's viewpoint image , which is an image viewed when viewed from the driver, is executed as a first image conversion step for obtaining an image on a first virtual screen virtualized outside the vehicle, and driving Performing a blind spot area specifying step of obtaining a driver viewpoint image blind spot area as an area occupied by the vehicle structure viewed from the driver in the driver viewpoint image;
Based on the positional relationship between the first virtual screen and the second virtual screen virtualized at the position of the vehicle structure in the vehicle, the image of the driver viewpoint image blind spot area obtained in the blind spot area specifying step is converted. A second image conversion step for obtaining an image on the second virtual screen;
The converted image obtained in the second image conversion step is an image that is free from distortion and displacement with respect to the real image of the scene outside the vehicle, according to the three-dimensional arrangement configuration and the three-dimensional inner surface shape of the vehicle structure. And further executing an interior shape correction step of correcting the
The corrected image obtained in the interior shape correcting step is projected and displayed on the vehicle structure.

In this method, a predetermined first virtual screen is assumed outside the vehicle, and the display according to the present application is handled on the assumption that an external scene is reflected on the first virtual screen. As the first virtual screen of this type, a flat screen standing vertically in front of the car may be imagined, a cylindrical screen having an axis in the vertical direction around the car, or the car as the center. A spherical screen may be assumed. Furthermore, a virtual screen that is enlarged according to the shape of the vehicle body may be assumed outside. In addition, assuming a second virtual screen at the position of the vehicle structure in the vehicle and displaying an image of the driver viewpoint image blind spot area obtained in the blind spot area specifying step on the second virtual screen, the display according to the present application is displayed. Will be handled.
Further, in this method, an image outside the vehicle is taken, and the viewpoint of the driver is detected.
The viewpoint detected in this way is used to obtain a driver viewpoint image, and is used for specifying a blind spot area as referred to in the present application.
Here, the driver viewpoint image is a scene image visually recognized when a scene outside the vehicle is viewed from the driver, and can be obtained from the camera viewpoint image according to the positional relationship between the imaging means and the viewpoint.

  On the other hand, the blind spot area is an area that becomes a blind spot due to the driver's viewpoint and a vehicle structure such as a pillar to be imaged in the present application, and this area is also determined and determined by the positional relationship between the viewpoint and the vehicle structure. This is a virtual region part that can be used.

Therefore, in the present application, the viewpoint is detected, the driver viewpoint image is obtained in the first image conversion process based on the position of the viewpoint, the blind spot area in the image is specified, An image of the driver blind spot area located at is projected and displayed on the vehicle structure through the second image conversion process and the interior shape correction process .

By doing in this way, an external image without excess and deficiency can be appropriately and rapidly imaged on the vehicle structure. In addition, by executing the interior shape correction process, even if the three-dimensional arrangement configuration and the three-dimensional inner surface shape of the vehicle structure are relatively complicated, an appropriate driver viewpoint image is displayed corresponding to this. It becomes possible.

As a vehicle external display device of this configuration,
Imaging means for imaging a scene outside the vehicle as a camera viewpoint image;
A viewpoint detection means for detecting the viewpoint of the driver,
Based on the driver's viewpoint obtained by the viewpoint detection means,
A first conversion means for converting the camera viewpoint image into a driver viewpoint image that is an image viewed when viewed from the driver, as an image on a first virtual screen virtualized outside the vehicle ;
A blind spot area specifying means for obtaining, as a driver viewpoint image blind spot area, an area occupied by the vehicle structure viewed from the driver in the driver viewpoint image;
Based on the positional relationship between the first virtual screen and the second virtual screen virtualized at the position of the vehicle structure in the vehicle, the image of the driver viewpoint image blind spot area obtained by the blind spot area specifying means , Second conversion means for converting to an image on a second virtual screen;
According to the three-dimensional arrangement configuration and the three-dimensional inner surface shape of the vehicle structure, the image after conversion obtained by the second conversion unit is an image that is free from distortion and displacement with respect to the real image of the scene outside the vehicle. Further comprising an interior shape correcting means for correcting to
The corrected image obtained by the interior shape correcting means can be projected and displayed on the vehicle structure.

Furthermore, in the above vehicle external display method ,
An image region on the first virtual screen of the vehicle structure as viewed from OPERATION's perspective as a vehicle structural blind area, an area portion of the driver's viewpoint image located in the vehicle structure blind area, the operation It is preferable that the image is a person's viewpoint image blind spot area image.

Assuming a first virtual screen such as this, the on it, identify and processing of the conversion and the blind area from the camera viewpoint images to the driver viewpoint image becomes very simple. As a result, quick and accurate processing is possible.

As a vehicle external display device using this method ,
An image region on said first virtual screen of the vehicle structure as viewed from OPERATION's perspective as a vehicle structural blind area, an area portion of the driver's viewpoint image located in the vehicle structure blind area, the operation It can be configured as an image of a person's viewpoint image blind spot area.

In the external display method described so far,
A conversion table in which the camera viewpoint image is the driver viewpoint image is prepared in advance with the position of the viewpoint as a parameter,
Preferably, the conversion table to be used is specified based on the detected viewpoint, and the camera viewpoint image is converted into the driver viewpoint image using the specified conversion table.

In the device of the present application, if the position of the imaging means and the position of the viewpoint are determined, the conversion table for the viewpoint conversion is automatically determined. Therefore, a conversion table using the position of the viewpoint as a parameter is prepared, and according to the detected viewpoint, a conversion table to be used is determined and a driver viewpoint image is obtained. The target image can be obtained by the processing.

External display devices that use this technique are:
A conversion table in which the camera viewpoint image is the driver viewpoint image is prepared in advance with the position of the viewpoint as a parameter,
The conversion table to be used is specified based on the detected viewpoint, and the camera viewpoint image is converted into the driver viewpoint image using the specified conversion table.

On the other hand, a similar configuration can be adopted for the blind spot area.
That is, the blind spot area of the vehicle structure in the first virtual screen is obtained in advance with the position of the viewpoint as a parameter,
Identifying the vehicle structure blind spot area to be used based on the detected viewpoint, and using the identified vehicle structure blind spot area,
The image of the driver viewpoint image blind spot area is obtained.

In the device of the present application, if the viewpoint is determined, the vehicle structure blind spot area on the first virtual screen is automatically determined. Therefore, by preparing information on the vehicle structure blind spot area on the first virtual screen with the position of the viewpoint as a parameter, and specifying where the blind spot area is according to the detected viewpoint, it is quick and constant. The target image can be obtained by the processing according to the scheme.

  External display devices that use this technique are:
  The vehicle structure blind spot area is obtained in advance with the position of the viewpoint as a parameter,
  The vehicle structure blind spot area to be used is specified based on the detected viewpoint, and an image of the driver viewpoint image blind spot area is obtained using the specified vehicle structure blind spot area.

As described above, the vehicle external display method according to the present application greatly depends on the position of the detected viewpoint.
Therefore, as a method for detecting the position of this type of viewpoint, the position of the driver's eyes obtained by capturing a driver image in the seated state, the shape of the seat in the seated state in which the driver is seated, or the vehicle It can be determined based on any one or more of the provided sheet positions.

As a detection device,
It can be obtained based on any one or more of the position of the eyes of the driver image in the seated state, the seat shape in the seated state where the driver is seated, or the seat position provided in the vehicle.

  FIG. 1 shows a configuration of a vehicle external display device 100 according to the present application. The devices 102, 103, and 104 provided in the vehicle are illustrated in accordance with their arrangement positions and A display device main body 101 that is an arithmetic processing device is shown in a functional block diagram thereof.

  In the example shown in the figure, the vehicle external display device 100 includes an imaging camera 102 for imaging the front and left and right directions of the car 1, a viewpoint detection device 103 for detecting the position of the viewpoint 3 of the driver 2, and a projector. And a display device main body 101 that gives information necessary for projection to the projector 104. The display device main body 101 includes an input / output device 101a that communicates with the driver 2 as a user. The imaging camera is not limited to the one for imaging the front and left and right directions of the car, and may be provided with a camera for imaging the rear side and the rear side.

The display device body 101 is configured to display an image (projector image) to be projected by the projector 104 based on the camera viewpoint image Ic obtained from the imaging camera 102 and the position information of the viewpoint 3 of the driver 2 obtained from the viewpoint detection device 103. Ip).
Further, the display device main body 101 is configured to receive ON / OFF information of the ignition key 4. The ON / OFF of the vehicle external display device 101 is linked to the ON / OFF of the ignition key 4. It is said that.

Hereinafter, each apparatus which comprises the vehicle external display apparatus 101 is demonstrated in order.
1 imaging camera 102 (an example of imaging means)
In the example shown in FIG. 1, an imaging camera 102 for taking external images in the front and front left and right directions of the car 1 is provided in front of the car 1 and at a vehicle body center position. A scene of the pillar back area that becomes a blind spot by the pillar 6 for the driver 2 can also be imaged. As this type of camera 102, for example, a CCD camera, a CMOS camera, or the like is used.

2 viewpoint detection device 103 (an example of viewpoint detection means)
In the example shown in the figure, the viewpoint detection device 103 includes an infrared camera 8 installed in the vicinity of the support portion of the rearview mirror 7 and a plurality of load cells 10 provided in the seat support frame 9. Detection information from the infrared camera 8 and the plurality of load cells 10 is sent to the viewpoint deriving means C1 provided in the viewpoint detection apparatus 103, and converted into viewpoint positions.
The infrared camera 8 is configured to place the face of the driver 2 within its field of view. In the viewpoint deriving means C1, the eye position is estimated from the face image, and the up and down direction and the left and right direction (in the vehicle) The position of the viewpoint 3 in the width direction) is derived.
On the other hand, the load cell 10 provided in the seat support frame 9 is provided in each frame of the seat support frame 9 configured as a frame having a rectangular planar shape, and in the viewpoint deriving means C1, two pairs that form a pair in the arrangement relationship, respectively. From the output of the load cell 10, the position in the front-rear direction of the viewpoint 3 in the vehicle is derived. The position in the front-rear direction may be obtained from the position of the seat.
Therefore, this viewpoint deriving means C1 makes it possible to obtain vertical, horizontal, and longitudinal positions in the vehicle of the viewpoint 2 of the driver 2.

3 Projector 104
In the example shown in FIG. 1, the projector 104 is provided on the pillar inner surface 6 a located diagonally right front of the driver 2 for projecting an image outside the pillar, and as shown in FIG. 104a is disposed so as to be substantially orthogonal to the virtual screen S2 representing the pillar inner surface 6a, and is disposed in a positional relationship capable of being projected on the entire pillar inner surface 6a. Specifically, the projector 104 is a so-called laser projector which is a scanning projection apparatus.
The projector 104 receives the projector image Ip to be projected from the display device main body 101 and projects the image Ip on the pillar inner surface 6a. FIG. 2 shows the external image projected on the inner surface 6a of the pillar 6 located on the right front side of the driver. The external image projected on the pillar inner surface 6a and the glass portion 11 around the pillar appear to pass through. It can be seen that an integrated natural state is obtained with the external real image.
The projector 104 is configured to be able to adjust the position and the direction of the image plane (the direction of the optical axis).

4 Display device body 101
The display device main body 101 is a part that controls the operation of the entire vehicle external display device 100 and a part that performs processing for obtaining an image necessary for display in the projector 104.
As shown in FIG. 1, the input to the display device main body 101 is ON / OFF information of the ignition key 4, camera viewpoint image Ic, and viewpoint 3 information.

  On the other hand, as shown in FIG. 1, a storage unit m is provided in the main body 101, and lens correction information i1 for correcting distortion caused by an optical lens provided in the apparatus 100 is stored in the storage unit m. The pillar-projector conversion information i2 and the interior shape correction information i3 for performing correction based on the interior shape of the car 1 are stored, and the camera viewpoint image with the position of the viewpoint 3 as a parameter is used as the driver viewpoint image. A viewpoint conversion table i4 to be converted and blind spot area information i5 storing a blind spot area on the driver viewpoint image are stored.

The processing in the display device main body 101 is to convert the camera viewpoint image Ic into the driver viewpoint image Id, specify the blind spot area D in the driver viewpoint image Id, and project the image of the blind spot area D from the projector 104. The processing proceeds in the order of generation of the projector image Ip necessary for the operation.
Then, the projector image Ip thus obtained is sent to the projector 104, and an image of an appropriate blind spot area D is projected on the pillar 6.

Further, the display device main body 101 is provided with an initialization processing means C2, which is configured to execute a predetermined initialization process in response to an ON operation of the ignition key 4, and if necessary, The projector 104 itself is configured to adjust the position and the optical axis direction. However, the adjustment of the position of the projector 104 is performed by the driver 2 who is a user.
In the vehicle external display device 100, in the normal state, the projector image Ip is generated based on the camera viewpoint image Ic and the driver's viewpoint 3 that are always detected, and this is projected well. On the other hand, for example, when the vehicle external display device 100 is installed for the first time or after the installation, a relatively long period of time has passed, and it is considered that the position / direction of the projector 104 has deviated from the original position / direction. In such a situation, the input / output device 101a provided for the display device main body 101 is used to adjust the position of the projector 104 (so-called calibration) based on the question / instruction from the display device main body 101 side. It can be carried out.

  Hereinafter, the lens correcting unit C3, the first converting unit C4, the blind spot area specifying unit C5, and the projector image generating unit C6 (second converting unit C6a and interior shape correcting unit C6b) shown in FIG. The function will be explained with specific examples.

Generation of the driver viewpoint image from the camera viewpoint image This image generation is an image conversion performed by the first conversion unit C4 on the image from which the influence of the optical lens by the lens correction unit C3 is eliminated, and is stored in the storage unit m. This is performed using the stored viewpoint conversion table i4.

In the vehicle external display device 100 according to the present application, the position of the imaging camera 102 and the imaging direction thereof are fixed with respect to the car 1, and at the same time, as shown above, the position of the viewpoint 3 of the driver 2 is Identified. Accordingly, as shown in FIGS. 3A and 3B, the relationship between the camera viewpoint image Ic captured by the imaging camera 102 and the driver viewpoint image Id viewed from the viewpoint 3 of the driver 2 is uniquely determined. The That is, the imaging camera 102 is disposed at the center position in the width direction of the car 1 with the imaging direction facing forward, and the viewpoint 3 of the driver 2 is located on the right side of the center position in the width direction of the car 1. When looking at the vehicle, another vehicle 200 that is largely visible on the front right side of the vehicle 1 is reduced by the difference in the position in the front-rear direction when viewed from the driver's viewpoint, and further becomes an image on the left side. As shown in FIG. 3A, this situation is the same as viewing the image shown on the virtual screen S1 disposed in front of the car 1 from a different viewpoint. Therefore, this type of transformation can be coordinate transformation such as so-called affine transformation.
An example of this conversion will be described with reference to FIG. Here, in order to simplify the description, conversion of five pixels arranged in a straight line will be described.
In FIG. 4, FIG. 4 (a) shows an arrangement of five pixels arranged in a straight line as seen from the camera viewpoint, and FIG. 4 (e) shows how this is seen from the driver's viewpoint. It is shown whether it becomes a correct arrangement.
The conversion from the viewpoint of the driver of the five pixels shown in the figure assumes an angle difference θ between the optical axis of the camera and the driver's line of sight with respect to the camera viewpoint and the driver viewpoint, and assumes a screen tilted by the angle difference θ. Can be identified. In this case, the center of the original image (the position of the third pixel among the five arranged) is set as the rotation center. This state is the state shown in FIG. Then, the intersection of the line connecting the pixel on the original screen and the driver's viewpoint and the tilted screen becomes the position of the pixel after conversion. Accordingly, as shown in (c) and (d), the distance between the pixels changes, and the ratio of the pixel arrangement changes. Further, as shown in FIG. 4E, the image size is adjusted from the positional relationship between the driver viewpoint and the converted image plane.
In this way, a set of pixels formed on the tilted screen associated with the driver's viewpoint 3 becomes the image after the viewpoint conversion. That is, the driver viewpoint image described above can be obtained.

  In view of this, in the present application, a viewpoint conversion table i4, which is a conversion table between the camera viewpoint image Ic and the driver viewpoint image Id, is prepared using the position of the viewpoint 3 as a parameter, and the first conversion means C4 performs detection. The viewpoint conversion table i4 is selected and specified based on the viewpoint 3 of the driver 2 to be generated, and the current driver viewpoint image Id is generated from the current camera viewpoint image Ic.

Specification of blind spot area D in driver viewpoint image Id This specification of blind spot area D is a process performed by blind spot area specifying means C5.
In the vehicle external display device 100 according to the present application, the viewpoint 3 of the driver 2 is detected. On the other hand, the position of the pillar 6 with respect to the viewpoint 3 is specified because the shape of the vehicle 1 is fixed. Therefore, the external area that is a blind spot for the driver 2 is determined based on the position of the viewpoint 3.
In FIG. 5, the vehicle 1 is arranged in the front direction of the vehicle 1 with respect to the viewpoints 3 (viewpoint 3a shown as “viewpoint 1” and viewpoint 3b shown as “viewpoint 2” in the figure) at different positions. The blind spot area (driver viewpoint blind spot area referred to in the present application) D on the virtual screen S1 that can be assumed to be present is shown. As can be seen from this figure, it can be seen that the blind spot area D moves inward in the width direction on the virtual screen S1 as the viewpoint 3 moves outward in the width direction of the car 1. The blind spot area D on the virtual screen S1 is determined based on the relationship between the position of the viewpoint 3 and the position of the pillar 6.
Therefore, in the vehicle external display device 100 according to the present application, the blind spot area information i5 on the virtual screen S1 with the position of the viewpoint 3 as a parameter is stored in the storage unit m, and the detected driver 2's Based on the position of the viewpoint 3, the blind spot area specifying means C5 specifies the blind spot area D on the virtual screen S1 formed by the pillar 6.

Specifically, a virtual mask image M that masks an area other than the blind spot area D is generated for the blind spot area D on the virtual screen S1, and the mask image M is sent to the lower side.
FIG. 6 shows a blind spot area D on the virtual screen S1 corresponding to different viewpoints 3 (shown as “viewpoint 1”, “viewpoint 2”, and “viewpoint 3” in the figure), and the blind spot area D is divided into sections. A mask image M is shown. According to the driver's viewpoint 3 (“viewpoint 1”, “viewpoint 2”, “viewpoint 3”), it can be seen that the blind spot area D moves from the right side to the left side according to the positional relationship between the viewpoint 3 and the pillar 6. .

Extraction of an image located in the blind spot area in the driver viewpoint image Through the above steps, as shown in FIG. 7, the driver viewpoint image Id and the mask image M should be used to image the pillar as the overlapping area. Extract images. FIG. 7 shows this image as a transmissive part image Idp.

Generation of Projector Image Ip The transmission part image Idp obtained as described above is sent to the projector image generation means C6, and the conversion determined from the relationship between the viewpoint 3 of the driver 2 and the projector 104, the shape of the pillar 6, and the three-dimensional shape Receive correction from shape. That is, conversion / correction based on the pillar-projector conversion information i2 and the interior shape correction information i3 stored in the storage unit m is performed.
The former conversion is mainly the same conversion as described above in the conversion relationship between the camera viewpoint image Ic and the driver viewpoint image Id, and this conversion is executed by the second conversion means C6a.
Regarding the conversion at this stage, since the transmission part image Idp is obtained, assuming the vertical virtual screen S2 at a position representing the pillar 6, the image to be projected is determined.
This image to be projected is somewhat reduced in accordance with the positional relationship between the distance from the viewpoint 3 to the virtual screen S1 set outside the vehicle and the virtual screen S2 set to a position representing the pillar 6, and both screens are displayed. It is influenced by the inclination of S1 and S2.
On the other hand, the position of the pillar 6 and the position of the projector 104 are fixed, and as shown in FIG. 8A, the optical axis 104a of the projector 104 is orthogonal to the virtual screen S2 at the position representing the pillar 6. Therefore, when the virtual screen S2 is arranged in parallel with the projection surface of the projector 104, an image obtained by performing viewpoint correction on the above-described image to be projected by the arrangement relationship between the pillar and the virtual screen S2. Can be used as the projector image Ip. These conversions are executed by the second conversion means C6a shown in FIG.

In the normal pillar 6, the pillar 6 has a three-dimensional arrangement configuration extending obliquely forward, and its cross-sectional surface shape is also a complicated shape as shown in FIG.
Therefore, the projector image generation means 6 is provided with an interior shape correction means C6b in addition to the second conversion means C6a.
The information regarding this correction can be determined in advance because the positional relationship between the projector 104 and the pillar 6 is determined, and the three-dimensional shape of the pillar 6 is also determined. Stored as i3.

  This correction will be described with respect to the case where the pillar 6 is composed of two surfaces 60 and 61 extending in the front-rear direction and the width direction of the vehicle 1 shown in FIG. FIG. 8A shows a state in which the pillar 6 is viewed from the driver 2, and as the pillar inner surface 6a is directed downward, the pillar 6 is located on the back side (front side of the car) as viewed from the driver 2. It is shown that the pillar inner surface 6a is composed of a ridge line 62 protruding to the driver 2 side, and two surfaces 60 and 61 extending from the ridge line 62 to both sides and extending to the side away from the driver 2.

  Now, let us consider a case where the image of the blind spot area D to be formed on the virtual screen S2 imaginary at a position representing the pillar 6 described above is a simple square image A as shown in FIG. In this case, as shown in FIG. 8C, if the image A is projected on the pillar inner surface 6a as it is, a combination of straight lines in which the upper side a and the lower side b of the square are bent is not preferable. Therefore, as an image to be supplied to the projector 104, as shown in FIG. 8D, the upper side a and the lower side b projected on the surfaces 60 and 61 extending in the direction away from the driver 2 are The image is corrected so that it becomes the division lines a1, a2, b1, b2 bent at the ridgeline 62. By doing so, as shown in FIG. 8E, the image projected from the projector 104 onto the pillar inner surface 6a can project the external image without an unnatural impression.

  In the above description, the processing in the projector image generation means C6 has been described as processing by separate means of the second conversion means C6a and the interior shape correction means C6b. However, when the viewpoint 3 of the driver 2 is identified, The image to be projected on the pillar inner surface 6 a is specified from the relationship between the position of the virtual screen S <b> 1 and the position of the pillar 6.

  On the other hand, since the three-dimensional shape of the pillar inner surface 6a and the arrangement relationship of the projector 104 are fixed, the projector image Ip which is an image used for projection by the projector 104 is the position of the virtual screen S1 set outside the vehicle and the pillar. 6 and the three-dimensional shape of the pillar inner surface 6 a and the arrangement relationship of the projector 104 can be uniquely obtained.

  Therefore, the projector image generation means 6 can project the image of the blind spot area D on the pillar inner surface 6a satisfactorily by performing a process in which the conversion by the second conversion means C6a and the correction by the interior shape correction means are integrated. Become.

Initialization Process In the vehicle external display device 100 according to the present application, the current position (including direction) of the projector 104 can be appropriately adjusted by the driver to the position where the projector 104 should be.

An initialization processing unit C2 is provided in the display device main body 101.
The purpose of this initialization process is to capture information necessary for external display and to adjust the position of the projector 104 (calibration). The acquisition of the necessary information is a process that is always executed at the start of operation of the device. The latter adjustment is performed by the driver 2 at the time of initial installation of the vehicle external display device 100 or after a predetermined period of use, for example. It is executed as appropriate when it is determined that adjustment is necessary.
The concept of this adjustment method is schematically shown in FIG. The left side of FIG. 9 shows the pillar 6 positioned on the right front side of the driver, and the right side shows the projector 104 and the initialization image Ii projected from the projector 104 during this adjustment.
When the vehicle external display device 100 is used, the markers 12 are installed at predetermined three positions with respect to the pillar 6 as shown in the left figure. The position of the marker 12 is a position determined in advance in relation to the vehicle type and the position of the projector 104 in accordance with the shape of the pillar 6 in the vehicle, and the position and optical axis of the pillar 6 and the projector 104 are determined. This is a position that is set to keep it appropriate. FIG. 9 shows a case where the marker 12 is provided at one point on the upper end of the pillar 6 and two points on the lower side.
On the other hand, the storage unit m stores a position (a position of a target point that is a marker position on the vehicle shape data) with respect to the pillar 6 where the initialization setting marker 12 is to be disposed, and corresponds to this position. Thus, a configuration is adopted in which the initialization image Ii shown on the right side of the figure can be generated and the marker point i12 as the target point can be projected onto the pillar 6. When the projector 104 is properly arranged with respect to the pillar 6, the marker point i <b> 12 of the projected image is projected as a bright spot on the marker 12 positioned on the pillar 6.

  The operation flow of this initialization processing means C2 is shown in FIG. On the other hand, FIG. 11 shows a main flow of the vehicle external display device 100. The initialization process is located in the first stage (step S1) of the main flow, and is always executed when the operation of the vehicle external display device 100 is started (for example, when the ignition key 4 is turned on).

This initialization process will be described according to the flow shown in FIG.
Step 101 (S101)
When the initialization process is started, the lens correction information i1 stored in the storage unit m is read. When the lens correction information i1 is stored as a conversion table, it can be used as it is. When the lens correction information i1 is stored as an expression representing lens characteristics, the lens correction information i1 is converted to adapt the lens correction to the image information. It will change to the form of the table.

Steps 102 and 103 (S102 and 103)
The information i2 and i3 for forming the projector image Ip from the image of the blind spot area D described above is called from the storage unit m. In this case, the pillar-projector conversion information i2 and the internal shape correction information i3 described above are called together and the projector image Ip can be generated.

The above three steps are steps automatically executed on the apparatus side in accordance with the initialization process.
Hereinafter, steps from step 104 (S104) to step 109 (S109) are processing steps relating to adjustment (calibration) of the projector 104, and are processing steps performed between the apparatus 100 and the driver 2.
Step 104 (S104)
When this step is reached, the input / output device 101a provided with the display device main body 101 is inquired about whether or not the projector 104 needs to be adjusted (calibrated), and a calibration start trigger from the driver 2 side is inquired. The presence or absence of is determined.
For example, by turning on the uppermost lighting button of the input / output device 101a, it is urged to determine whether adjustment is necessary. When this button is pressed (with a start trigger), it is determined that adjustment is necessary. It is. On the other hand, if no operation is performed within a certain time, the start trigger can be absent.
As described above, for example, when the vehicle external display device 100 is first installed or used for a long period of time, the driver 2 feels a large shift in the display image. The driver 2 inputs this start trigger.
If there is a start trigger (step 104; yes), the adjustment process from step 105 to step 109 is entered. On the other hand, when there is no start trigger (step 104; no), these processes are omitted and the initialization process is ended (end).

Step 105 (S105)
In this step, the position of the target point on the initialization image, which is the position where the marker 12 described above is provided, is read from the storage unit m, and the initialization image Ii shown on the right side of FIG. 9 is generated.
Step 104 and step 105 described above are executed as a series of processing on the apparatus side.

Subsequently, a series of processing shown in Step 106 to Step 108 is executed for each marker 12. In these steps, an adjustment operation of the projector 104 by the driver 2 intervenes.
Step 106 (S106)
In this step, the scan line is projected at the position of the target point using the initialization image Ii.
Step 107 and Step 108 (S107 and 108)
In these steps, the device sequentially inquires of the driver 2 whether or not the position of the marker 12 provided on the pillar 6 and the position of the scanning line projected from the projector 104 are the same (S107). If they do not match, the driver 2 is instructed to adjust the projector 104 and adjusted by the driver 2 (S108).
In the case of the input / output device 101a described above, the lighting is performed in the order of the second, third, and fourth from the top in the order of the image of the bright spot. Completion can be confirmed.
Step 109 (S109)
In this step, it is confirmed between the driver 2 and the device 100 whether or not the adjustment for all the points has been completed for the three markers 12, and the initialization process is completed in a state in which all the adjustments have been completed. .

The above is the configuration of the vehicle external display device 100 according to the present application and the content of the initialization process. Hereinafter, the external display will be described based on the main flow shown in FIG.
The main flow is a (start) processing flow that is started when the ignition key 4 is turned on. After the initialization processing (step 1) is executed, the camera viewpoint image Ic is appropriately captured from the imaging camera 102 and necessary. After performing this process, the projector image Ip referred to in the present application is generated and projected onto the pillar.
Step 1 (S1)
The initialization processing described above is executed by the initialization processing means C2.
At the stage of completing this step, lens correction information i1, lens-projector conversion information i2, and interior shape correction information i3 for lens correction are incorporated so that they can be applied to arithmetic processing. It has been adjusted to an appropriate positional relationship.
Step 2 (S2)
A camera viewpoint image Ic is acquired from the imaging camera 102. This acquisition is executed at a predetermined acquisition interval set in advance. This acquisition interval is determined in consideration of the time required for the projection processing of a single specific camera viewpoint image Ic from step 2 to step 8 in the main flow.
Step 3 (S3)
Image correction of the camera viewpoint image Ic is performed using the lens correction information i1 determined in advance by the initialization process. As a result, an image not affected by the lens is processed thereafter.
Step 4 (S4)
The viewpoint 3 of the driver 2 is acquired. In other words, the viewpoint 3 of the driver 2 is derived by the viewpoint position detection means C 1 based on the detection results of the infrared camera 8 and the load cell 10 and sent to the display device main body 101.
Step 5 (S5)
The lens-corrected camera viewpoint image Ic is converted into a driver viewpoint image Id. This conversion is executed by the first conversion means C4, and is converted into an image that can be seen when the external projection shown on the virtual screen S1 virtually provided outside the vehicle is viewed from the viewpoint 3 of the driver 2. Become.
At this time, a conversion table stored in the storage unit m using the position of the viewpoint 3 as a parameter is specified and used.
Step 6 (S6)
Separately, based on the detection result of the viewpoint 3 of the driver 2, a blind spot area D on the virtual screen S1 virtually provided outside the vehicle is specified. This process is a process executed by the blind spot area specifying means C5, and is a process of specifying and extracting the blind spot area D stored in the storage unit m using the viewpoint 3 as a parameter. This results in mask selection / extraction.
Using the driver viewpoint image Id obtained in step 5 and the mask obtained in step 6, a transmission part image Idp is generated and sent to the projector side.
Step 7 (S7)
Then, a projector image Ip that is drawing data is extracted and generated. This process is a process executed by the projector image generation unit C6. The conversion by the second conversion unit C6a considering the virtual screens S1 and S2, the viewpoint 3, and the position of the projector 104, and the interior shape correction unit considering the interior shape. This is accompanied by correction by C6b.
Step 8 (S8)
The projector image (image consisting of drawing data) Ip generated in this way is sent to the projector 104 and displayed as an image.
At this time, the scanning range of the projector 104 also depends on the position of the viewpoint 3 (indicated as “viewpoint 1”, “viewpoint 2”, and “viewpoint 3” in the drawing) and the surface shape of the interior. The example shown in FIG. 12 is an example when the viewpoint 3 moves around the position of the projector.
As shown in the figure, the maximum irradiation range of the projector is scanned so as to cover the entire pillar 6 as indicated by “viewpoint 1” in the positional relationship with the pillar. In this case, the region corresponding to the lower inner surface 6d is a region that does not scan. In the case of “viewpoint 3” shown in the figure, the region corresponding to the upper inner surface 6u with respect to the scanning region of “viewpoint 1”. Is a region not scanned.
By performing a series of processes from step 2 to step 8 on a specific image, a scene outside the pillar can be projected on the pillar 6 quickly and satisfactorily.
Step 9 (S9)
This step is a step for determining whether or not to continue to perform image display. It is determined whether or not the ignition key 4 is turned off. If it is turned off, the operation is stopped and the ON operation is performed. If it is maintained, the process returns to step 2 to display a new image.
In the above description, the vehicle external display device 100 according to the present application is operated based on ON / OFF of the ignition key 4, but of course, an original ON / OFF switch is provided to operate the vehicle. It may be done.

By executing the above operation, an external image can be projected on the pillar well.
[Another embodiment]
a. In the above embodiment, an external scene is projected on the pillar in front of the driver. However, with respect to a part that performs this kind of projection, a vehicle body part that forms a blind spot for the driver. If so, an external image can be projected on any part.
For example, it may be projected on a rear pillar, a rear seat, a hood, or the like.
b. In the above embodiment, the projector image generation means is provided, and the projector image scanned from the projector is used while the scanning line moves parallel to the upper side or the lower side with respect to the rectangular image region. Explained the case of executing. The scanning direction in this case is shown on the lower side of FIG.
However, in order to display accurately on the pillar surface without performing the conversion / correction by the second conversion means and the interior shape correction means, the scanning direction and position of the scanning lines in this image are determined between the pillar and the projector. You may change based on a positional relationship and the specific inner surface shape of a pillar. In this example, when the image is the same as the image shown in FIG. 5B, the scanning direction and position are converted as shown in FIG. In this case, the process associated with Step 102 and Step 103 in the initialization process is not necessary, and the image (transmission part image Idp) that is fixed in the blind spot area of the driver viewpoint image is directly sent from the display device body to the projector. In the image to be sent, the image is displayed in accordance with the virtual screen S1, the positional relationship between the pillar-projector, the scanning direction determined based on the pillar shape, and the position on the image.
c. In the above embodiment, the mask is information stored in advance in the storage unit as information with the viewpoint as a parameter. However, the virtual screen having the three-dimensional vehicle shape data and virtual outside the vehicle (the previous screen) Based on S1) shown in the example, it may be obtained by calculation every time a viewpoint is detected on the virtual screen.
d. In the above embodiment, the position of the viewpoint is detected as the position of the eyes of the driver image or from the seat shape and the position of the seat, but is set in advance without positive detection. The position of the viewpoint, for example, the position of the viewpoint according to the position of the headrest may be stored in the storage unit, and processing according to the information may be performed.

  For example, by applying established image processing to the inner surface of the pillar located in front of the driver, it is possible to project an external scene that changes from moment to moment in real time without giving an unnatural impression. It was possible to provide a vehicle external display method and apparatus capable of providing quick and accurate support.

Explanatory drawing of a structure of the vehicle external display apparatus in use condition Figure showing an external scene projected on a pillar Explanatory drawing showing the generation configuration of the driver viewpoint image The figure which shows the example of the conversion form between pixels in the production | generation of a driver | operator viewpoint image Explanatory drawing showing the blind spot area that will vary depending on the viewpoint The figure which shows the relationship between the blind spot area and the mask image that will differ depending on the viewpoint The figure which shows the image of the blind spot area | region of a driver | operator viewpoint image using a mask image and a driver | operator viewpoint image. Illustration of interior shape correction The figure which shows the projection state with respect to the marker at the time of initialization processing Flow chart showing initialization process flow Flow chart showing main flow Illustration of scanning area based on different viewpoint positions

1: Car 2: Driver 3: Viewpoint 6: Pillar 100: Vehicle external display device 101: Display device body 102: Imaging camera 103: Viewpoint detection device 104: Projector

Claims (12)

An imaging step of obtaining an image of a scene outside the vehicle captured by the imaging means as a camera viewpoint image;
From the camera viewpoint image, performing a first image conversion step of obtaining a driver viewpoint image, which is an image viewed when viewed from the driver, as an image on a first virtual screen virtualized outside the vehicle ,
A blind spot area image extracting step of extracting an image of a blind spot area for the driver by the vehicle structure from the driver viewpoint image ;
Based on the positional relationship between the first virtual screen and the second virtual screen virtualized at the position of the vehicle structure in the vehicle , the image of the area that becomes the blind spot obtained in the blind spot area image extraction step is converted, A second image conversion step for obtaining an image on the second virtual screen;
The converted image obtained in the second image conversion step is an image that is free from distortion and displacement with respect to the real image of the scene outside the vehicle, according to the three-dimensional arrangement configuration and the three-dimensional inner surface shape of the vehicle structure. And an interior shape correction process for correcting the
A vehicle external display method for projecting and displaying a corrected image obtained in the interior shape correction step on the vehicle structure. A viewpoint detection process for detecting a driver's viewpoint is executed, and the first image conversion process and the blind spot area image extraction process are executed based on the driver's viewpoint detected in the viewpoint detection process. The vehicle external display method according to claim 1. An imaging step of obtaining an image of a scene outside the vehicle captured by the imaging means as a camera viewpoint image, and a viewpoint detection step of detecting the driver's viewpoint,
Based on the driver's viewpoint obtained in the viewpoint detection step,
From the camera viewpoint image , a driver's viewpoint image , which is an image viewed when viewed from the driver, is executed as a first image conversion step for obtaining an image on a first virtual screen virtualized outside the vehicle, and driving Performing a blind spot area specifying step of obtaining a driver viewpoint image blind spot area as an area occupied by the vehicle structure viewed from the driver in the driver viewpoint image;
Based on the positional relationship between the first virtual screen and the second virtual screen virtualized at the position of the vehicle structure in the vehicle, the image of the driver viewpoint image blind spot area obtained in the blind spot area specifying step is converted. A second image conversion step for obtaining an image on the second virtual screen;
The converted image obtained in the second image conversion step is an image that is free from distortion and displacement with respect to the real image of the scene outside the vehicle, according to the three-dimensional arrangement configuration and the three-dimensional inner surface shape of the vehicle structure. And further executing an interior shape correction step of correcting the
A vehicle external display method for projecting and displaying a corrected image obtained in the interior shape correction step on the vehicle structure. An image region on the first virtual screen of the vehicle structure as viewed from OPERATION's perspective as a vehicle structural blind area, an area portion of the driver's viewpoint image located in the vehicle structure blind area, the operation The vehicle external display method according to claim 3, wherein the image is a person's viewpoint image blind spot area image. A conversion table in which the camera viewpoint image is the driver viewpoint image is prepared in advance with the position of the viewpoint as a parameter,
The vehicle exterior according to claim 3 or 4, wherein the conversion table to be used is specified based on the detected viewpoint, and the camera viewpoint image is converted into the driver viewpoint image using the specified conversion table. Display method. The vehicle structure blind spot area in the first virtual screen is obtained in advance with the position of the viewpoint as a parameter,
Identifying the vehicle structure blind spot area to be used based on the detected viewpoint, and using the identified vehicle structure blind spot area,
The vehicle external display method according to claim 4, wherein an image of the driver viewpoint image blind spot area is obtained. The position of the viewpoint is either the position of the eyes of the driver obtained by taking a driver image in the seated state, the seat shape in the seated state where the driver is seated, or the seat position provided in the vehicle The vehicle external display method according to any one of claims 3 to 6 , which is obtained based on one or more. Imaging means for imaging a scene outside the vehicle as a camera viewpoint image;
A viewpoint detection means for detecting the viewpoint of the driver,
Based on the driver's viewpoint obtained by the viewpoint detection means,
A first conversion means for converting the camera viewpoint image into a driver viewpoint image that is an image viewed when viewed from the driver, as an image on a first virtual screen virtualized outside the vehicle ;
A blind spot area specifying means for obtaining, as a driver viewpoint image blind spot area, an area occupied by the vehicle structure viewed from the driver in the driver viewpoint image;
Based on the positional relationship between the first virtual screen and the second virtual screen virtualized at the position of the vehicle structure in the vehicle, the image of the driver viewpoint image blind spot area obtained by the blind spot area specifying means , Second conversion means for converting to an image on a second virtual screen;
According to the three-dimensional arrangement configuration and the three-dimensional inner surface shape of the vehicle structure, the image after conversion obtained by the second conversion unit is an image that is free from distortion and displacement with respect to the real image of the scene outside the vehicle. Further comprising an interior shape correcting means for correcting to
A vehicle external display device that projects and displays a corrected image obtained by the interior shape correcting means on the vehicle structure. An image region on the first virtual screen of the vehicle structure as viewed from OPERATION's perspective as a vehicle structural blind area, an area portion of the driver's viewpoint image located in the vehicle structure blind area, the operation The vehicle external display device according to claim 8 , wherein the vehicle viewpoint image is a blind spot area image. A conversion table in which the camera viewpoint image is the driver viewpoint image is prepared in advance with the position of the viewpoint as a parameter,
The vehicle exterior according to claim 8 or 9, wherein the conversion table to be used is specified based on the detected viewpoint, and the camera viewpoint image is converted into the driver viewpoint image using the specified conversion table. Display device. The vehicle structure blind spot area is obtained in advance with the position of the viewpoint as a parameter,
Identify the vehicle structure blind area to be used on the basis of the detected viewpoint, using said identified vehicle structure blind spot regions, according to claim 9, wherein obtaining the image of the driver's viewpoint image blind area Vehicle external display device. The position of the viewpoint is obtained based on any one or more of a position of an eye of a driver image in a seated state, a seat shape in a seated state in which the driver is seated, or a seat position provided in a vehicle. Item 12. The vehicle external display device according to any one of Items 8 to 11 .

Images