JP2005269010A - Image creating device, program and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image creating device in which a created virtual visual point image is displayed while taking account of convenience of a user. <P>SOLUTION: The image creating device comprises a space reconfiguration means for mapping input images from one or a plurality of cameras fixed to a vehicle into a spatial model, a means for detecting the operating conditions of the vehicle, a means for acquiring dead angle information indicative of the dead angle of a person in the vehicle based on the detection results and setting a virtual visual point in a three-dimensional space based on the dead angle information, a visual point conversion means for forming a virtual visual point image viewed from the virtual visual point in the three-dimensional space with reference to spatial data mapped by the space reconfiguration means, and a means for controlling the display mode of the virtual visual point image. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention does not display images independently of each other for a plurality of images taken by one to several cameras, but the entire area photographed by the one or several cameras. The present invention relates to a device and a method for displaying a composite image so that the situation can be intuitively understood. For example, the present invention is applied to a monitor device in a store or a vehicle surroundings monitor device as an auxiliary for safety confirmation when driving a vehicle. And a suitable technique.

  In recent years, there has been disclosed an image generation apparatus that displays images captured by a plurality of cameras in an easy-to-view manner (for example, Patent Document 1). This Patent Document 1 discloses an image generation apparatus that combines images of an area (for example, the vicinity of a vehicle) captured by a plurality of cameras as a continuous image and displays the combined image.

  Further, in Patent Document 1, a space model that is appropriately set in advance or a space model that is set according to the distance to an obstacle around the vehicle detected by the obstacle detection means is created by the space model creation means. The An image around the vehicle input from a camera installed on the vehicle by the image input means is mapped to the spatial model by the mapping means. Subsequently, one image viewed from the viewpoint determined by the viewpoint conversion means is synthesized from the mapped image and displayed on the display means.

In this way, in the device installed in the vehicle, what kind of object exists in the vicinity of the vehicle over the entire periphery of the vehicle is synthesized as a single image so that it is as easy to understand as possible, and driving The image can be provided to the person. Further, at this time, it is possible to display an image from the viewpoint desired by the driver by the viewpoint conversion means.
Japanese Patent No. 3286306 JP 05-265547 A Japanese Patent Laid-Open No. 06-266828 Fumiaki Tomita, “High-function 3D visual system”, Information processing, IPSJ, Volume 42, No.4

  However, in Patent Document 1, an image of an area (for example, the vicinity of a vehicle) captured by a plurality of cameras is synthesized as a single continuous image, and the synthesized image is mapped to a virtual three-dimensional space model. It is a technology that focuses on how to create an image (virtual viewpoint image) in which the viewpoint of the mapped data is virtually changed in three dimensions. The user interface for the display method and display form We do not make a concrete proposal about improving the convenience of the company.

  In view of the above problems, the present invention provides an image generation apparatus that displays a virtual viewpoint image in consideration of user convenience.

  According to the first aspect of the present invention, there is provided a spatial reconstruction means for mapping an input image from one or a plurality of cameras attached to a vehicle to a spatial model, and an operation of the vehicle. Vehicle motion detection means for detecting a situation, and based on the detection result, obtain blind spot information indicating a blind spot of a passenger riding on the vehicle, and based on the blind spot information, a virtual viewpoint in a three-dimensional space Virtual viewpoint setting means for setting, and viewpoint conversion means for creating a virtual viewpoint image that is an image viewed from the virtual viewpoint in the three-dimensional space with reference to the spatial data mapped by the space reconstruction means, This can be achieved by providing an image generation apparatus comprising display control means for controlling a display form of the virtual viewpoint image.

By comprising in this way, the virtual viewpoint image of the part used as a blind spot of a driver | operator can be displayed according to the operation | movement condition of a vehicle.
According to the second aspect of the present invention, the display control means has the blind spot in a virtual viewpoint image including the spot that is the blind spot and an image around the spot. It can achieve by providing the image generation device according to claim 1, wherein the display is controlled so that the location can be distinguished from other locations.

With this configuration, the virtual viewpoint image can be displayed so that the blind spot area and the area around the blind spot area can be distinguished.
According to the third aspect of the present invention, the display control means displays the virtual viewpoint image so that the portion that becomes the blind spot is divided by color development different from other portions. It can achieve by providing the image generation apparatus of Claim 1 characterized by the above-mentioned.

With this configuration, the virtual viewpoint image can be displayed so that the blind spot area and the area around the blind spot area can be distinguished.
According to the fourth aspect of the present invention, the virtual viewpoint setting means obtains, as the blind spot information, information related to the generation tendency of the blind spot that changes according to the operation on the vehicle. The image generating apparatus according to claim 1, wherein the image generating apparatus is configured to adaptively set a virtual viewpoint in the three-dimensional space so as to conform to the generation tendency of the blind spot. Can be achieved.

By configuring in this way, it is possible to display a virtual viewpoint image according to the generation tendency of the blind spot that changes according to the operation.
According to the invention described in claim 5, the above object is achieved by a space reconstruction unit that maps an input image from one or more cameras attached to a vehicle to a space model, and the space reconstruction. A viewpoint conversion unit that creates a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the unit, a display unit that displays the virtual viewpoint image, and the virtual The virtual viewpoint image corresponding to a landscape that cannot be seen as a blind spot on the display means that is installed in a part of the vehicle that causes a blind spot of a passenger by controlling the display form of the viewpoint image This can be achieved by providing an image generation device comprising display control means for displaying the image.

By comprising in this way, the virtual viewpoint image equivalent to the scenery when there is no obstacle in the obstacle which becomes the driver's blind spot can be displayed on the display device provided on the obstacle surface. .
According to the invention described in claim 6, the above object is achieved by a space reconstruction unit that maps an input image from one or more cameras attached to a vehicle to a space model, and the space reconstruction. A viewpoint conversion unit that creates a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the unit, a display unit that displays the virtual viewpoint image, and the virtual The virtual view image of the virtual viewpoint in a direction virtually reflected by the display means when the rider views the display means by controlling the table form of the viewpoint image, and the rider And a display means for displaying the virtual viewpoint image that is satisfied so as not to cause an unsightly portion as a blind spot of the image. It can be.

By comprising in this way, a display means can be used like a room mirror, and also the scenery of the other side originally interrupted by the part can be displayed on the part used as a blind spot on the part.
According to the seventh aspect of the present invention, the above-mentioned problem is satisfied so that the virtual viewpoint image displayed by the display control means does not have a blind spot and cannot be seen. The image generation apparatus according to claim 4, wherein the image generation apparatus according to claim 4 is displayed by highlighting a region corresponding to the portion.

By configuring in this way, it is possible to distinguish a virtual viewpoint image of a landscape that is originally a blind spot and cannot be viewed from a virtual viewpoint image of a landscape that is not.
According to the invention described in claim 8, the display control means distorts the virtual visual field image and displays the virtual visual field image on the display means with a wide visual field. This can be achieved by providing the image generating device described in item 4.

With this configuration, a convex mirror effect can be provided.
According to the ninth aspect of the present invention, there is provided a spatial reconstruction process for mapping an input image from one or a plurality of cameras attached to a vehicle to a spatial model, and an operation of the vehicle. Based on the vehicle motion detection process for detecting the situation and the detection result, the blind spot information indicating the blind spot of the rider who gets on the vehicle is acquired, and the virtual viewpoint in the three-dimensional space is acquired based on the blind spot information. A viewpoint conversion process for creating a virtual viewpoint image that is an image viewed from the virtual viewpoint in the three-dimensional space with reference to the spatial data mapped by the spatial reconstruction process; This can be achieved by providing an image generation program for causing a computer to execute display processing for displaying the virtual viewpoint image.

By comprising in this way, the virtual viewpoint image of the part used as a blind spot of a driver | operator can be displayed according to the operation | movement condition of a vehicle.
According to the invention described in claim 10, the above-described problem is a spatial reconstruction process for mapping an input image from one or more cameras attached to a vehicle to a spatial model, and the spatial reconstruction A viewpoint conversion process for creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the process, and a display form of the virtual viewpoint image, Display control processing for causing the computer to display the virtual viewpoint image corresponding to a landscape that cannot be seen as a blind spot on a display device that is installed in a portion of the vehicle that causes a blind spot of a passenger is executed on a computer This can be achieved by providing an image generation program.

By comprising in this way, the virtual viewpoint image equivalent to the scenery when there is no obstacle in the obstacle which becomes the driver's blind spot can be displayed on the display device provided on the obstacle surface. .
According to the eleventh aspect of the present invention, there is provided a spatial reconstruction process for mapping an input image from one or a plurality of cameras attached to a vehicle to a spatial model, and the spatial reconstruction. A viewpoint conversion process for creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the process, and a table form of the virtual viewpoint image to control a passenger When viewing the display device, the virtual visual field image of the virtual viewpoint in the direction virtually reflected by the display device and the blind spot of the occupant does not cause a portion that cannot be seen. And a display control process for displaying the virtual viewpoint image satisfied by the image generation program.

By comprising in this way, a display means can be used like a room mirror, and also the scenery of the other side originally interrupted by the part can be displayed on the part used as a blind spot on the part.
According to the invention described in claim 12, the above-described problem is a spatial reconstruction step of mapping an input image from one or a plurality of cameras attached to a vehicle to a spatial model, and the operation of the vehicle A vehicle motion detection step for detecting a situation, and based on the result of the detection, obtain blind spot information indicating a blind spot of a rider riding in the vehicle, and based on the blind spot information, a virtual viewpoint in a three-dimensional space A viewpoint conversion step for creating a virtual viewpoint image that is an image viewed from the virtual viewpoint in the three-dimensional space with reference to the spatial data mapped by the space reconstruction step; This can be achieved by providing an image generation method characterized by performing a display step of displaying the virtual viewpoint image.

By comprising in this way, the virtual viewpoint image of the part used as a blind spot of a driver | operator can be displayed according to the operation | movement condition of a vehicle.
According to the invention described in claim 13, the above-described problem is a space reconstruction step of mapping an input image from one or more cameras attached to a vehicle to a space model, and the space reconstruction A viewpoint conversion step of creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the process, and a display form of the virtual viewpoint image, A display control step of displaying the virtual viewpoint image corresponding to a landscape that cannot be seen as a blind spot on a display device that is installed in a part of the vehicle that causes a blind spot of a passenger. This can be achieved by providing a featured image generation method.

By comprising in this way, the virtual viewpoint image equivalent to the scenery when there is no obstacle in the obstacle which becomes the driver's blind spot can be displayed on the display device provided on the obstacle surface. .
According to the invention described in claim 14, the above object is a space reconstruction step of mapping an input image from one or more cameras attached to a vehicle to a space model, and the space reconstruction. A viewpoint conversion process for creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped in the process, and a table form of the virtual viewpoint image is controlled to get on When the person looks at the display device, the virtual visual field image of the virtual viewpoint in the direction virtually reflected by the display device and the blind spot of the occupant does not cause an unobservable part. And a display control step for displaying the virtual viewpoint image that has been satisfied in this manner.

By comprising in this way, a display means can be used like a room mirror, and also the scenery of the other side originally interrupted by the part can be displayed on the part used as a blind spot on the part.
According to the invention described in claim 15 of the claim, the object of the present invention is that the space reconstruction means for mapping the input image from one or more cameras attached to the imaging means arranged object to the space model, Based on the result of the detection, the imaging means placement object motion detection means for detecting the motion state of the imaging means placement object, and obtaining blind spot information indicating a spot that is a blind spot of the observer operating the imaging means placement object, A virtual viewpoint setting unit that sets a virtual viewpoint in a three-dimensional space based on the blind spot information, and an image viewed from the virtual viewpoint in the three-dimensional space with reference to the spatial data mapped by the space reconstruction unit An image generation apparatus comprising: viewpoint conversion means for creating a virtual viewpoint image; and display control means for controlling a display form of the virtual viewpoint image. It can be achieved by subjecting.

By configuring in this way, it is possible to display a virtual viewpoint image of a portion that becomes a blind spot of the user according to the operation state of the imaging means arranged object.
According to the invention described in claim 16, the display control means has the blind spot in a virtual viewpoint image including the spot that is the blind spot and an image around the spot. It can achieve by providing the image generation device according to claim 15, wherein the display is controlled so that the location can be distinguished from other locations.

With this configuration, the virtual viewpoint image can be displayed so that the blind spot area and the area around the blind spot area can be distinguished.
According to the invention described in claim 17, the display control means displays the virtual viewpoint image so as to classify the spot that becomes the blind spot by color development different from other parts. It can achieve by providing the image generation apparatus of Claim 15 characterized by the above-mentioned.

With this configuration, the virtual viewpoint image can be displayed so that the blind spot area and the area around the blind spot area can be distinguished.
According to the invention described in claim 18, the virtual viewpoint setting means, as the blind spot information, is information relating to the generation tendency of the blind spot that changes in accordance with an operation on the imaging means arranged object. The image generation apparatus according to claim 1, wherein the image generation apparatus is configured to adaptively set a virtual viewpoint in the three-dimensional space so as to match the generation tendency of the blind spot. Can be achieved by providing.

With this configuration, it is possible to display a virtual viewpoint image according to the generation tendency of the blind spot that changes according to the operation.
According to the invention described in claim 19, the above object is a spatial reconstruction unit that maps an input image from one or more cameras attached to an imaging unit arranged object to a spatial model, and Viewpoint conversion means for creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the space reconstruction means, and display means for displaying the virtual viewpoint image , By controlling the display form of the virtual viewpoint image, the display means installed in a part that is in the imaging means arranged object and causes the blind spot of the observer, to the landscape that cannot be seen as the blind spot This can be achieved by providing an image generation apparatus comprising display control means for displaying the corresponding virtual viewpoint image.

By comprising in this way, the virtual viewpoint image equivalent to the scenery when the obstruction which becomes a blind spot of a user does not have the obstruction can be displayed on the display device provided on the obstruction surface.
According to the invention described in claim 20, the above object is a space reconstruction unit that maps input images from one or more cameras attached to an imaging unit arranged object to a space model, and Viewpoint conversion means for creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the space reconstruction means, and display means for displaying the virtual viewpoint image The virtual view image of the virtual viewpoint in a direction virtually reflected by the display means when an observer views the display means by controlling a table form of the virtual viewpoint image, and Provided is an image generation apparatus comprising: display means for displaying the virtual viewpoint image that is satisfied so as not to cause a blind spot of the observer and an unobservable part. It can be achieved by the.

By comprising in this way, a display means can be used like a room mirror, and also the scenery of the other side originally interrupted by the part can be displayed on the part used as a blind spot on the part.
According to the invention described in claim 21, the above-mentioned problem is satisfied so that the virtual viewpoint image displayed by the display control means does not have an unsightly part as the blind spot. 21. The image generation apparatus according to claim 20, wherein the image generation apparatus according to claim 20 is displayed by highlighting a region corresponding to the portion.

By configuring in this way, it is possible to distinguish a virtual viewpoint image of a landscape that is originally a blind spot and cannot be viewed from a virtual viewpoint image of a landscape that is not.
According to the invention described in claim 22, the display control means distorts the virtual visual field image and displays the virtual visual field image on the display means with a wide visual field. This can be achieved by providing the image generating device according to 20.

With this configuration, a convex mirror effect can be provided.
According to the invention described in claim 23 of the scope of the invention, according to the invention described in claim 23, the spatial reconstruction processing for mapping the input images from one or a plurality of cameras attached to the imaging means arranged object to the spatial model, Based on the result of the detection, an imaging means placement object motion detection process that detects the motion status of the imaging means placement object, and obtains blind spot information indicating a blind spot of an observer who operates the imaging means placement object, A virtual viewpoint setting process for setting a virtual viewpoint in the three-dimensional space based on the blind spot information and an image viewed from the virtual viewpoint in the three-dimensional space with reference to the spatial data mapped by the spatial reconstruction process An image generation program for causing a computer to execute viewpoint conversion processing for creating a certain virtual viewpoint image and display processing for displaying the virtual viewpoint image. It can be achieved by subjecting.

By configuring in this way, it is possible to display a virtual viewpoint image of a portion that becomes a blind spot of the user according to the operation state of the imaging means arranged object.
According to the invention described in claim 24 of the scope of the invention, according to the invention described in claim 24, the spatial reconstruction processing for mapping the input images from one or a plurality of cameras attached to the imaging means arranged object to the spatial model, Controls the viewpoint conversion process for creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the spatial reconstruction process, and the display form of the virtual viewpoint image Display control processing for displaying the virtual viewpoint image corresponding to a landscape that cannot be seen as the blind spot on a display device that is installed in the imaging device-arranged object and that generates a blind spot of the observer Can be achieved by providing an image generation program for causing a computer to execute.

By comprising in this way, the virtual viewpoint image equivalent to the scenery when the obstruction which becomes a blind spot of a user does not have the obstruction can be displayed on the display device provided on the obstruction surface.
According to the invention described in claim 25 of the scope of the invention, according to the invention described in claim 25, the spatial reconstruction processing for mapping the input images from one or a plurality of cameras attached to the imaging means arranged object to the spatial model, By referring to the spatial data mapped by the spatial reconstruction process, a viewpoint conversion process for creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space, and a table form of the virtual viewpoint image are controlled. When the observer views the display device, the virtual visual field image of the virtual viewpoint in the direction virtually reflected by the display device, and a portion that cannot be seen as a blind spot of the observer This can be achieved by providing a display control process for displaying the virtual viewpoint image that is satisfied so as not to occur, and an image generation program for causing the computer to execute the display control process.

By comprising in this way, a display means can be used like a room mirror, and also the scenery of the other side originally interrupted by the part can be displayed on the part used as a blind spot on the part.
According to the invention described in claim 26 of the scope of claims, the spatial reconstruction step of mapping an input image from one or a plurality of cameras attached to an imaging means arranged object to a spatial model, An imaging means placement object motion detection step for detecting the motion status of the imaging means placement object, and based on the result of the detection, obtains blind spot information indicating a spot that is a blind spot of an observer who operates the imaging means placement object, An image viewed from the virtual viewpoint in the three-dimensional space with reference to the virtual viewpoint setting step of setting a virtual viewpoint in the three-dimensional space based on the blind spot information and the spatial data mapped by the space reconstruction step By providing an image generation method characterized by performing a viewpoint conversion step of creating a virtual viewpoint image and a display step of displaying the virtual viewpoint image. It can be achieved Te.

By configuring in this way, it is possible to display a virtual viewpoint image of a portion that becomes a blind spot of the user according to the operation state of the imaging means arranged object.
According to the invention described in claim 27 of the scope of claims, the spatial reconstruction step of mapping input images from one or a plurality of cameras attached to an imaging means arranged object onto a spatial model, A viewpoint conversion process for creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the spatial reconstruction process, and a display form of the virtual viewpoint image is controlled. And a display control step of displaying the virtual viewpoint image corresponding to a landscape that cannot be seen as the blind spot on a display device that is installed in the imaging means placement object and that causes the blind spot of the observer. By providing an image generation method characterized in that

By comprising in this way, the virtual viewpoint image equivalent to the scenery when the obstruction which becomes a blind spot of a user does not have the obstruction can be displayed on the display device provided on the obstruction surface.
According to the invention described in claim 28 of the claim, the object is a spatial reconstruction step of mapping an input image from one or more cameras attached to an imaging means arranged object to a spatial model, A viewpoint conversion process for creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the spatial reconstruction process, and a table form of the virtual viewpoint image is controlled. When the observer views the display device, the virtual visual field image of the virtual viewpoint in the direction virtually reflected by the display device, and a portion that cannot be seen as a blind spot of the observer And a display control step for displaying the virtual viewpoint image that is satisfied so as not to occur. This can be achieved by providing an image generation method.

  By comprising in this way, a display means can be used like a room mirror, and also the scenery of the other side originally interrupted by the part can be displayed on the part used as a blind spot on the part.

  ADVANTAGE OF THE INVENTION According to this invention, the technique for improving the convenience further about the user interface which concerns on the display of a virtual viewpoint image can be embodied.

<First Embodiment>
In the present embodiment, a blind spot area that is a blind spot for the driver is detected, a viewpoint for observing the blind spot area is set, and a virtual viewpoint image from such a viewpoint is displayed so as to be visible from the driver. Alternatively, from such a blind spot area, a blind spot area to be displayed is selected by driving operation information or a driver's operation, a viewpoint for observing the selected blind spot area is set, and a virtual viewpoint image from such a viewpoint is set. Is displayed to the driver in a visible manner. Hereinafter, such embodiments will be specifically described sequentially with reference to the drawings.

  FIG. 1 shows an image generation apparatus 10000 in the present embodiment. In the figure, an image generation apparatus 10000 includes one or more cameras 101, a camera parameter table 103, a spatial reconstruction unit 104, a spatial data buffer 105, a viewpoint conversion unit 106, a display unit 107, a display control unit 10001, a virtual It comprises a viewpoint setting means 10002 and vehicle motion detection means 10003.

  The plurality of cameras 101 are provided in a state suitable for grasping the status of the monitoring target area. The camera 101 is, for example, a plurality of television cameras that capture images of a space to be monitored such as a situation around the vehicle. In general, it is preferable to use a camera 101 having a large angle of view so that a large field of view can be obtained. The number of cameras 101 installed, the installation state, and the like may be a known mode as disclosed in Patent Document 1, for example. In the illustrated example, a plurality of cameras are used. However, it may be used so as to obtain the same imaging data as when a plurality of cameras are provided by sequentially moving the installation position of one camera. . This is the same for each example described below.

  The camera parameter table 103 stores camera parameters indicating the characteristics of the camera 101. Here, the camera parameters will be described. The image generation apparatus 10000 is provided with calibration means (not shown), and performs camera calibration. Camera calibration refers to a camera placed in a three-dimensional space, such as the camera mounting position, camera mounting angle, camera lens distortion correction value, camera lens focal length, etc., in the three-dimensional space. 101 is to determine and correct the camera parameters representing the characteristics of 101. The calibration means and the camera parameter table 103 are also described in detail in, for example, Patent Document 1.

  The space reconstruction unit 104 creates spatial data in which an input image from the camera 101 is mapped to a three-dimensional space model based on the camera parameters. That is, the space reconstruction unit 104 associates each pixel constituting the input image from the camera 101 with a point in the three-dimensional space based on the camera parameter calculated by the calibration unit (not shown). Create

  That is, the space reconstruction unit 104 calculates where each object included in the image photographed from the camera 101 exists in the three-dimensional space, and stores the spatial data as the calculation result in the spatial data buffer 105. To do. The spatial model may be a predetermined (predetermined) one, or one generated from a plurality of input images on the spot, or one generated based on the output of a separate sensor. May be.

  For example, as described in Patent Document 1, a space model is a space model composed of five planes, a bowl-shaped space model, a space model composed of a combination of a plane and a curved surface, or a space model introduced with a surface. Or a combination thereof. In addition, it is not limited to these space models, A space model will not be specifically limited if it is a combination of a plane, a curved surface, or a combination of a plane and a curved surface. Moreover, you may create a spatial model based on the stereo vision image obtained from the stereo sensor etc. (for example, patent document 2, patent document 3) which acquire the distance image which calculates a distance image by a triangulation measurement.

  Note that it is not necessary to configure the spatial data using all of the respective pixels constituting the input image from the camera 101. For example, if the input image includes a region located above the horizontal line, it is not necessary to map the pixels included in the region above the horizontal line to the road surface. Alternatively, it is not necessary to map the pixels representing the vehicle body. In addition, when the input image has a high resolution, it may be possible to increase the processing speed by skipping every few pixels and mapping it to spatial data. The space reconstruction unit 104 is described in detail in, for example, Patent Document 1.

The spatial data buffer 105 temporarily stores the spatial data created by the spatial reconstruction unit 104. The spatial data buffer 105 is also described in detail in Patent Document 1, for example.
The viewpoint conversion means 106 creates an image viewed from an arbitrary viewpoint with reference to the spatial data. That is, referring to the spatial data created by the spatial reconstruction means 104, an image is created by setting the camera at an arbitrary viewpoint. The viewpoint conversion means 106 can also have the configuration detailed in, for example, Patent Document 1.

  The vehicle operation detection means 10003 detects the operation status of the vehicle. For example, whether the vehicle is turning right or left is detected by the steering angle of the steering wheel or whether the brake is applied. Thus, in order to detect the state of the vehicle, sensors and instruments are attached to each part of the vehicle.

The virtual viewpoint setting unit 10002 sets parameters regarding the virtual viewpoint to be transmitted to the viewpoint conversion unit 106. Here, the parameter can be set in accordance with the vehicle state detected by the vehicle motion detection means 10003.
The display control unit 10001 controls the display form in order to display the virtual viewpoint image generated by the viewpoint conversion unit 106 on the display unit 107 (for example, a display).

  FIG. 2 shows a display flow of the virtual viewpoint image in the present embodiment. First, the space reconstruction unit 104 calculates the correspondence between each pixel constituting the image obtained from the camera 101 and a point in the three-dimensional coordinate system, and creates space data (S1). This calculation is performed for all pixels of the image obtained from each camera 101. For this processing itself, for example, a known mode disclosed in Patent Document 1 can be applied.

Next, as described above, after the vehicle operation state is detected by the vehicle operation detection unit 10003 such as various sensors (S2), the virtual viewpoint setting unit 10002 displays the vehicle state detected by the vehicle operation detection unit 10003. Accordingly, a virtual viewpoint is set (S3).
Next, the viewpoint conversion means 106 reproduces the image from the viewpoint specified in S3 from the spatial data (S4). For this processing itself, for example, a known mode disclosed in Patent Document 1 can be applied. Thereafter, the display form of the reproduced image is controlled by the display control unit 10001 (S5). The process of S5 will be described later. Thereafter, the controlled image is output to the display means 107 and displayed on the display means 107 (S6).

  FIG. 3 shows an example of detecting the driver's blind spot based on the driving operation of the driver in the present embodiment. The figure shows a blind spot 10011 detected when the vehicle 10010 turns right. When the vehicle 10010 makes a right turn, it is necessary to check the right front tire in order to prevent an accident involving the inner ring.

  However, in this case, the field of view is blocked by the right front door, the bonnet, and the instrument panel, and the inner ring difference portion becomes a blind spot. Moreover, such a part is a part which becomes a blind spot even if it passes through a side mirror. Therefore, in the present embodiment, such a blind spot is detected based on the driving operation of the driver.

  In the case of the figure, the driver makes a right turn (or a left turn) by rotating the steering wheel of the vehicle. This right turn (or left turn) motion is detected by the vehicle motion detection means 10003 (S2 in FIG. 2). At this time, the degree of rotation of the steering wheel is detected, that is, whether it is clockwise or counterclockwise, how much angle, how fast, and how much acceleration is turned right (or left). This detected information is transmitted to the virtual viewpoint setting unit 10002. The virtual viewpoint setting unit 10002 recognizes the driving operation of the driver based on the detected information, and specifies the portion 10011 that becomes the blind spot of the driver.

  The position of the driver's viewpoint is acquired in advance. For example, a driver's face image is acquired by a camera that monitors the interior of the vehicle, and the position of the viewpoint is acquired by measuring the viewpoint by extracting the eyeball position from the face image using a general image processing technique. good. Alternatively, the user's viewpoint may be calculated based on the driver's posture and the like.

  For example, since the position of the driver's head can be measured from the height (or sitting height) of the driver registered in advance and the current inclination angle of the seat, the approximate position of the viewpoint of the driver can be known therefrom. In addition, since there is an upper limit on the position of the human viewpoint when driving (restriction of the vehicle height from the seat to the roof), if it is assumed that there is not much difference in the position of the viewpoint for each person, a predetermined value (average value, Statistical values etc. may be set as default. In this way, a virtual viewpoint is obtained.

  Then, by using CAD (Computer Aided Design) data, a blind spot area from the driver's viewpoint is obtained from the acquired driver's viewpoint position and the CAD data of the vehicle. The virtual viewpoint information (including information indicating which direction it is facing) acquired in step (1) is transmitted to the viewpoint conversion means 106.

  Based on the received information, the viewpoint conversion means 106 generates a virtual viewpoint image viewed from the virtual viewpoint (S4 in FIG. 2). At this time, a virtual viewpoint image including a blind spot area and an area around the blind spot area is generated (a virtual viewpoint image of only the blind spot area may be generated depending on the application).

  The virtual viewpoint image generated here is composed of the blind spot area and the area around the blind spot area as described above, and is displayed in such a manner that the blind spot area is distinguished from the area around the blind spot area. Like that. For example, the blind spot area and the area around the blind spot area may be distinguished by different colors, or the blind spot area may be highlighted.

  Moreover, you may make it switch the display form of the blind spot image displayed based on the blind spot information detected by the vehicle operation | movement detection means. That is, information on the generation tendency of the blind spot that changes according to the detected information is obtained, and the virtual viewpoint in the three-dimensional space is adaptively set so as to match the generation tendency of the blind spot. For example, as shown in FIG. 4, preset right turn mode, left turn mode, and the like may be switched and applied according to the detected blind spot information.

  FIG. 4 shows an example of the mode of operation status of the vehicle in the present embodiment. The operation status modes of this embodiment include “right turn mode”, “left turn mode”, “starting ambient monitoring mode”, “in-vehicle monitoring mode”, “high speed running mode”, “rear monitoring mode”, “rainy weather” There are “travel mode”, “parallel parking mode”, and “garage entry mode”. Now, each mode will be described.

  “Right turn mode” displays an image of the direction of turning forward. Specifically, when the vehicle turns right, a forward image and a right turn image are displayed. The “left turn mode” displays an image in a direction of turning forward. Specifically, when the vehicle makes a left turn, a front image and a left turn image are displayed.

  In the “starting-up ambient monitoring mode”, a monitoring video around the vehicle is displayed when the vehicle starts up. “In-vehicle monitoring mode” displays an in-vehicle monitoring image. The “high speed running mode” displays an image of a far front in high speed running. In the “rear monitoring mode”, a rear image for confirming whether or not sudden braking can be applied, that is, whether there is a distance from the succeeding vehicle that can be stopped by applying sudden braking is displayed.

  In the “rainy driving mode”, the visibility is deteriorated when it rains, and a direction that is easily overlooked may occur. Therefore, an image in such a direction that is easily overlooked and / or an image from which water droplets are removed by image processing should be displayed. . The direction that is easily overlooked may be a direction obtained statistically or empirically, or may be arbitrarily set by the user.

In the “Parallel parking mode”, the front and rear images on the side to be widened are displayed so as not to contact the front vehicle and the rear vehicle during the parallel parking. In the “garage entry mode”, an image is displayed in a direction in which it is easy to contact the wall of the garage when entering the garage.
As described above, the mode is determined from the detected operation state of the vehicle, and the virtual viewpoint image is displayed in a display form corresponding to this mode.

  As described above, a virtual viewpoint image of a portion that becomes a blind spot of the driver can be provided to the driver. In FIG. 3, the portion where the inner wheel difference at the time of the right turn is determined as the blind angle, but is not limited thereto. For example, the blind angle may be the outer wheel difference portion or the rear portion of the vehicle during reverse travel. It may be any other blind spot that occurs during driving operation.

  In addition, in order to detect such blind spots, various parts of the vehicle have various sensors (infrared, temperature, humidity, pressure, illuminance, sensors that detect mechanical movements) and cameras (for in-vehicle use, car body photography). A measuring instrument is attached (or an existing measuring instrument such as a tachometer, a speedometer, a coolant temperature gauge, a hydraulic pressure gauge, a fuel fuel gauge, etc. may be used).

  In addition to the above method, for example, the invention described in Patent Document 1 may be used as a method for acquiring information representing blind spots. That is, the blind spot of the driver is obtained by subtracting the virtual viewpoint image based on the virtual viewpoint from the viewpoint of the driver sitting on the driver's seat from the virtual viewpoint image based on the virtual viewpoint from above the vehicle. In this way, it is also possible to acquire information representing the driver's blind spot.

  In the present embodiment, the blind spot 10011 in FIG. 3 has been described as an example, but there is a blind spot in an exception when turning right. Therefore, when there are a plurality of blind spots in a predetermined operation situation of the vehicle, it is possible to select which blind spot portion of the virtual viewpoint image is to be displayed. Further, the selected information can be stored as history information in a storage device in the image generation apparatus. Since the user's selection frequency can be obtained from the history information, a virtual viewpoint image for the blind spot portion can be automatically displayed based on the selection frequency. Further, the virtual viewpoint image to be displayed can be set in advance in relation to a predetermined operation state of the vehicle.

  As described above, since the virtual viewpoint image of the part that becomes a blind spot can be displayed corresponding to the operation state of the vehicle, the vehicle can be safely driven while checking the blind spot during the operation. In addition, as understood from the above description, in the present specification, the “blind spot” is an area that cannot be seen from the driver's seat in any way (even if the neck is turned or a mirror is used). In addition, as another intention, it is required (set) as a “blind spot” depending on the situation or extracted (selected) from a plurality of “blind spots” depending on the driving situation. Have a will.

<Second Embodiment>
In the present embodiment, a virtual viewpoint image corresponding to a landscape when the obstacle that is a blind spot of the driver is not present is displayed on a display device provided on the obstacle surface. Thereby, in this embodiment, a more intuitive display is realized by optimizing the installation position of the image display device.

  FIG. 5 is a diagram for explaining an example of the case where the image generation apparatus according to the present embodiment is used. From the inside of the vehicle (driver's seat) to the front pillar 10021 (the pillar between the door and the roof that supports the strength of the vehicle. And shows another vehicle 10027 outside the vehicle (referring to a pillar 10021 between the windshield 10020 and the side window 10022).

FIG. 5A shows a case where the image generation apparatus according to the present embodiment is not used, and the vehicle body of the vehicle 10025 is blocked by the front pillar 10021 (that is, a blind spot) and is blocked by the driver. I can't see the part.
FIG. 5B shows a case where the image generation apparatus according to the present embodiment is used, and an image can be displayed on the surface of the front pillar 10021a (for example, a liquid crystal display, a plasma display, Alternatively, a flat panel display such as an organic EL display or an electronic paper display is attached to the front pillar 10021.)

  In this way, an image can be displayed on the front pillar 10021a. In the present embodiment, a virtual viewpoint image of an external landscape that is originally blocked by the front pillar 10021a (ie, a blind spot) and cannot be seen by the driver is displayed on the front pillar 10021a. . In FIG. 5B, a virtual viewpoint image corresponding to the blocked portion of the vehicle 10025 is displayed on the front pillar 10021.

  FIG. 6 shows a display flow of the virtual viewpoint image in the present embodiment. First, the space reconstruction unit 104 calculates the correspondence between each pixel constituting the image obtained from the camera 101 and a point in the three-dimensional coordinate system, and creates space data (S11). This process is the same as S1 in FIG.

  Next, a virtual viewpoint for generating a virtual viewpoint image is designated (S12). The virtual viewpoint is a viewpoint direction toward each part that becomes a blind spot in the vehicle with the driver's viewpoint as the center. These may be fixed and set in advance, or may be set every time the passenger gets on.

Next, the viewpoint conversion means 106 generates a virtual viewpoint image viewed from the virtual viewpoint specified in S12 (S13). This process is the same as S4 in FIG. At this time, a virtual viewpoint image about the landscape is created except for information on the vehicle.
Next, the display control means 10001 extracts an image part corresponding to the landscape blocked by the part that becomes a blind spot from the virtual viewpoint image generated in S13 (S14). In FIG. 5, an image of only a portion to be displayed on the front pillar 10021a is extracted from the generated virtual viewpoint image. At this time, since the size, position, shape, and the like of the obstacle used as the display means are registered in advance in the storage device in the image generation device 10000, based on such information, the portion to be extracted from the virtual viewpoint image Extract images.

  In S14, the following processing can be considered in addition to the above extraction processing. For example, a virtual viewpoint image generated without considering information related to the host vehicle (that is, a virtual viewpoint image generated considering parameters related to the host vehicle) and a virtual viewpoint image generated considering information related to the host vehicle are calculated. Then, a blind spot area is obtained by obtaining a difference between the two virtual viewpoint images. Therefore, the image of the difference area is an image displayed on the display means.

Next, the extracted image is displayed on the display device 107. In FIG. 5, the extracted image is displayed on the front pillar 10021a (S15).
From the above, it is possible to display the scenery on the other side that was originally blocked by the part in the blind spot, so that it looks as if the part is see-through. In addition, as the blind spot portion to be displayed, the front pillar is used as an example in the above, but is not limited thereto. For example, the headrest, instrument panel, seat, etc. ) Can be attached to a display device (for example, a liquid crystal display, a plasma display, an organic EL display, an electronic paper display, etc.).

<Third Embodiment>
In the present embodiment, assuming that the display means is a room mirror, a virtual viewpoint image corresponding to an image displayed if it is reflected by the room mirror is displayed. At this time, similarly to the second embodiment, the display means displays the scenery on the other side, which is originally blocked by the part, in the part that becomes a blind spot. Thereby, a more intuitive display is realized by optimizing the installation position of the image display device.

  FIG. 7 shows an image generation device 10000 in the present embodiment. In the figure, an image generation apparatus 10000 includes a plurality of cameras 101, a camera parameter table 103, a spatial reconstruction unit 104, a spatial data buffer 105, a viewpoint conversion unit 106, a display unit 107, a display control unit 10001, and a viewpoint detection unit 10030. Consists of Except for the viewpoint detection means 10030, it is the same as FIG.

  As described above, in this embodiment, the display means 107 can be used like a mirror. In other words, it is necessary for the display means that is regarded as a mirror to display an image as if it is an image reflected by the mirror when the display means is viewed by the driver.

  Therefore, it is necessary to determine the relative position of the driver's viewpoint with respect to the installation position of the display means 107. Therefore, in the present embodiment, the viewpoint detection unit 10030 is used to detect the relative position of the driver's viewpoint with respect to the installation position of the display unit 107. As in the first embodiment, the viewpoint detection unit 10030 obtains the driver's face image with a camera that monitors the interior of the vehicle, for example, and extracts the eyeball position from the face image using a general image processing technique. The position of the viewpoint may be acquired by measuring the viewpoint with the above, or the user's viewpoint may be calculated from the driver's attitude by estimating the position of the virtual viewpoint. May be set.

  Thereby, the position of the driver's viewpoint and the direction of the line of sight can be detected. In addition, an installation angle of the display unit 107 with respect to the vehicle is set in advance. Therefore, when the driver views the display means 107 from these pieces of information, the incident angle of the driver's line of sight with respect to the display surface of the display display 107 is calculated, and as a result, the reflection angle is obtained.

  Therefore, the image displayed on the display means 107 displays a virtual viewpoint image of the landscape in the direction reflected at the reflection angle. Here, the generated virtual viewpoint image is an image generated without taking into consideration information related to the own vehicle (vehicle-mounted article, seat, front pillar, rear pillar, etc.), and is horizontally reversed when displayed on the display means. .

  At this time, for example, a landscape that is a blind spot and cannot be viewed may be displayed in a wire frame so that a virtual viewpoint image corresponding to the landscape that is originally a blind spot and a landscape that is not visible can be distinguished from a landscape that is not originally a blind spot. However, it may be distinguished by a color different from other parts, or may be highlighted.

  As described above, CAD data is used in the same manner as in the first embodiment to distinguish between a virtual viewpoint image of a landscape that is originally a blind spot and cannot be viewed, and a virtual viewpoint image of a landscape that is not. That is, information on the blind spot area from the driver's viewpoint is obtained, and a portion corresponding to the blind spot area is displayed, for example, in a wire frame.

  Further, as a method not using CAD data, a virtual viewpoint image generated without considering information about the host vehicle and a virtual viewpoint image generated considering information about the host vehicle are calculated, and the two virtual viewpoint images are calculated. A blind spot area can be obtained by calculating the difference. Then, based on the information on the blind spot area, the part corresponding to the scenery that cannot be seen as the blind spot is displayed with, for example, a wire frame.

  Further, the image displayed on the display means displays a virtual viewpoint image from which the blind spot portion is removed so that the blind spot portion is see-through (visible through). That is, the virtual viewpoint image corresponding to a landscape that is a blind spot and cannot be seen is displayed on a display unit that is installed in a part of the vehicle that causes the blind spot of the driver. An example of this is shown in FIG.

  FIG. 8 shows a display form of an image displayed on the display means in the present embodiment. FIG. 8A shows a landscape (following vehicle 10042) in the direction of the rear window 10040 displayed on a normal room mirror 10040. FIG. As shown in FIG. 8 (a), the reflection image displayed on the normal room mirror 10040 includes a passenger seat 10044, a rear seat 10045, and a rear window frame 10043. Since this is a blind spot for a person, the landscape beyond this blind spot (the lower part and the right front part of the following vehicle 10041 in FIG. 8A) cannot be seen.

  FIG. 8 (b) uses the present invention to display a landscape in the rear window direction on the display means 10046 so as to be displayed on a normal room mirror, and to become a driver's blind spot and cannot be seen. A state in which the corresponding virtual viewpoint image is displayed is shown. In FIG. 8 (b), the lower and right front portions of the following vehicle 10042 that could not be seen due to blind spots caused by the passenger seat 10044, the rear seat 10045, and the rear window frame 10043 shown in FIG. 8 (a). It is displayed so that the field of view is satisfied.

  Moreover, in order to show where it corresponds to a blind spot, it is a display form which can distinguish the part corresponding to the landscape which becomes a blind spot and cannot be seen, and the part which is not so. For example, as shown in FIG. 8B, this display form may display a blind spot with a wire frame or distinguish it with a color different from other parts. However, it may be highlighted.

  The display means used in this embodiment may have a structure in which a half mirror is attached to the surface of the display so that it can function as a normal room mirror. Further, the image displayed on the display means may be distorted to display a wide-field image. That is, you may make it give a convex mirror effect.

  Furthermore, the display means will be described in detail. The rearview mirror is composed of a half mirror, a flat panel display is arranged on the back of the mirror, and the viewpoint position of the driver is detected by the viewpoint position detection means, and the relationship between the viewpoint and the half mirror. From the back of the half mirror, a superimposed image for navigation may be displayed.

  Moreover, you may provide said display means (for example, a liquid crystal display, a plasma display, an organic electroluminescent display, an electronic paper display etc.) in a part of side window. Thereby, the state of the back can be confirmed by displaying the state of the back which has been confirmed by the door mirror on the display means.

  Moreover, since it can be displayed with an image larger than the image displayed on the side mirror, the state of the back can be confirmed in more detail. For this reason, a side mirror becomes unnecessary and space saving of a parking space can be achieved, for example. Moreover, there is no fear that the side mirror of the oncoming vehicle and the side mirror of the host vehicle come into contact with each other even when separated from the oncoming vehicle in a narrow passage.

  Further, as shown in FIG. 4, the display form displayed on the display means may be switched in accordance with the operation state mode of the vehicle. Further, for example, it may be possible to follow the viewpoint variation by using functions such as pan, tilt, and zoom. Further, the horizontal aspect may be reduced and displayed so that the horizontal direction can easily enter the visual field.

  As described above, the display can have the same effect as the room mirror, and further, a virtual viewpoint image corresponding to a landscape that is originally a blind spot and cannot be seen can be displayed. Further, the viewpoint to be displayed on the mirror can be calculated by the viewpoint detection means, and a natural mirror image (reflected image displayed on the mirror) can be displayed on this display including the interpolation display of the blind spot portion.

  In addition, when displaying the portion corresponding to the blind spot, the outline of the blind spot is displayed with a wire frame or the like, so that the driver can recognize that the image is actually not directly visible to the driver. Can do. In addition, this display has a viewing angle suitable for the driver, so that passengers do not see non-intuitive images. Further, the viewpoint from the driver's seat is set as the virtual viewpoint, and the virtual viewpoint linked to the driver's viewpoint is set.

<Fourth Embodiment>
In 1st-3rd embodiment, although demonstrated centering on a vehicle, it is applicable not only to a vehicle but to a broad field | area. Therefore, in the present embodiment, an example in which the present invention is applied to a device other than a vehicle is shown below.

  As an example in this embodiment, for example, the observer is a person walking indoors, on a street, etc., and the imaging device placement object is indoors, outdoors, room interiors, and other mobile objects (vehicles, robots). Such a monitoring system may be used. The present invention can also be applied to a blind spot check caused by the open / closed state of a door, a blind spot check caused by the state of a door of a household appliance or furniture, or a blind spot check behind a person.

  FIG. 9 shows an example in which the image generation apparatus according to the present embodiment is applied to an HMD (head mounted display). The hatched portions 10054, 10055, 10056 in the figure are blind spots. FIG. 9A shows a state in which the door 10052 and the refrigerator 10053 in the room 10050 are closed. FIG. 9B shows a state in which the doors 10052 and refrigerators 10053 in the room 10050 are opened.

  In this case, the person 10051 who is an observer wears an HMD or the like, for example, and can observe a virtual viewpoint image, and the observer's own position captured by the camera of the imaging device arrangement object (here, assumed to be a room 10050), The posture and direction are measured (this part can also be measured by a GPS, a gyro sensor, a camera device, a person's line-of-sight detection device, etc. that are attached to a person who is an observer).

  From these pieces of information, it is possible to calculate which part can be directly observed by a person and which part cannot be seen. As a result, the person can grasp his / her blind spot range, for example, blind spots 10054, 10055, 10056 on a virtual viewpoint image displayed on the HMD, etc. You can recognize manholes and grooves with open lids.

  In the first to fourth embodiments, the camera 101 is used to create a virtual viewpoint image. However, these cameras may have an AF (autofocus) function. As a result, the lens focus is adjusted to the short distance side when close to each other in stereo view. In other words, the mode is adjusted to a mode in which large shooting is generally performed at a position close to a shooting target called a macro mode. In this way, an image focused for three-dimensional reconstruction can be obtained at a short distance.

Further, when shooting a subject at a long distance, focusing on the subject by the AF function can similarly obtain a high-accuracy image for a long-distance image, leading to an improvement in the accuracy of the long distance.
FIG. 10 is a configuration block diagram of a hardware environment of the image generation device 10000 in the first to fourth embodiments. In the figure, an image generation device 10000 includes at least a control device 10080 such as a central processing unit (CPU) and a storage device 10081 such as a read only memory (ROM), a random access memory (RAM), or a mass storage device. And an output I / F (interface, the same applies hereinafter) 10082, an input I / F 10083, a communication I / F 10084, and a bus 10085 for connecting them, to which output means 107 such as a display, input I / F or communication There are various devices connected to the I / F.

As an apparatus connected to the input I / F, for example, a camera 101, an in-vehicle camera, a stereo sensor and various sensors, an input device such as a keyboard and a mouse, a reading device for a portable storage medium such as a CD-ROM and a DVD, Other peripheral devices are listed.
Examples of the device connected to the communication I / F 10084 include a car navigation system and a communication device connected to the Internet or GPS. Note that the communication medium may be a communication network such as the Internet, a LAN, a WAN, a dedicated line, a wired line, and a wireless line.

  As an example of the storage device 10081, various types of storage devices such as a hard disk and a magnetic disk can be used. The flow program described in the above embodiment, various tables (for example, a table storing various setting values, etc.) ), CAD data and the like are stored. This program is read by the control device 10080, and each process of the flow is executed.

  This program may be provided from the program provider via the communication I / F 10084 via the Internet and stored in the storage device 10081, or may be stored in a commercially available portable storage medium, It can also be set in the reader and executed by the controller. Various types of storage media such as CD-ROM, DVD, flexible disk, optical disk, magneto-optical disk, and IC card can be used as the portable storage medium, and the program stored in such a storage medium can be read by a reading device. Read by.

As the input device, a keyboard, a mouse, an electronic camera, a microphone, a scanner, a sensor, a tablet, or the like can be used. In addition, other peripheral devices can be connected.
In each of the above-described embodiments, the plurality of imaging devices may be used so as to configure a so-called trinocular stereo camera or may be used so as to configure a four-eye stereo camera. Thus, it is known that when a three-eye or four-eye stereo camera is used, a more reliable and stable processing result can be obtained in a three-dimensional reconstruction process or the like (for example, Non-Patent Document 1). In particular, it is known that when a plurality of cameras are arranged so as to have a baseline length in two directions, three-dimensional reconstruction can be performed in a more complicated scene. If a plurality of cameras are arranged in one baseline length direction, a so-called multi-baseline stereo camera can be realized, and more accurate stereo measurement can be performed.

It is a figure which shows the image generation apparatus 10000 in 1st Embodiment. It is a figure which shows the display flow of the virtual viewpoint image in 1st Embodiment. It is a figure which shows an example which detects a driver | operator's blind spot based on the driving | operation operation | movement of the driver | operator in 1st Embodiment. It is a figure which shows an example of the mode of the operating condition of the vehicle in 1st Embodiment. It is a figure for demonstrating an example at the time of using the image generation apparatus in 2nd Embodiment. It is a figure which shows the display flow of the virtual viewpoint image in 2nd Embodiment. It is a figure which shows the image generation apparatus 10000 in 3rd Embodiment. It is a figure which shows the display form of the image displayed on the display means in 3rd Embodiment. It is a figure which shows an example at the time of applying the image generation apparatus in 4th Embodiment to HMD (head mounted display). It is a block diagram of the hardware environment of the image generation apparatus in the first to fourth embodiments.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Camera 103 Camera parameter table 104 Spatial reconstruction means 105 Spatial data buffer 106 View point conversion means 107 Display means 10000 Image generation apparatus 10001 Display control means 10002 Virtual viewpoint setting means 10003 Vehicle motion detection means 10030 View point detection means 10080 Control apparatus 10081 Storage device 10082 Output I / F
10083 Input I / F
10084 Communication I / F
10085 Bus

Claims (28)

Spatial reconstruction means for mapping input images from one or more cameras attached to the vehicle to a spatial model;
Vehicle operation detection means for detecting the operation status of the vehicle;
Virtual viewpoint setting means for acquiring blind spot information indicating a blind spot of a rider riding in the vehicle based on the detection result, and setting a virtual viewpoint in a three-dimensional space based on the blind spot information;
Viewpoint conversion means for creating a virtual viewpoint image that is an image viewed from the virtual viewpoint in the three-dimensional space with reference to the spatial data mapped by the space reconstruction means;
Display control means for controlling the display form of the virtual viewpoint image;
An image generation apparatus comprising:   The display control means is configured to control display in a virtual viewpoint image including a location that becomes the blind spot and an image around the location so that the location that becomes the blind spot can be distinguished from other locations. The image generation apparatus according to claim 1, wherein the image generation apparatus is an image generation apparatus.   2. The display control unit according to claim 1, wherein the display control unit is configured to control the display of the virtual viewpoint image so as to classify the portion that becomes the blind spot by color development different from other portions. The image generating apparatus described.   The virtual viewpoint setting means obtains information regarding the blind spot occurrence tendency that changes according to the operation on the vehicle as the blind spot information, and adapts the virtual viewpoint in the three-dimensional space so as to match the blind spot occurrence tendency. The image generation apparatus according to claim 1, wherein the image generation apparatus is configured to be set automatically. Spatial reconstruction means for mapping input images from one or more cameras attached to the vehicle to a spatial model;
Viewpoint conversion means for creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the spatial reconstruction means;
Display means for displaying the virtual viewpoint image;
Controlling the display form of the virtual viewpoint image, the display means installed in a part of the vehicle that causes a blind spot of a rider, the virtual corresponding to a landscape that cannot be seen as the blind spot Display control means for displaying a viewpoint image;
An image generation apparatus comprising: Spatial reconstruction means for mapping input images from one or more cameras attached to the vehicle to a spatial model;
Viewpoint conversion means for creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the spatial reconstruction means;
Display means for displaying the virtual viewpoint image;
The virtual view image of the virtual viewpoint in a direction virtually reflected by the display means when a rider views the display means by controlling a table form of the virtual viewpoint image, and Display means for displaying the virtual viewpoint image that is satisfied so that a blind spot of the passenger does not occur and an unsightly part does not occur;
An image generation apparatus comprising:   5. The virtual viewpoint image displayed by the display control means highlights and displays a region corresponding to a portion that has been satisfied so that the blind spot does not cause an unviewable portion. The image generating apparatus described in 1.   The image generation apparatus according to claim 4, wherein the display control unit distorts the virtual visual field image and displays the virtual visual field image on the display unit with a wide field of view. A spatial reconstruction process for mapping input images from one or more cameras attached to the vehicle to a spatial model;
Vehicle operation detection processing for detecting the operation status of the vehicle;
A virtual viewpoint setting process for acquiring blind spot information indicating a blind spot of a rider who rides the vehicle based on the detection result, and setting a virtual viewpoint in a three-dimensional space based on the blind spot information;
A viewpoint conversion process for creating a virtual viewpoint image that is an image viewed from the virtual viewpoint in the three-dimensional space with reference to the spatial data mapped by the spatial reconstruction process;
Display processing for displaying the virtual viewpoint image;
Generation program for causing a computer to execute the program. A spatial reconstruction process for mapping input images from one or more cameras attached to the vehicle to a spatial model;
A viewpoint conversion process for creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the spatial reconstruction process;
The virtual viewpoint corresponding to a landscape that cannot be seen as a blind spot on a display device that is installed in a part of the vehicle that causes a blind spot of a passenger by controlling a display form of the virtual viewpoint image Display control processing for displaying images;
Generation program for causing a computer to execute the program. A spatial reconstruction process for mapping input images from one or more cameras attached to the vehicle to a spatial model;
Viewpoint conversion processing for creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the spatial reconstruction processing;
The virtual view image of the virtual viewpoint in a direction virtually reflected by the display device when the rider views the display device by controlling a table form of the virtual viewpoint image, and the ride Display control processing for displaying the virtual viewpoint image that is satisfied so as not to cause a blind spot of the person to be unsightly,
Generation program for causing a computer to execute the program. A spatial reconstruction step of mapping an input image from one or more cameras attached to the vehicle to a spatial model;
A vehicle operation detection step for detecting the operation status of the vehicle;
A virtual viewpoint setting step of acquiring blind spot information indicating a blind spot of a rider who rides the vehicle based on the detection result, and setting a virtual viewpoint in a three-dimensional space based on the blind spot information;
A viewpoint conversion step of creating a virtual viewpoint image that is an image viewed from the virtual viewpoint in the three-dimensional space with reference to the spatial data mapped by the spatial reconstruction step;
A display step of displaying the virtual viewpoint image;
An image generation method characterized in that A spatial reconstruction step of mapping an input image from one or more cameras attached to the vehicle to a spatial model;
A viewpoint conversion step of creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the spatial reconstruction processing;
The virtual viewpoint corresponding to a landscape that cannot be seen as a blind spot on a display device that is installed in a part of the vehicle that causes a blind spot of a passenger by controlling a display form of the virtual viewpoint image A display control process for displaying an image;
An image generation method characterized in that A spatial reconstruction step of mapping an input image from one or more cameras attached to the vehicle to a spatial model;
A viewpoint conversion step of creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the spatial reconstruction step;
The virtual view image of the virtual viewpoint in a direction virtually reflected by the display device when the rider views the display device by controlling the table form of the virtual viewpoint image, and the ride A display control step of displaying the virtual viewpoint image that is satisfied so that a blind spot of the person does not become an unsightly part,
An image generation method characterized in that Spatial reconstruction means for mapping an input image from one or more cameras attached to the imaging means arranged object to a spatial model;
An imaging means arranged object motion detecting means for detecting an operation state of the imaging means arranged object;
Based on the result of the detection, virtual viewpoint setting means for acquiring blind spot information indicating a blind spot of an observer who operates the imaging means arranged object and setting a virtual viewpoint in a three-dimensional space based on the blind spot information. When,
Viewpoint conversion means for creating a virtual viewpoint image that is an image viewed from the virtual viewpoint in the three-dimensional space with reference to the spatial data mapped by the space reconstruction means;
Display control means for controlling the display form of the virtual viewpoint image;
An image generation apparatus comprising:   The display control means is configured to control display in a virtual viewpoint image including a location that becomes the blind spot and an image around the location so that the location that becomes the blind spot can be distinguished from other locations. The image generation apparatus according to claim 15, wherein the image generation apparatus is an image generation apparatus.   The said display control means is comprised so that the display of the said virtual viewpoint image may be controlled so that the location used as the said blind spot may be divided by the color development different from another location. The image generating apparatus described.   The virtual viewpoint setting unit obtains, as the blind spot information, information related to the generation tendency of the blind spot that changes in accordance with the operation on the object disposed on the imaging unit, and the virtual viewpoint setting unit fits the virtual blind spot generation tendency in the virtual space in the three-dimensional space. The image generation apparatus according to claim 1, wherein the image generation apparatus is configured to adaptively set a viewpoint. Spatial reconstruction means for mapping an input image from one or more cameras attached to the imaging means arranged object to a spatial model;
Viewpoint conversion means for creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the spatial reconstruction means;
Display means for displaying the virtual viewpoint image;
Controls the display form of the virtual viewpoint image, and corresponds to the landscape that cannot be seen as the blind spot in the display unit installed in the imaging unit-arranged object and causing the blind spot of the observer Display control means for displaying the virtual viewpoint image;
An image generation apparatus comprising: Spatial reconstruction means for mapping an input image from one or more cameras attached to the imaging means arranged object to a spatial model;
Viewpoint conversion means for creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the spatial reconstruction means;
Display means for displaying the virtual viewpoint image;
The virtual view image of the virtual viewpoint in a direction virtually reflected by the display means when the observer views the display means by controlling the table form of the virtual viewpoint image, and Display means for displaying the virtual viewpoint image that is satisfied so as not to cause an unobservable part as a blind spot of the observer;
An image generation apparatus comprising:   21. The virtual viewpoint image displayed by the display control means highlights and displays a region corresponding to a portion that is satisfied so that the blind spot does not cause an unsightly portion. The image generating apparatus described in 1.   21. The image generation apparatus according to claim 20, wherein the display control unit distorts the virtual visual field image and displays the virtual visual field image on the display unit with a wide field of view. A spatial reconstruction process for mapping an input image from one or more cameras attached to the imaging means arranged object to a spatial model;
An imaging means placement object motion detection process for detecting an operation status of the imaging means placement object;
Virtual viewpoint setting for acquiring blind spot information indicating a blind spot of an observer who operates the imaging unit placement object based on the detection result and setting a virtual viewpoint in the three-dimensional space based on the blind spot information Processing,
A viewpoint conversion process for creating a virtual viewpoint image that is an image viewed from the virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the spatial reconstruction process;
Display processing for displaying the virtual viewpoint image;
Generation program for causing a computer to execute the program. A spatial reconstruction process for mapping an input image from one or more cameras attached to the imaging means arranged object to a spatial model;
A viewpoint conversion process for creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the spatial reconstruction process;
The display form of the virtual viewpoint image is controlled to correspond to a landscape that cannot be seen as the blind spot on the display device that is installed in the imaging means placement object and that causes the blind spot of the observer. Display control processing for displaying the virtual viewpoint image;
Generation program for causing a computer to execute the program. A spatial reconstruction process for mapping an input image from one or more cameras attached to the imaging means arranged object to a spatial model;
A viewpoint conversion process for creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the spatial reconstruction process;
The virtual view image of the virtual viewpoint in a direction virtually reflected by the display device when the observer views the display device by controlling a table form of the virtual viewpoint image, and the observation Display control processing for displaying the virtual viewpoint image that is satisfied so that a blind spot of the person does not become an unsightly part,
Generation program for causing a computer to execute the program. A spatial reconstruction step of mapping input images from one or more cameras attached to the imaging means arranged object onto a spatial model;
An imaging means placement object motion detection step for detecting an operation status of the imaging means placement object;
A virtual viewpoint setting step of acquiring blind spot information indicating a blind spot of an observer who operates the imaging means placement object based on the detection result, and setting a virtual viewpoint in a three-dimensional space based on the blind spot information When,
A viewpoint conversion step of creating a virtual viewpoint image that is an image viewed from the virtual viewpoint in the three-dimensional space with reference to the spatial data mapped by the spatial reconstruction step;
A display step of displaying the virtual viewpoint image;
An image generation method characterized in that A spatial reconstruction step of mapping input images from one or more cameras attached to the imaging means arranged object onto a spatial model;
A viewpoint conversion step of creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the spatial reconstruction processing;
The display form of the virtual viewpoint image is controlled to correspond to a landscape that cannot be seen as the blind spot on the display device that is installed in the imaging means placement object and that causes the blind spot of the observer. A display control step for displaying the virtual viewpoint image;
An image generation method characterized in that A spatial reconstruction step of mapping input images from one or more cameras attached to the imaging means arranged object onto a spatial model;
A viewpoint conversion step of creating a virtual viewpoint image that is an image viewed from an arbitrary virtual viewpoint in a three-dimensional space with reference to the spatial data mapped by the spatial reconstruction step;
The virtual view image of the virtual viewpoint in a direction virtually reflected by the display device when the observer views the display device by controlling a table form of the virtual viewpoint image, and the observation A display control step of displaying the virtual viewpoint image that is satisfied so that a blind spot of the person does not become an unsightly part,
An image generation method characterized in that

Images