JP4493434B2 - Image generation method and apparatus - Google Patents

Description

  The present invention relates to an image generation method and apparatus, and in particular, based on images taken by a plurality of imaging means, synthesized as an image whose viewpoint has been changed as if it were actually taken from a different viewpoint from the imaging means. The present invention relates to an image generation method and apparatus capable of displaying an image without distortion in a generated image.

  In general, when monitoring with a monitoring camera or the like, a configuration in which a captured image of a camera unit is displayed on a monitor is adopted, and a captured image from a camera attached to a desired portion of a monitoring area is arranged in a monitoring room. Display on multiple monitors. In addition, by mounting a camera on the vehicle, using the camera directed to the rear of the vehicle, the area that the driver cannot see directly or indirectly is photographed and displayed on the monitor provided in the driver's seat, contributing to safe driving I try to let them.

  However, since these monitoring devices display images on a camera-by-camera basis, the number of installations increases when shooting a wide area, and if a wide-angle camera is used, the number of installations decreases, but the accuracy of the image displayed on the monitor is reduced. Since it is rough, the display image is difficult to see and the monitoring function is degraded. For this reason, a technique has been proposed in which images from a plurality of cameras are combined and displayed as one image. For example, as shown in Patent Document 1, a plurality of camera images are divided and displayed on a single monitor, or as shown in Patent Document 2, a part of captured images are mutually displayed. In some cases, the images are arranged so as to overlap with each other, and the images are combined at one overlapping portion to be combined into one image. Also, as disclosed in Patent Document 3, there is a technique in which images from a plurality of cameras are coordinate-transformed and combined into one image to display a combined image from an arbitrary viewpoint.

  In the method disclosed in Patent Document 3, image data from a plurality of cameras is centrally captured, and a three-dimensional spatial model generated based on known information or the like is generated. Based on the parameters, mapping is performed by associating information of each pixel constituting the input image from the camera to create spatial data. In this way, after associating images from all independent cameras as points in one three-dimensional space, a viewpoint-converted image is generated and displayed from an arbitrary virtual viewpoint instead of a real camera viewpoint. To do. According to such a viewpoint conversion image display method, there is an advantage that the entire monitoring area is displayed from one arbitrary viewpoint without degrading the image accuracy, and the area to be monitored can be confirmed from the arbitrary viewpoint.

  By the way, when a moving object or an obstacle (for example, around the vehicle) is pasted as a spatial model on the viewpoint conversion image, the above-described conventional method obtains distance information and creates a screen-like position on the image according to the distance. The method of setting the vertical plane space model is adopted. When there is no distance information in the spatial model, if this is pasted on the viewpoint conversion image, the spatial model is distorted, resulting in a display image that is difficult to visually recognize. Therefore, the distance information is acquired and the three-dimensional shape may be restored and pasted by imaging with a stereo camera, but the baseline length (distance between the optical axes of the camera pair) in each stereo camera is fixed. Therefore, there is a problem that the image becomes rough depending on the distance to the object, and display with high accuracy cannot be performed.

The subject is imaged with a stereo camera, the correlation between a pair of images is obtained, and the distance is obtained from the parallax for the same object using the camera parameters such as the installation position and focal length of the stereo camera to determine the distance for image processing. There is also an example of use (Patent Document 4).
JP 05-310078 A JP-A-10-164666 Japanese Patent No. 3286306 Japanese Patent Laid-Open No. 5-114099

  The present invention pays attention to the above-mentioned conventional problems, and in the case of constructing a viewpoint-converted image three-dimensionally using an imaging means using a stereo camera, a clear image is obtained regardless of whether the distance to the imaging object is near or near. It is an object to provide an image generation method and apparatus capable of constructing a converted image in which an image is pasted on a spatial model with extremely small image distortion even when viewpoint conversion processing is performed. .

  That is, the present invention obtains an image in which parameters such as distortion are approximated and the resolution and accuracy of distortion correction are very close when acquiring an image by the stereo imaging means, so that the resolution of the image after distortion correction is obtained. An object of the present invention is to provide an image generation method and apparatus capable of making the distributions the same, improving the accuracy of stereo matching, and generating a spatial model more appropriately.

In order to achieve the above object, an image generation method according to the present invention is a method for generating a viewpoint-converted image based on image information by one or a plurality of imaging means arranged in a vehicle and a spatial model, A plurality of stereo camera units that form an image with a single imaging lens after dividing the optical path of the imaging means are arranged at intervals, and the spatial model is a short-distance space model or a long-distance space based on the perspective information. When updating with the image information by the short baseline length imaging means and the image information by the long baseline length imaging means according to whether it is a model, the image information by the long baseline length imaging means is a one-sided visual field on the same side of the pair of stereo camera units It is characterized by being acquired using image information.

An image generation apparatus according to the present invention is an image generation apparatus that generates a viewpoint conversion image based on image information supplied from one or a plurality of imaging units arranged in a vehicle and a spatial model, and the imaging An imaging system in which a plurality of stereo camera units that form an image with a single imaging lens after the optical path is divided are arranged at intervals and a long baseline length imaging is possible with a combination stereo camera of the pair of stereo camera units Camera switching means is provided, and the imaging camera switching means is configured to be switchable so that an image is picked up from one side view on the same side in the stereo camera unit.

Further, the present invention is an image generation apparatus that generates a viewpoint conversion image based on image information supplied from one or a plurality of imaging units arranged in a vehicle and a space model, and the imaging unit performs optical path division Later, multiple stereo camera units that form images with a single imaging lens are arranged at intervals, and a short baseline length image is captured by a single stereo camera unit and a combination of a pair of stereo camera units is used. An imaging camera switching unit that switches image capturing is provided, and the imaging camera switching unit can be switched so as to capture an image from one side visual field on the same side of the stereo camera unit, and image information by the stereo camera unit alone and the combined stereo camera The image information may be generated from the image information by

In the image generation apparatus, the stereo camera unit can acquire a divided image by an imaging unit having a stereo adapter and an imaging optical system.
The present invention, as an image generating method, 1 or the image information by a plurality of image pickup means, a method of generating a viewpoint conversion image based on a spatial model, a single imaging lens of the imaging unit after the optical path splitting A plurality of stereo camera units that form an image are arranged at intervals, and the spatial model is determined by a short baseline length imaging unit depending on whether the spatial model is a short-distance spatial model or a long-distance spatial model from the perspective information. When updating the image information and the image information by the long baseline length imaging means, the image information by the long baseline length imaging means may be acquired by using one-side view image information on the same side of the pair of stereo camera units.

Furthermore, the present invention is, as an image generating apparatus, in particular, the image information supplied from one or more of the imaging means, an image generating apparatus for generating a viewpoint conversion image based on a spatial model, the optical path of said image pickup means Imaging camera switching means that comprises a plurality of stereo camera units that are imaged by a single imaging lens after being divided and arranged at intervals, and that enables long baseline length imaging by a combined stereo camera of the stereo camera unit pair The imaging camera switching means can be configured to be switchable so as to capture an image from the one-side field of view on the same side in the stereo camera unit.

The image generation apparatus generates a viewpoint conversion image based on image information supplied from one or a plurality of imaging means and a spatial model, and the imaging means is connected by a single imaging lens after dividing the optical path. Imaging camera switching means configured to arrange a plurality of stereo camera units to be imaged at intervals, and to switch between imaging of a short baseline length image by a single stereo camera unit and a combination of a stereo camera unit pair and imaging of a long baseline length image by a stereo camera The imaging camera switching means can be switched so as to capture images from one side view on the same side in the stereo camera unit, and image generation can be performed from image information from the stereo camera unit alone and image information from the combination stereo camera. It can be.

In the image generating apparatus, the imaging unit may be an imaging unit-arranged object in which a plurality of stereo camera units are arranged in two baseline directions . Further, the imaging means is characterized in that a plurality of stereo camera units are arranged in two base line directions on an imaging means arranged object.

Further, the image pickup means may be arranged such that a plurality of stereo camera units are arranged in two base line directions on the image pickup means arranged object. The stereo camera unit may acquire a divided image by an imaging unit having a stereo adapter and an imaging optical system.

According to the above-described configuration, a plurality of stereo camera units that form an image with a single imaging lens after the optical path of the imaging unit is divided are arranged at intervals, and the spatial model is represented by a short-distance space based on the perspective information. When updating the image information by the short baseline length imaging means and the image information by the long baseline length imaging means depending on whether the model is a long-distance space model, the image information by the long baseline length imaging means is a pair of stereo cameras Since it is configured to acquire using one-side view image information on the same side of the unit, an image with an approximate distribution of distortion (distortion) in the screen can be obtained by using a split screen on the left and right sides. The distortion correction model to be used for stereo distance measurement can be shared, and the magnification at the time of distortion correction has a value close to the corresponding point. The resolution distribution after the image become the same, improved stereo matching accuracy, it can be more appropriately generated a spatial model.

  In the generation of the spatial model on the long baseline length side, one side image on the same side is cut out from the divided field of view in the image using the stereo camera unit and used as a stereo pair image for generating a spatial model. A space model having a short baseline length and a space model having a long baseline length are input to the space reconstruction device. By using these selectively at a desired distance, a more detailed spatial model is generated.

DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, exemplary embodiments of an image generation method and apparatus according to the invention will be described in detail with reference to the drawings.
FIG. 1 is a system configuration block diagram of an image generation apparatus according to the embodiment. FIG. 2 is a diagram illustrating that the image generation apparatus according to the embodiment is mounted on a vehicle 10 so that the surrounding situation can be monitored for assistance in driving the vehicle. FIG.

  The basic configuration of this system is a plurality of stereo camera units 12 as imaging means, and the image data acquired by the stereo camera unit 12 is processed and reproduced as a composite image viewed from a virtual viewpoint different from the camera viewpoint. It is composed of a viewpoint conversion composite image generation / display device 14 for display. Here, in FIGS. 1 and 2, additional symbols such as each stereo camera unit 12 and a light receiving unit 18 described later are F and R for identifying front and rear camera groups, L and R for identifying left and right stereo cameras, and a stereo camera unit. 12, L and R of the identification symbols of the left and right light receiving portions 18 are sequentially appended.

  First, as shown in FIG. 2, a plurality of stereo camera units 12 as imaging means are provided in the front and rear portions of the vehicle 10 as the imaging means arranged object. In the illustrated example, a front camera group 12F (12FR, 12FL) is mounted on the front portion of the vehicle 10. Further, a rear camera group 12R (12RR, 12RL) as an imaging unit is also provided at the rear part of the vehicle. In this embodiment, an image selection device 28 (28F, 28R) for capturing image data from the front and rear camera groups 12F, 12R is provided, and an image from the viewpoint conversion image generation / display device 14 provided near the driver's seat is provided. In response to the selection command, a necessary image is selected and returned to the viewpoint converted image generation / display device 14 as image information. Data transmission / reception may be performed through, for example, an in-vehicle LAN line.

  Each stereo camera unit 12 attached to the front and rear of the vehicle 10 uses a mirror type stereo camera unit 12 as shown in FIG. In other words, the imaging lens 17 is arranged in the center of the casing 16, and the deflection mirror prism 21 and the two right and left deflection mirrors 22 are installed in front of the imaging lens 17. An imaging element 20 that receives imaging light input through the imaging lens 17 as an imaging signal is disposed at the center of the back surface of the casing 16. From the left and right deflection mirrors 22 in the left and right areas of the imaging element 20, respectively. The subject image is formed. Reference numeral 23 denotes the acquired divided image.

  As a result, an image from two viewpoints is bent by the deflection mirror 22 and the deflection mirror prism 21 serving as optical path dividing means via the left and right light receiving portions 18R and 18L, and is incident on one imaging lens 17, and the subject 19 is captured in stereo. can do.

  As shown in FIG. 2B, the left and right stereo camera units 12 distort the left and right images by dividing the optical path by the deflecting mirror 22 and forming an image with a single imaging lens 17. So the distortion is different.

  Moreover, in this embodiment, in the front camera group 12F and the rear camera group 12R, it is comprised so that a base line length can be changed by changing the camera unit to view in stereo. That is, as shown in FIG. 1, the optical axes between the pair of left and right light receiving portions 18R, 18L in each stereo camera unit 12 are set to be parallel to each other, and the distance between the optical axes. The baseline length L1 is a fixed length. On the other hand, using the one-side light receiving portions 18FRL and 18FLL (or 18FRR and 18FLR) on the same side of both the left and right stereo camera units 12FR and 12FL in the front camera group 12F, the light receiving portions 18FRL and 18FLL for stereo viewing are used. Stereo imaging can be performed with L2 being the distance between the optical axes (the separation distance of the stereo camera unit 12) as the base line length. Since the base line length L2 is longer than the above-described base line length L1, this is used as the long base line length L2, and the base line length L1 (relatively, L1 is a short base line) when stereo viewing is performed with a single stereo camera unit 12FR (or 12FL). It is possible to obtain a longer stereo image than This is provided with imaging camera switching means, which will be described later, so that images are taken from one side of the same side of the pair of stereo camera units 12.

  As shown in FIG. 1, the imaging region by the one-side light receiving unit of each stereo camera unit 12 is shown as a broken-line fan shape, and the overlapping portion thereof is a short-distance stereo imaging region (hatching portion) based on stereo vision with a short baseline length. ) In the figure, reference numeral 24 denotes an imaging area of an image used at a short distance by the stereo camera unit 12FR.

  On the other hand, if a temporary stereo camera unit is configured by combining the left light-receiving portions 18FRL and 18FLL of the paired stereo camera units 12FR and 12FL as shown by the solid line fan shape in FIG. A solid line sector-shaped overlapping portion, which is a region, becomes a long-distance imaging region based on stereo viewing with a long baseline length. In the figure, reference numeral 25 denotes an imaging region (hatched portion) of a long-distance use image by a temporary stereo camera unit in which the left light receiving units 18FRL and 18FLL are combined. The base line length L2 between the pair of stereo camera units is longer than the base line length L1 of the stereo camera unit 12 alone, and distance measurement at a long distance that is not clear with the stereo camera unit 12 alone becomes possible.

  Image data is transmitted from each stereo camera unit 12 to the viewpoint conversion composite image generation / display device 14 that receives the image data from the stereo camera unit 12 via a communication control device (not shown). Since the image data to be acquired from the camera unit 12 is determined by the set virtual viewpoint, as described above and as shown in FIG. 1, the image selection device 28 is used to acquire the image data corresponding to the set virtual viewpoint. Is provided. An image data packet corresponding to the set virtual viewpoint is selected from image data packets input from an arbitrary stereo camera unit 12 by the image selection device 28, and is used for image composition processing in the subsequent stage.

  The basic processing in the viewpoint conversion composite image generation / display device 14 is to input an image photographed from the viewpoint of each imaging means, set a three-dimensional space where an imaging means placement object such as a vehicle is placed, and this tertiary The original space is specified by an arbitrarily set origin (virtual viewpoint), and the pixel of the image data is coordinated and corresponded to the three-dimensional space seen from the specified virtual viewpoint, on the image plane viewed from the virtual viewpoint. A process of rearranging the pixels is performed. As a result, an image is obtained by rearranging the pixels of the image data obtained from the camera viewpoint in the three-dimensional space defined by the virtual viewpoint, and creating a composite image from the desired viewpoint that is not the camera viewpoint. It can be output and displayed.

  In this system, an ID is added to each image data taken by each stereo camera, and at least one of a time stamp, imaging means position and orientation information, imaging means internal parameters, and exposure information is included. As a result, the image data sent from each stereo camera unit 12 is given an ID, and includes a time stamp and other shooting information, and continuously from the buffer device to the viewpoint conversion composite image generation / display device 14. Packets are transmitted.

  In the viewpoint conversion image generation / display device 14 to which image data or the like is sent, a plurality of stereo camera units 12 are switched in accordance with the viewpoint conversion by the image selection device 28 serving as an imaging camera switching unit, and an imaging target is also selected. Depending on the distance of an object, imaging with a short baseline length by the stereo camera unit 12 and imaging with a long baseline length by a stereo camera configured by a combination of a pair of stereo camera units 12 are switched. Further, in the case of the long baseline length, the stereo pair that is a combination of the visual fields on the same side in the pair of stereo camera units 12 is switched.

  Since the stereo camera captured image described above is temporarily stored in the camera buffer by packet communication in units of ID-attached image data, it is possible to combine image data at the same time using ID information. For this reason, the viewpoint conversion composite image generation / display device 14 arranges the captured images from the plurality of stereo camera units 12 in time series based on the ID information, and stores the photographed image data storage device 30 in time series. It has. Also, if the parameters of the acquired image data are not synchronized, the composite image is far from the actual situation. Therefore, as described above, the ID includes at least one of the time stamp, the imaging unit position / orientation information, the imaging unit internal parameters, and the exposure information, and the image data pasted in the three-dimensional space is adjusted as necessary. I am doing so.

  The vehicle 10 equipped with the system according to this embodiment is provided with a distance measuring device 46 that measures the distance to the moving obstacle. In this embodiment, the distance measuring device 46 may use distance measurement by a laser radar, a millimeter wave radar, or the like and distance measurement by stereo imaging. The measurement by the radar 44 may be performed using a normal system that measures the time difference between the transmission signal and the reflected signal. In the distance measurement by stereo imaging, the same subject is photographed from a plurality of different viewpoints, the correspondence of the same point of the subject in these images is obtained, and the distance to the subject is calculated by the principle of triangulation. More specifically, the entire right image of the image captured by the stereo imaging means is divided into small areas to determine a range for performing stereo distance measurement, and then the position of the image that is the same as the left image is detected. Then, the position difference between these images is calculated, and the distance to the object may be calculated from the relationship between the left and right camera mounting positions. Distance image data is generated and stored in the storage device 47 based on distance information obtained by stereo ranging between two or more images captured by the stereo camera.

  Further, the viewpoint conversion image generation / display device 14 is provided with a spatial model generation device 48. The spatial model generation device 48 generates a spatial model using image data, distance image data obtained by the distance measuring device 46, and calibration data.

  The calibration device 50 has an image pickup means (stereo camera unit 12) arranged in the three-dimensional real world, the attachment position, the attachment angle, the lens distortion correction value, and the lens focal length in the three-dimensional real world. The camera parameters representing the camera characteristics such as are determined and specified. Camera parameters obtained by the calibration are stored in the storage device 52 as calibration data.

  Therefore, the space model generation device 48 generates a space model using the image data, the distance image data, and the calibration data. The generated space model is stored in the storage device 54.

  Each pixel of the selectively captured image data is associated with a point in the three-dimensional space by the space reconstruction device 32 and reconstructed as space data. This calculates where each object constituting the selected image exists in the three-dimensional space, and temporarily stores the spatial data as a calculation result in the spatial data storage device 34. The calculation is performed for all the pixels of the image obtained from each imaging means.

  The viewpoint conversion device 36 serving as a viewpoint conversion means can convert the image data of the spatial data storage device 34 of the spatial reconstruction device 32 into an image viewed from an arbitrary viewpoint position, and can specify an arbitrarily set viewpoint. That is, it designates from which position in the three-dimensional coordinate system, at which angle, and at what magnification, the image is to be viewed. As a result, an image viewed from a new conversion viewpoint is generated by the data read from the spatial data storage device 34, temporarily stored in the viewpoint conversion image data storage device 38, and then converted into a viewpoint conversion image by the display device 40. Will be displayed.

  Note that the viewpoint conversion image generation / display device 14 is provided with an imaging device arranged object model storage device 56 that stores and stores the vehicle model so that the vehicle model can be displayed simultaneously when the space is reconstructed. Yes. In addition, a viewpoint selection device 58 is provided, and image data corresponding to a preset virtual viewpoint specified in advance is stored in the storage device 60, and the corresponding image is immediately used when the viewpoint selection processing is performed. Are transmitted to the viewpoint conversion device 36, and a converted image corresponding to the selected virtual viewpoint is displayed.

  Here, in the present invention, when the viewpoint conversion image is generated based on the image information by one or a plurality of imaging units arranged in the vehicle 10 and the space model, the imaging unit is configured by the stereo camera unit 12, and the space When the model is updated with the image information by the short baseline length imaging means and the image information by the long baseline length imaging means according to whether the model is a near distance space model or a far distance space model, the long baseline length imaging is performed. The image information obtained by the means is obtained by using one-side visual field image information on the same side of the pair of stereo camera units 12.

  That is, when imaging a short-distance object and a long-distance object from the vehicle 10, an appropriate display can be performed when generating a spatial model by changing the baseline length of the stereo camera unit. Yes. As shown in FIG. 1, when imaging an object existing at a short distance of the stereo camera unit 12, short base length imaging is performed by the single stereo camera unit 12. The base line length L1 of the stereo camera unit 12 is the distance between the optical axes of the fixed light receiving units 18L and 18R, and is the base line length having the optimum length for imaging a short distance. For this reason, highly accurate distance image data can be obtained on a captured image at a short distance.

  On the other hand, when imaging a subject at a long distance from the mounted stereo camera unit 12 of the vehicle 10, a combination of light receiving units on the same side in a pair of stereo camera units 12FR, 12FL (or 12RR, 12RL), for example, a front stereo camera The left light-receiving portions 18FRL and 18FLL of the units 12FR and 12FL are treated as a combination of stereo light-receiving portions constituting one stereo camera unit, and a stereo vision image by these light-receiving portions is acquired as an input image. The base line length L2 that is the distance between the optical axes of the light receiving units 18FRL and 18FLL is longer than the base line length L1 between the single stereo camera units 12, and stereo imaging with a long base line length is possible. Therefore, in this embodiment, a stereo image with a short baseline length and a stereo image with a long baseline length can be easily realized by switching the imaging means.

  In the present embodiment, when performing stereo imaging with a long baseline length, single-side view image information on the same side of the pair of stereo camera units 12 is used. An example in which distance image data is acquired by imaging with the stereo camera units 12FL and 12FR will be described with reference to the schematic diagram of FIG. When the subject 19 is imaged, the subject 19 is imaged on the imaging elements 20 of the stereo camera units 12FL and 12FR as shown in FIG. When a long baseline length image is obtained using a field image common to one side of the two stereo camera units 12FL and 12FR (a combination of right and right and left and left), as shown in FIG. The side (right and right) images are cut out to create a stereo image pair. In this way, distortion (distortion aberration) due to the lens is generated in the same way in the left and right images. Therefore, even when the rectification is performed as shown in FIG. A difference in resolution is unlikely to occur, and a corresponding point search by stereo matching can be performed more accurately.

  FIG. 5 shows a flowchart showing generation of a distance image by stereo matching of stereo imaging according to the present embodiment. Hereinafter, an example of using a right-side (18FRR-18FLR) visual field image forming a stereo pair among images input from the left stereo camera unit and the right stereo camera unit will be described.

The right visual field portion of the captured image of each stereo camera unit 12FR, 12FL is cut into a predetermined size by right visual field clipping processing (S200), (S204), and a stereo left image and a stereo right image are generated.
(S202) The obtained stereo left image.
(S206) The obtained stereo right image.
(S208) The obtained rectification calibration data. The calibration device 50 performs calibration for rectification in advance, and generates calibration data such as baseline length, internal and external camera parameters according to the right camera or left camera of the selected stereo camera unit.
(S210) Next, the left and right stereo images are corrected for distortion aberration of the stereo left and right images based on the calibration data for rectification so that the corresponding points of the left and right images are on the same line on the epipolar line. A rectification process for geometrically transforming the image is performed.
(S212) The obtained stereo left image after rectification.
(S214) The obtained stereo right image after rectification.
(S216) The left and right images after this rectification are subjected to stereo matching processing, corresponding point search is performed, and parallax is calculated. Thereby, a map of the amount of parallax at each point on the image is generated, and this becomes parallax data.
(S218) The obtained parallax data.
(S220) The obtained stereo distance calibration data. Calibration for use in stereo distance measurement is performed in advance by the calibration device 50, and calibration data such as baseline length, internal and external camera parameters corresponding to the selected stereo camera unit is generated.
(S222) By the stereo distance calibration, the parallax amount is converted into a distance from the reference point, and distance image data is generated.
(S224) The obtained distance image data.

  By performing the processing as described above, distance image data can be calculated from images of a plurality of stereo cameras. The obtained distance image data is used for generating a spatial model. The distance image data can be acquired by performing the same process for the left-side (18FRL-18FLL) visual field image forming the stereo pair.

FIG. 6 shows a processing flow of the viewpoint conversion image of the viewpoint conversion image generation apparatus having the above configuration. Hereinafter, an image generation method of the viewpoint conversion image will be described.
(S102) The viewpoint selecting device 58 selects an arbitrary virtual viewpoint such as the front or rear of the host vehicle.
(S104) The image selection device 28 selects an imaging device installed in the host vehicle corresponding to the selected virtual viewpoint. The base line length is arbitrarily selected by the image selection device 28 serving as a control means for switching the short base line length by the stereo camera unit 12 or the long base line length by the stereo camera configured by the combination of the pair of stereo camera units 12.
(S110) The distance information 46 by the radar 44 is used to acquire approximate distance information of the captured image, and simple distance image data is generated.
(S112) An obstacle in the simple distance image data is recognized. Further, the obstacle is measured by the distance measuring device 46.
(S114) Based on the approximate distance of the detected obstacle, it is selected whether to use the long baseline length, the short baseline length, or the distance image data outside the stereo ranging area to generate the spatial model of the obstacle portion. Judgment. Further, the distance measuring device 46 becomes a perspective switching update unit based on the approximate distance of the obstacle, and switches the perspective of the image information used for updating the space model.
(S120) Long baseline length imaging is performed between the stereo camera units 12 serving as a long baseline length image pair.
(S122) Calibration data. Calibration for use in stereo matching is performed in advance by the calibration device 50, and calibration data such as baseline length, internal and external camera parameters corresponding to the selected stereo camera unit 12 is generated.
(S124) Stereo matching of the picked-up image selected based on the calibration data is performed. In other words, a predetermined window is cut out from the left and right images viewed in stereo, and the normalized correlation value of the window image is calculated while scanning on the epipolar line to search for corresponding points, and the parallax between the pixels of the left and right images is calculated. calculate. The distance is calculated from the parallax based on the calibration data, and the obtained distance data is used as the long baseline long distance image data.
(S126) Long baseline long distance image data obtained.
(S128) The measured distance image data is output to the spatial model generation device 48 in accordance with the obstacle distance determination described above.
(S130) Short baseline length imaging is performed by the stereo camera unit 12 serving as a short baseline length image pair.
(S132) Calibration data. Calibration for use in stereo matching is performed in advance by the calibration device 50, and calibration data such as baseline length, internal and external camera parameters corresponding to the selected stereo camera unit 12 is generated.
(S134) Stereo matching of the picked-up image selected based on the calibration data is performed. In other words, a predetermined window is cut out from the left and right images viewed in stereo, and the normalized correlation value of the window image is calculated while scanning on the epipolar line to search for corresponding points, and the parallax between the pixels of the left and right images is calculated. calculate. The distance is calculated from the parallax based on the calibration data, and the obtained distance data is used as the short baseline long distance image data.
(S136) The obtained short baseline long distance image data.
(S138) The measured distance image data is output to the spatial model generation device 48 in accordance with the obstacle distance determination described above.
(S140) Outside of the stereo distance measurement area, a simple space model in which an obstacle is represented by a screen-like plane is represented with respect to the space model based on the distance measurement results of the distance measurement device 46, that is, laser radar or millimeter wave radar. The distance measuring device distance image data is generated.
(S142) The obtained distance measuring device distance image data.
(S144) The distance model data of the long baseline length, the distance image data of the short baseline length, and the distance image data by the distance measuring device 46 are input to the spatial model generation device 48 serving as the spatial model update unit by the input unit. By using these selectively at a desired distance, a more detailed spatial model is generated.
(S146) The actual image data corresponding to the space model is mapped to the space model in the three-dimensional space according to the calibration data. Spatial data subjected to texture mapping is generated.
(S148) With reference to the spatial data created by the space reconstruction device 32, the viewpoint conversion image viewed from the desired virtual viewpoint is generated by the viewpoint conversion device 36. The generated viewpoint-converted image data is temporarily stored in the viewpoint-converted image data storage device 38.
(S150) The generated viewpoint-converted image data is displayed on the display device 40. Such a process is sequentially repeated according to a change in the distance or shape of the host vehicle or the obstacle. As a result, the spatial model is updated, and the virtual viewpoint images can be sequentially displayed.

  As described above, according to the image generation method, the viewpoint conversion image is generated based on the image information by one or a plurality of imaging means arranged in the vehicle and the space model, and the imaging means is a stereo camera. It is composed of units, and the space model is updated by the image information by the short baseline length imaging means and the image information by the long baseline length imaging means according to whether the space model is a short distance space model or a far distance space model. At this time, since the image information by the long baseline length imaging means is obtained using the one-side visual field image information on the same side of the pair of stereo camera units, the distance measurement of the subject on the captured image at a short distance and a long distance is performed. It can be performed.

  For this reason, by using the divided screens on the left and right sides, it is possible to obtain an image that approximates the distribution of distortion in the screen. In addition, it is possible to share a distortion correction model for stereo distance measurement, and the magnification at the time of distortion correction has a value close to the corresponding point, so that the resolution distribution of the image after distortion correction is the same. Therefore, the accuracy of stereo matching is improved, and a spatial model can be generated more appropriately.

Further, if the generated virtual viewpoint image is displayed on the monitor screen of the vehicle, it becomes possible to recognize the vehicle surrounding situation in detail, and the safety can be greatly improved.
In each of the above-described examples, the plurality of imaging devices may be used so as to configure a so-called trinocular stereo camera or may be used so as to configure a four-eye stereo camera. In this way, it is known that using a three-lens or four-eye stereo camera provides more reliable and stable processing results in three-dimensional reconstruction processing (Fumiaki Tomita: published by the Information Processing Society of Japan) Information processing "Volume 42 No.4" High-function 3D visual system "). In particular, when a plurality of cameras are arranged in two base length directions, a so-called multi-baseline stereo camera can be realized, and more accurate stereo distance measurement can be realized.

  In addition, although the target which installs imaging means, such as a camera in a predetermined form, shows an example mounted on a vehicle, the imaging device placement object is, for example, indoors such as a pedestrian, a street, a store or a residence, an office, etc. Even when attached, the same image generation can be performed. With this configuration, it is possible to adapt to a monitoring camera or a wearable computer that is attached to a person and performs video-based information acquisition.

  The image generation method and apparatus according to the present invention can display the peripheral information outside the vehicle as an image viewed from a virtual viewpoint different from the camera viewpoint on the display device installed in the driver's seat of the vehicle, and is also a safety guard. It can be used as a building or indoor and outdoor monitoring device.

1 is a system configuration block diagram of an image generation apparatus according to an embodiment. It is a block diagram of the configuration when the image generation device is mounted on a vehicle. It is sectional drawing of the stereo camera unit applied to embodiment. It is explanatory drawing of the image process by the long base line length in the image generation method which concerns on embodiment. It is a flowchart which shows the production | generation process of the distance image data at the time of the image production | generation concerning the embodiment. It is a flowchart of viewpoint conversion image generation concerning an embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ......... Vehicle, 12 ......... Stereo camera unit, 14 ......... Viewpoint conversion image generation / display device, 16 ......... Case, 17 ......... imaging lens, 18 ......... Light receiving unit, 19 ... ... Subject, 20 ......... Image sensor, 21 ......... Deflection mirror prism, 22 ......... Deflection mirror, 24 ......... Use image pickup area at short distance, 25 ...... Use image pickup area at long distance, 28 ......... Image selection device, 30 ......... Real image data storage device, 32 ......... Spatial reconstruction device, 34 ......... Spatial data storage device, 36 ......... Viewpoint conversion device, 38 ......... Viewpoint conversion image Data storage device, 40 ......... Display device, 44 ......... Radar, 46 ......... Distance measuring device, 47 ......... Distance image data storage device, 48 ......... Spatial model generation device, 50 ......... Calibration Equipment 52 ... Calibration Yondeta storage device, 54 ......... space model storage unit, 56 ......... imaging device disposed object model storage unit 58 ......... viewpoint selecting unit, 60 ......... virtual viewpoint data storage device.

Claims (11)

A method for generating a viewpoint-converted image based on image information from one or a plurality of imaging means arranged in a vehicle and a spatial model,
An interval a stereo camera unit imaged in a single imaging lens after the image pickup means and the optical path splitting configured by arranging a plurality said whether the short-range space model from the perspective information space model far When updating with the image information by the short baseline length imaging means and the image information by the long baseline length imaging means according to whether it is a spatial model, the image information by the long baseline length imaging means is one side of the same side of the pair of stereo camera units An image generation method characterized by obtaining using field-of-view image information. An image generation device that generates a viewpoint conversion image based on image information supplied from one or a plurality of imaging units arranged in a vehicle and a spatial model,
A plurality of stereo camera units that form an image with a single imaging lens after the optical path of the imaging means is divided, and a plurality of stereo camera units are arranged at intervals, and long baseline length imaging with a combination stereo camera of the stereo camera unit pair is possible. An image generation apparatus characterized in that the imaging camera switching means is configured to be switchable so that an image is picked up from one side visual field on the same side in the stereo camera unit. An image generation device that generates a viewpoint conversion image based on image information supplied from one or a plurality of imaging units arranged in a vehicle and a spatial model,
The imaging means is configured by arranging a plurality of stereo camera units that form an image with a single imaging lens after dividing an optical path at intervals, and for taking a short baseline length image by a single stereo camera unit and for a pair of stereo camera units. An imaging camera switching unit that switches imaging of a long baseline length image by a combination stereo camera is provided, and the imaging camera switching unit can be switched so as to capture an image from one side view on the same side of the stereo camera unit, and the stereo camera unit alone An image generation apparatus characterized by enabling image generation from image information and image information obtained by the combination stereo camera.   4. The image generation apparatus according to claim 2, wherein the stereo camera unit is capable of acquiring a divided image by an imaging unit having a stereo adapter and an imaging optical system. A method of generating a viewpoint conversion image based on image information by one or a plurality of imaging means and a spatial model,
An interval a stereo camera unit imaged in a single imaging lens after the image pickup means and the optical path splitting configured by arranging a plurality said whether the short-range space model from the perspective information space model far When updating with the image information by the short baseline length imaging means and the image information by the long baseline length imaging means according to whether it is a spatial model, the image information by the long baseline length imaging means is one side of the same side of the pair of stereo camera units An image generation method characterized by obtaining using field-of-view image information. An image generation apparatus that generates a viewpoint conversion image based on image information supplied from one or a plurality of imaging means and a space model,
A plurality of stereo camera units that form an image with a single imaging lens after the optical path of the imaging means is divided, and a plurality of stereo camera units are arranged at intervals, and long baseline length imaging with a combination stereo camera of the stereo camera unit pair is possible. An image generation apparatus characterized in that the imaging camera switching means is configured to be switchable so that an image is picked up from one side visual field on the same side in the stereo camera unit. An image generation apparatus that generates a viewpoint conversion image based on image information supplied from one or a plurality of imaging means and a space model,
The imaging means is configured by arranging a plurality of stereo camera units that form an image with a single imaging lens after dividing an optical path at intervals, and for taking a short baseline length image by a single stereo camera unit and for a pair of stereo camera units. An imaging camera switching unit that switches imaging of a long baseline length image by a combination stereo camera is provided, and the imaging camera switching unit can be switched so as to capture an image from one side view on the same side of the stereo camera unit, and the stereo camera unit alone An image generation apparatus characterized by enabling image generation from image information and image information obtained by the combination stereo camera. 6. The image generation method according to claim 5, wherein the imaging unit is an imaging unit-arranged object in which a plurality of stereo camera units are arranged in two baseline directions . The image generating apparatus according to claim 6, wherein the imaging unit includes a plurality of stereo camera units arranged in two baseline directions on an imaging unit arranged object. The image generating apparatus according to claim 7, wherein the imaging unit includes a plurality of stereo camera units arranged in two baseline directions on an imaging unit arranged object.   The image generation apparatus according to claim 6 or 7, wherein the stereo camera unit is capable of acquiring a divided image by an imaging unit having a stereo adapter and an imaging optical system.

Images