JP2006060424A - Image generating method and apparatus thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image generating apparatus capable of rapidly performing image processing for viewpoint conversion without futility. <P>SOLUTION: The image generating apparatus generates a viewpoint conversion image from a virtual viewpoint by using image information acquired by one or a plurality of image pickup means arranged in an object in which the image pickup means are arranged. The apparatus is provided with a temporary storage means for previously acquiring necessary picked-up images for each different virtual viewpoint to temporarily store the acquired images, a temporary storage selecting means for selecting a temporary storage means having a corresponding picked-up image in response to the switching of virtual viewpoint, and a viewpoint conversion image generating means for rapidly generating a viewpoint conversion image from a picked-up image of the rapidly selected temporary storage means. The apparatus causes to output and display the generated viewpoint conversion image. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image generation apparatus, and in particular, based on images taken by a plurality of imaging means, it is synthesized and displayed as an image whose viewpoint has been changed as if it was actually taken from a different viewpoint from the imaging means. For technology.

  In general, when monitoring with a monitoring camera or the like, a configuration in which a captured image of a camera unit is displayed on a monitor is adopted, and a captured image from a camera attached to a desired portion of a monitoring area is arranged in a monitoring room. Display on multiple monitors. In addition, by mounting a camera on the vehicle, using the camera directed to the rear of the vehicle, the area that the driver cannot see directly or indirectly is photographed and displayed on the monitor provided in the driver's seat, contributing to safe driving I try to let them.

  However, since these monitoring devices display images on a camera-by-camera basis, the number of installations increases when shooting a wide area, and if a wide-angle camera is used, the number of installations decreases, but the accuracy of the image displayed on the monitor is reduced. Since it is rough, the display image is difficult to see and the monitoring function is degraded. For this reason, a technique has been proposed in which images from a plurality of cameras are combined and displayed as one image. For example, as shown in Patent Document 1, a plurality of cameras are arranged so that a part of the captured images overlap each other, and the images are combined at one overlapping portion to be combined into one image, As shown in Patent Document 2, coordinates of images from a plurality of cameras are transformed and synthesized into one image, and a synthesized image from an arbitrary viewpoint is displayed.

  By the way, in the method disclosed in Patent Document 2, image data from a plurality of cameras is integrated. Then, based on the acquired image data based on the camera parameters, a process of creating spatial data in which each pixel constituting the input image from the camera is associated with a point in the three-dimensional space is performed on all the cameras. After associating images from all independent cameras as points in one three-dimensional space in this way, the composite image viewed from the conversion viewpoint set at an arbitrary position is reconstructed by inverse transformation and displayed. To do.

According to such a composite image display method, there is an advantage that the entire monitoring area is displayed on one screen without degrading the image accuracy, and the entire area to be monitored can be confirmed instantaneously from an arbitrary viewpoint.
JP-A-10-164666 Japanese Patent No. 3286306

  However, in the above-described prior art, it is necessary to quickly perform processing with a large amount of data, particularly image data. However, image data from all cameras are collectively loaded into an image calculation processing unit, and all of the data is stored in a three-dimensional space. Data is not actually used in addition to the enormous amount of data to be processed because it is processed in the procedure to generate a composite image after selecting image data corresponding to each virtual viewpoint after processing corresponding to points It had a problem that it would be wastefully processed.

  An object of the present invention is to provide an image generation apparatus capable of performing image processing for viewpoint conversion quickly and without waste, paying attention to the above-described conventional problems.

  In the present invention, even when a large number of imaging means are used, when a viewpoint-converted image viewed from a virtual viewpoint is synthesized, the image of the imaging means is determined from the viewpoint that the image data to be used and the imaging means to be used are determined. Image data used for each imaging means or virtual viewpoint unit is stored in a buffer, and when a virtual viewpoint is set, only the image data corresponding to the virtual viewpoint is immediately selected, and image composition processing is performed. This is obtained from the knowledge that the processing can be speeded up by performing the above. That is, for example, in order to generate the front viewpoint converted image, a read command is sent to the front camera image buffer device so that it is not read from the rear camera image buffer device, or conversely, For generation, a read command is sent to the rear camera image buffer device, and high-speed processing is enabled by not reading from the front camera image buffer device.

  From such a point of view, the image generation method according to the present invention is an image generation method for generating a viewpoint-converted image from a virtual viewpoint using image information obtained by one or a plurality of imaging means arranged on an imaging means arranged object. In this case, a captured image necessary for each different virtual viewpoint is acquired in advance from each imaging means and temporarily stored, and a corresponding temporarily stored captured image is selected according to the virtual viewpoint switching to generate a viewpoint converted image. It was configured to display the output.

  An image generation apparatus according to the present invention is an image generation apparatus that generates a viewpoint-converted image from a virtual viewpoint using image information obtained by one or a plurality of imaging units arranged on an imaging unit arranged object. Temporary storage means for acquiring and temporarily storing captured images required for different virtual viewpoints from individual imaging means, and temporary storage selection means for selecting temporary storage means having corresponding captured images according to virtual viewpoint switching The viewpoint-converted image generating means for generating the viewpoint-converted image from the captured image of the temporary storage means selected immediately and the generated viewpoint-converted image are output and displayed.

  In the above configuration, the temporary storage means may be grouped for each virtual viewpoint and synchronized and temporarily stored, and a temporary storage means corresponding to the number of preset viewpoints is secured, so that the viewpoint change display is speeded up. You can do it. Further, the image pickup means and the temporary storage means may be connected by an analog line, and the temporary storage means and the viewpoint conversion image generation means may be connected by a digital line. It can be a LAN. The imaging means arrangement object may be at least one of a vehicle, a building, and a human body wearing tool.

  According to the above configuration, when monitoring the surroundings of the traveling vehicle or monitoring the surroundings of the building by being installed in various places or persons in the building, the captured images of the respective imaging means are packet-transmitted to the temporary storage means and stored. Since the image generation means is configured to be able to select and store image data in a temporary storage means corresponding to a virtual viewpoint defined in advance, the viewpoint conversion image can be selected by selecting the virtual viewpoint. Image data required for each virtual viewpoint stored in the temporary storage means can be collectively selected by selecting the temporary storage means, and only necessary image data corresponding to the virtual viewpoint is selected and extracted. Thus, a composite image can be reconstructed from the coordinate conversion image. Accordingly, it is not necessary to perform useless data processing, and data conversion can be performed by selecting and using only a portion where a large amount of image data is used, thereby shortening the processing time. In particular, when mounted on a high-speed moving body, the display delay of the composite image becomes fatal, whereas in the present invention, the processing capability can be greatly improved.

DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, exemplary embodiments of an image generation method and apparatus according to the invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram of a configuration in which the image generation apparatus according to the embodiment is mounted on a vehicle so that the surrounding situation can be monitored for assistance during vehicle operation. As shown in the figure, a plurality of cameras 12 as imaging means are provided in front and rear portions of the vehicle 10 as the imaging means arranged object. In the illustrated example, for simplicity of explanation, a front camera group 12F (12FR, 12FC, 12FL) is mounted on the front portion of the vehicle 10, and each camera is oriented 45 degrees on the right side in front of the vehicle, 45 degrees on the center, and 45 degrees on the left side. The direction of the camera is imaged. In addition, a rear camera group 12R (12RR, 12RC, 12RL) as an image pickup means is also provided at the rear part of the vehicle, and similarly, each camera takes an image of 45 degrees on the right side, 45 degrees on the right side and 45 degrees on the left side of the vehicle. Like to do. Thereby, it is possible to synthesize and generate a vehicle front image viewed from the virtual viewpoint set above the vehicle from the image obtained by the front camera group 12F and to display the image above the vehicle from the image obtained by the rear camera group 12R. A vehicle rear image viewed from another set virtual viewpoint is synthesized and generated and displayed. Of course, a right front virtual viewpoint using the front center camera 12FC and the right camera 12FR can be arbitrarily set by a combination of cameras, but is omitted here.

  The vehicle 10 is equipped with a viewpoint conversion composite image generation / display device 16 that combines images as if the images captured by the camera 12 were captured from an arbitrary viewpoint different from the viewpoint of the camera 12. Image data from each camera 12 is input to the viewpoint conversion composite image generation / display device 16, but this is not input directly, but is input via the buffer device 14 set for each camera. I am doing so. These buffer devices 14 transmit images of each camera 12 and temporarily store them, but an ID is assigned to transmitted image data. Further, the above-described viewpoint conversion composite image generation / display device 16 and the buffer device 14 are connected by an in-vehicle LAN line 18 and have a so-called LAN connection configuration. Each camera 12 and the buffer device 14 are connected via an analog line 20.

  In this system, captured image data is packet-transmitted from the buffer device 14 of each camera 12 via the LAN line 18, and as a result, as shown in FIG. The camera buffer device 14 includes an ID adding unit 24 for adding an ID to a captured image, a packet generating unit 26 for packetizing the generated image with ID, and a continuous transmission control unit 28 for the packet. In particular, the ID added to the captured image data unit by the ID adding unit 24 includes at least one of a time stamp 241, imaging unit position / orientation information 242, imaging unit internal parameters 243, and exposure information 244. . As a result, an ID is attached to the image data sent from each camera, and the viewpoint conversion composite image is generated / displayed from the buffer device 14 via the LAN line 18 in a state in which shooting information is included in addition to the time stamp 241. Packets are continuously transmitted toward the device 16.

  On the other hand, the basic processing in the viewpoint conversion composite image generation / display device 16 is to input an image photographed from the viewpoint of each camera 12, set a three-dimensional space where the vehicle 10 is placed, and arbitrarily set this three-dimensional space. Stipulated by the origin (virtual viewpoint) set to, the pixels of the image data are coordinated and corresponded to each other in the three-dimensional space seen from the stipulated virtual viewpoint, and the pixels are rearranged on the image plane viewed from the virtual viewpoint To perform the process. As a result, an image is obtained by rearranging the pixels of the image data obtained from the camera viewpoint in the three-dimensional space defined by the virtual viewpoint, and creating a composite image from the desired viewpoint that is not the camera viewpoint. It can be output and displayed.

  Here, in the embodiment according to the present invention, necessary images are temporarily stored together for each virtual viewpoint unit defined in advance, and when the viewpoint conversion is requested, only the data corresponding to the corresponding viewpoint is batched. In addition to reading, other data is not read to speed up data processing. In other words, from which camera the image data should be acquired is uniquely determined by the set virtual viewpoint. For this reason, in the viewpoint conversion composite image generation / display device 16 of this embodiment, as shown in FIG. 3, for each virtual viewpoint in the viewpoint conversion image, the necessary imaging means (camera 12) passes through the communication control device 30. Are temporarily stored in the viewpoint buffer device 32 (front camera group buffer device 32F or rear camera group buffer device 32R) connected via the LAN line 18, and the virtual viewpoint defined from now on is stored. Is provided with a buffer selection device 34 that preferentially obtains image data corresponding to. For example, when a front image of a vehicle is synthesized from a virtual viewpoint set above the vehicle 10, image data from the front camera group 12F is necessary, and image data from the rear camera group 12R is unnecessary. Therefore, the buffer selection device 34 generates an image selection command in accordance with the designation of the virtual viewpoint. Only necessary data corresponding to the designated conversion viewpoint is acquired from the plurality of conversion viewpoint data temporarily stored corresponding to each virtual viewpoint from the front camera group buffer device 32F, and transmitted to the space reconstruction device 36. I have to. Similarly, image data of the front camera group 12F is unnecessary when a composite image for rear display is required. In this case, the buffer selection device 34 may select the buffer data input from the rear camera group 12R corresponding to the virtual viewpoint set to be displayed backward from the rear camera group viewpoint buffer device 32R and transmit the packet. .

  In the embodiment, the buffer selection device 34 generates an image at an arbitrary virtual viewpoint separately from the image selection from the viewpoint buffer devices 32F and 32R that temporarily stores necessary data in preset virtual viewpoint units. Image data can be directly acquired from the camera 12 via the communication control device 30 so that necessary image data can be selected.

  The captured image is transmitted from the camera 12 to the camera buffer device 14 via the analog line 20, while the camera buffer device 14 and later are temporarily stored in the buffer device 32 by packet communication in units of image data with ID. The image data at the same time can be combined using the ID information. For this reason, the viewpoint conversion composite image generation / display device 16 arranges the captured images from the plurality of cameras 12 in time series based on the ID information, and the image data storage device 40 stores them in time series. It has. Further, if the parameters of the acquired image data are not synchronized, the composite image is far from the entity. Therefore, as described above, the ID includes at least one of the time stamp 241, the imaging unit position / orientation information 242, the imaging unit internal parameter 243, and the exposure information 244, and the image data pasted in the three-dimensional space as necessary. It is sufficient to make adjustments.

  In the above embodiment, the buffer device 14 attached to the camera 12 is configured to transmit data to the viewpoint buffer device 32 (front or rear camera image buffer devices 32F and 32R) through the LAN line 18. As shown in FIG. 4, the video data may be directly transmitted from the camera 12 to each viewpoint buffer device 32 through the analog line 20 such as NTSC and temporarily stored. Each viewpoint buffer device 32 is a so-called temporary storage device and is connected to the LAN line 18. Each viewpoint buffer device 32 can send the image data through the communication control device 30 to the viewpoint conversion composite image generation / display device 16 via the LAN line 18 by A / D conversion processing on the temporarily stored image data and packet transmission. Therefore, the selected viewpoint buffer device 32 sends the image data to the viewpoint conversion composite image generation / display device 16 by calling the buffer selection device.

  With the configuration shown in FIG. 4, images from a plurality of cameras such as the front camera group can be taken into the buffer device in real time. This is because it is more advantageous to connect an analog line to a buffer device because an image can be transferred with a two-core NTSC video cable. Also, images can be transferred to the buffer device at the video rate. For this reason, it is possible to transmit an image separately from the LAN line 18, which is effective in speeding up transmission, saving wiring (mainly cable thickness), and suppressing the number of packets flowing through the entire LAN. In addition, since the wiring length is relatively short between the viewpoint buffer device 32 and the analog buffer, there is an advantage that the influence of noise can be minimized.

  Each pixel of the image data selectively fetched from the viewpoint buffer device 32 in this way is associated with a point in the three-dimensional space by the space reconstruction device 36 and reconstructed as space data. This calculates where each object constituting the selected image exists in the three-dimensional space, and temporarily stores the spatial data as a calculation result in the spatial data storage device 42. The viewpoint conversion means 43 reads the spatial data created by the space reconstruction device 36 from the storage device 42 and reconstructs the image viewed from the designated virtual viewpoint. This is an inverse conversion process of the process performed by the space reconstruction device 36. As a result, an image viewed from a new conversion viewpoint is generated by the data read from the spatial data storage device 42, temporarily stored in the viewpoint conversion image data storage device 44, and then displayed as a viewpoint conversion image on the display device 46. Will be displayed. In this embodiment, since the image data packet from the imaging means necessary according to the movement of the virtual viewpoint in the viewpoint converted image is preferentially acquired from the viewpoint buffer device 32 or the camera buffer device 14, there is no extra data processing, In this embodiment, the image composition processing speed is increased, and there is a high effect in application to a moving body such as a vehicle that requires immediacy.

  According to such an image generation apparatus, a plurality of cameras 12 and the viewpoint conversion composite image generation / display apparatus 16 are LAN-connected via the buffer apparatuses 14 and 32, and are uniquely determined for each set virtual viewpoint. Necessary image data is selected from the buffer devices 14 and 32, and is quickly captured by packet transmission, and is displayed after being synthesized, so that the image display speed is increased and the synthesized image can be displayed quickly. An extremely excellent image generating apparatus can be obtained.

  In the above-described embodiment, the application example to the vehicle 10 has been described. However, this is installed in a building for monitoring space inside the building, for example, monitoring of a store, monitoring of an absent room, monitoring of a road, etc. It is also possible. Furthermore, it can be applied to a wheelchair or the like, or can be equipped on a person's clothes or other attachments to be used for monitoring around the moving space.

  The image generation apparatus according to the present invention can display peripheral information outside the vehicle as an image viewed from a virtual viewpoint different from the camera viewpoint on a display device installed in the driver's seat of the vehicle, and for safety security It can be used as a monitoring device.

1 is a configuration block diagram in a case where an image generation device according to an embodiment is installed in a vehicle. It is a block diagram of the camera structure as an imaging means. It is a block diagram of an image generation / display apparatus. It is a principal part block diagram of the image generation / display apparatus which concerns on other embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ......... Vehicle, 12 ......... Camera, 12F ......... Front camera group, 12R ......... Back camera group, 14 ......... Camera buffer device, 16 ......... Viewpoint conversion composite image generation / display device, 18 ......... LAN line, 20 ......... Analog line, 24 ......... ID addition means, 241 ......... Time stamp, 242 ......... Imaging means position / orientation information, 243 ......... Imaging means internal parameters, 244 ... ... exposure information, 26 ......... packet generation means, 28 ......... continuous transmission control means, 30 ......... communication control device, 32 ......... viewpoint buffer device, 32F ......... front camera group buffer device, 32R ... ... rear camera group buffer device 34 ......... buffer selection device 36 ......... spatial reconstruction device 38 ......... alignment means 40 ... image data storage device 42 ... spatial data storage device 3 ......... viewpoint conversion means 44 ......... viewpoint conversion image data storage device, 46 ......... display device, 48 ......... obstacle detection unit, 50 ......... priority read control means.

Claims (7)

In an image generation method for generating a viewpoint conversion image from a virtual viewpoint using image information obtained by one or a plurality of imaging means arranged on an imaging means arranged object,
Necessary captured images for different virtual viewpoints are acquired in advance from individual imaging means and temporarily stored, and corresponding temporarily stored captured images are selected according to virtual viewpoint switching, and viewpoint converted images are generated and output for display. An image generation method characterized in that In an image generation device that generates a viewpoint conversion image from a virtual viewpoint using image information obtained by one or a plurality of imaging means arranged on an imaging means arranged object,
Temporary storage means for acquiring captured images necessary for each different virtual viewpoint in advance from individual imaging means and temporarily storing them, and temporary storage selection means for selecting temporary storage means having corresponding captured images according to virtual viewpoint switching And an image generation / display apparatus characterized in that a viewpoint conversion image generation means for generating a viewpoint conversion image from the captured image of the temporary storage means selected immediately and an output display of the generated viewpoint conversion image.   3. The image generation and display apparatus according to claim 2, wherein the temporary storage means is grouped for each virtual viewpoint and is temporarily stored after being synchronized.   3. The image generation and display apparatus according to claim 2, wherein a temporary storage unit corresponding to the number of viewpoints to be preset is secured to speed up the viewpoint change display.   3. The image generation display according to claim 2, wherein the imaging unit and the temporary storage unit are connected by an analog line, and the temporary storage unit and the viewpoint conversion image generation unit are connected by a digital line. apparatus.   6. The image generation and display apparatus according to claim 5, wherein the digital line is an in-vehicle LAN.   The image generating apparatus according to claim 2, wherein the imaging unit arranged object is at least one of a vehicle, a building, and a human body wearing tool.

Images