WO2006018951A1 - 画像生成方法および画像生成装置 - Google Patents
画像生成方法および画像生成装置 Download PDFInfo
- Publication number
- WO2006018951A1 WO2006018951A1 PCT/JP2005/013605 JP2005013605W WO2006018951A1 WO 2006018951 A1 WO2006018951 A1 WO 2006018951A1 JP 2005013605 W JP2005013605 W JP 2005013605W WO 2006018951 A1 WO2006018951 A1 WO 2006018951A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- viewpoint
- imaging
- image generation
- unit
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims description 22
- 238000003384 imaging method Methods 0.000 claims abstract description 108
- 230000008859 change Effects 0.000 claims abstract description 5
- 230000004044 response Effects 0.000 claims abstract description 3
- 238000006243 chemical reaction Methods 0.000 claims description 66
- 230000005540 biological transmission Effects 0.000 claims description 21
- 238000004891 communication Methods 0.000 claims description 20
- 230000006835 compression Effects 0.000 claims description 15
- 238000007906 compression Methods 0.000 claims description 15
- 230000001360 synchronised effect Effects 0.000 claims description 4
- 238000013507 mapping Methods 0.000 claims description 2
- 238000012545 processing Methods 0.000 abstract description 23
- 239000002131 composite material Substances 0.000 description 32
- 238000010586 diagram Methods 0.000 description 22
- 238000012544 monitoring process Methods 0.000 description 20
- 238000013500 data storage Methods 0.000 description 13
- 238000004364 calculation method Methods 0.000 description 5
- 238000005259 measurement Methods 0.000 description 5
- 238000004590 computer program Methods 0.000 description 3
- 230000007423 decrease Effects 0.000 description 3
- 238000004378 air conditioning Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000009434 installation Methods 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 238000012806 monitoring device Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 1
- 239000013256 coordination polymer Substances 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000006866 deterioration Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000004904 shortening Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
- B60R1/20—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
- B60R1/22—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle
- B60R1/28—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with an adjustable field of view
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
- G06T15/20—Perspective computation
- G06T15/205—Image-based rendering
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/243—Image signal generators using stereoscopic image cameras using three or more 2D image sensors
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/10—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used
- B60R2300/105—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used using multiple cameras
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/60—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective
Definitions
- the present invention relates to an image generation method and an image generation device, and in particular, based on images picked up by a plurality of image pickup devices, as if actually shooting a viewpoint power different from that of the image pickup device.
- the present invention relates to a technique for combining and displaying an image with a changed viewpoint.
- a configuration is adopted in which a captured image of each camera is displayed on a monitor, and a captured image from a camera attached to a desired location in a monitoring area is input to a monitoring room. It is displayed on multiple arranged monitors.
- a camera mounted on the vehicle, using a camera directed to the rear of the vehicle, the area that the driver cannot see directly or indirectly is photographed and displayed on a monitor provided in the driver's seat for safe driving. Make a contribution.
- image data obtained by a plurality of cameras is centrally captured and three-dimensionally generated in advance by laser radar, millimeter wave radar, triangulation using a stereo camera, or the like.
- a spatial model is generated for each pixel that forms the input image from the camera based on the acquired image data based on the camera parameters. Mapping is performed by associating information to create spatial data.
- a viewpoint-converted image with an arbitrary virtual viewpoint power is generated instead of a real camera viewpoint. Displayed.
- Such a viewpoint-converted image display method has an advantage that the entire area of the monitoring area can be displayed without any deterioration in image accuracy, and the area to be monitored can be confirmed from any viewpoint.
- Patent Document 1 Japanese Patent Publication: JP 05-310078 A
- Patent Document 2 Japanese Patent Publication: JP-A-10-164566
- Patent Document 3 Japanese Patent Gazette: Japanese Patent No. 3286306
- the input of the image data from the camera is performed collectively by the image calculation processing unit, so that the wiring becomes long and becomes multiple wiring, so that the space is effectively used. It was an obstacle to do. In particular, when it is applied when monitoring the surroundings of a vehicle, it has been an obstacle when trying to make effective use of the space of the wire harness stretched around the vehicle. In addition, it is necessary to perform processing with a large amount of data, particularly image data, quickly, but in order to capture all the image data from all powers and collect all the data corresponding to points in a three-dimensional space, When it comes to wasteful processing of data that is not actually used, there was a problem.
- the present invention focuses on the above-described conventional problems, and can reduce the wiring space even when an imaging device such as a large number of cameras is used, and can quickly perform image processing for viewpoint conversion and the like without waste.
- An object of the present invention is to provide an image generation apparatus capable of performing the above.
- the present invention is based on the viewpoint that, even when a large number of imaging devices are used, when the viewpoint-converted image viewed from the virtual viewpoint is synthesized, the image data to be used and thus the imaging device to be used are determined.
- the image data of the imaging device is collected into the buffer together with the image data used for the imaging device unit or the virtual viewpoint unit, and when the virtual viewpoint is set, only the image data corresponding to the virtual viewpoint is immediately obtained. This is obtained from the knowledge that the processing can be quickly performed by selecting and performing image composition processing. That is, for example, in order to generate a front viewpoint conversion image, a read command is read from the front camera image buffer device.
- the image generation method generates a viewpoint-converted image from a virtual viewpoint using image information obtained by one or a plurality of imaging devices arranged on an imaging device arrangement object.
- a captured image necessary for each different virtual viewpoint is acquired in advance from each imaging device and temporarily stored, and a corresponding temporarily stored captured image is obtained according to the virtual viewpoint switching. It was configured to generate a viewpoint-converted image and display it.
- the image generation apparatus generates an image that generates a viewpoint-converted image from a virtual viewpoint using image information obtained by one or a plurality of imaging devices arranged on an imaging device arrangement object.
- a temporary storage unit that acquires and temporarily stores the captured image necessary for each different virtual viewpoint in advance and temporarily stores a temporary storage unit that has a corresponding captured image according to the virtual viewpoint switching.
- the memory selection unit, the captured image power of the temporary storage unit selected immediately, the viewpoint conversion image generation unit that generates the viewpoint conversion image, and the generated viewpoint conversion image are output and displayed.
- the temporary storage unit secures a temporary storage unit corresponding to the number of viewpoints to be preset, which can be grouped and synchronized for each virtual viewpoint and temporarily stored. It is only necessary to speed up the display of changes. Further, the image pickup apparatus and the temporary storage unit may be connected via an analog line, and the temporary storage unit and the viewpoint conversion image generation unit may be connected via a digital line. Can be an in-vehicle LAN.
- the imaging device arrangement object may be at least one of a vehicle, a building, and a human body wearing tool.
- an image generation apparatus converts viewpoints by one or a plurality of imaging devices arranged on an imaging device arrangement object and a captured image taken by the imaging device.
- An image generation unit for generating an image and the imaging device arranged object It is characterized in that it is connected by a LAN provided in the above-mentioned apparatus, and the captured image of the imaging apparatus can be transmitted in packets to the image reproduction unit.
- the configuration includes an ID addition unit that adds an ID to an image captured by the imaging device, a packet generation unit that packetizes the generated image with ID, and a continuous transmission control unit for the packet. That's fine.
- the imaging apparatus preferably includes a communication control unit that performs communication by adding an ID to a captured image.
- the ID should include at least one of a time stamp, imaging device position and orientation information, imaging device internal parameters, and exposure information.
- a selection unit that preferentially obtains image data packets of necessary imaging device power according to the movement of the virtual viewpoint in the viewpoint-converted image may be provided. Furthermore, it is possible to provide an alignment unit that arranges images captured by a plurality of image pickup devices in time series based on the I blueprint and a storage unit that stores them in time series.
- An imaging device capable of imaging an obstacle recognized in the viewpoint conversion image is specified, and a control unit that preferentially reads image data of the imaging device and outputs the image data to the display unit is provided. be able to.
- the present invention provides a spatial reconstruction unit that maps a captured image captured by an imaging device placed on an imaging device placed object to a predetermined spatial model of a three-dimensional space, and the space Based on the spatial data mapped by the reconstruction unit, a viewpoint conversion unit that generates image data viewed from an arbitrary virtual viewpoint in the three-dimensional space, and based on the image data generated by the viewpoint conversion unit,
- An image generation device comprising a display unit that displays an image viewed from any virtual viewpoint power in a three-dimensional space, and identifies an imaging device capable of displaying an obstacle recognized in the space model or the viewpoint conversion image
- the control unit that preferentially reads out the captured image of the imaging apparatus to generate a viewpoint conversion image and outputs it to the display unit. It is characterized by having
- An image generation apparatus comprising a data transmission unit, and the image CP U is an image generating apparatus characterized in that at least one of the following items (1) to (10) is variable based on an image output of at least one of the plurality of cameras.
- the imaging device arrangement may be at least one of a vehicle, a building, and an attachment to a person.
- FIG. 1 is a block diagram showing a configuration when an image generating apparatus according to a first embodiment is installed in a vehicle.
- FIG. 2 is a block diagram of a camera configuration as an imaging apparatus.
- FIG. 3 is a block diagram of a viewpoint conversion composite image generation Z display device according to the first embodiment.
- FIG. 4 is a principal block diagram of a viewpoint conversion composite image generation Z display device according to a second embodiment.
- FIG. 5 is a block diagram showing a configuration when an image generating apparatus according to a third embodiment is installed in a vehicle.
- FIG. 6 is a block diagram of a viewpoint conversion composite image generation Z display device according to a third embodiment.
- FIG. 7 is a diagram for explaining the operation of the image generation apparatus according to the third embodiment.
- FIG. 8 is a system block diagram of an image generation apparatus according to a fourth embodiment.
- FIG. 9 is a system block diagram of an image generation apparatus according to a fifth embodiment.
- FIG. 10 is a system block diagram of an image generation apparatus according to a sixth embodiment.
- FIG. 11 is a system block diagram of an image generation apparatus according to a seventh embodiment.
- FIG. 1 is a block diagram of a configuration in which the image generation apparatus according to the first embodiment is mounted on a vehicle and configured to monitor a surrounding situation for assistance during driving of the vehicle.
- a plurality of cameras 12 as imaging devices are provided at the front and rear of a vehicle 10 as an imaging device arrangement object.
- the front camera group 12F (12FR, 12FC, 12FL) is mounted on the front of the vehicle 10, and each of the cameras 12FR, 12FC, 12FL is provided in front of the vehicle 10.
- An image is taken at 45 degrees on the right side, 45 degrees on the center, and 45 degrees on the left side.
- a rear camera group 12R (12RR, 12RC, 12RL) as an imaging device is also provided on the rear part of the vehicle 10, and each camera 12RR, 12RC, 12RL is similarly oriented 45 degrees to the right on the rear side of the vehicle 10. , Center, left 45 degrees.
- the image force obtained by the front camera group 12F can be generated and displayed by synthesizing and generating the vehicle front image viewed from the virtual viewpoint set above the vehicle 10, and obtained by the rear camera group 12R.
- the rear image of the vehicle in which the image power and the other virtual viewpoint force set above the vehicle 10 are also seen can be synthesized and displayed.
- the vehicle 10 is equipped with a viewpoint conversion composite image generation Z display device 16 that synthesizes an image captured by the camera 12 from an arbitrary viewpoint different from the viewpoint of the camera 12. ing.
- This viewpoint conversion composite image generation Z display device 16 has the power to input image data from each camera 12FR, 12FC, 12FL, 12RR, 12RC, 12RL. This is not directly input to each camera 12FR, 12FC,
- Each camera buffer unit 14FR, 14FC, 14FL, 14RR, 14RC, 14RL and viewpoint buffer unit 32 front camera group buffer unit 32F or rear camera group buffer set to 12FL, 12RR, 12RC, 12RL units The input is made via the key device 32R).
- These camera buffer devices 14FR, 14FC, 14FL, 14RR, 14RC, 14RL are the images that can be stored temporarily for each camera 12FR, 12FC, 12FL, 12RR, 12RC, 12RL Transmitted image data An ID is attached to. Further, the above-described viewpoint conversion composite image generation Z display device 16 and the camera buffer devices 14FR, 14FC, 14FL, 14RR, 14RC, and 14RL are connected by an in-vehicle LAN line 18 and have a so-called LAN connection configuration. Each camera 12FR, 12FC, 12FL, 12RR, 12RC, 12RL and the camera buffer device 14FR, 14FC, 14FL, 14RR, 14RC, 14RL are connected via an analog line 20.
- FIG. 2 is a block diagram of a camera configuration as an imaging apparatus.
- This image generation device packet transmits camera image data 14FR, 14FC, 14FL, 14RR, 14RC, 14R L for each camera 12FR, 12FC, 12FL, 12R R, 12RC, 12RL via LAN line 18
- the camera buffer device 14FR attached to the camera 12FR as an imaging device is an ID adding unit for adding an ID to a captured image. 24, a packet generation unit 26 for packetizing the generated ID-attached image, and a continuous transmission control unit 28 for the packet.
- the ID added to the captured image data unit by the ID adding unit 24 includes at least one shooting information of the time stamp 241, the imaging device position / orientation information 242, the imaging device internal parameter 243, and the exposure information 244. ing.
- the image data sent from the camera FR is given an ID and includes the shooting information such as the time stamp 241 and the like, from the camera buffer device 14FR via the LAN line 18, and by the communication control unit 29.
- the packet is continuously transmitted to the viewpoint conversion composite image generation Z display device 16.
- the basic processing in the viewpoint conversion composite image generation Z display device 16 is performed by inputting images taken from the viewpoints of the cameras 1 2FR, 12FC, 12FL, 12RR, 12RC, and 12RL.
- the three-dimensional space to be placed is set, this three-dimensional space is defined by an arbitrarily set origin (virtual viewpoint), and the pixels of the image data are coordinate-converted into the three-dimensional space viewed from the specified virtual viewpoint. And rearrange the pixels on the image plane viewed from the virtual viewpoint. This makes the pixel of the image data obtained from the camera viewpoint into a virtual viewpoint. Therefore, an image synthesized by rearranging in the defined three-dimensional space can be obtained, and a synthesized image from a desired viewpoint other than the camera viewpoint can be created, output, and displayed.
- the first embodiment in the first embodiment according to the present invention, necessary images are temporarily stored together for each virtual viewpoint unit specified in advance, and when a viewpoint conversion is requested, the corresponding viewpoint is displayed. Only the corresponding data is read in a batch, and other data is not read, so that the data processing speed is increased.
- the camera 12FR, 12FC, 12FL, 12RR, 12RC, 12RL from which the image data should be acquired is uniquely determined by the set virtual viewpoint.
- FIG. 3 is a block diagram of the viewpoint conversion composite image generation Z display device according to the first embodiment.
- a necessary imaging device (camera 12FR, 12F C, 12FL, 12RR, 12RC, 12RL) image data packets input via the communication control device 30 are connected via the LAN line 18 to the viewpoint buffer device 32 (front camera group buffer device 32F or rear camera group buffer device 32R).
- an image selection device 34 for preferentially acquiring image data corresponding to the virtual viewpoint specified from now on. For example, when a front image of the vehicle 10 is synthesized from a virtual viewpoint set above the vehicle 10, the image data from the rear camera group 12R is not necessary. Therefore, the image selection device 34 generates an image selection command in accordance with the designation of the virtual viewpoint.
- the image selection device 34 is separate from the image selection from the viewpoint buffer devices 32F and 32R that temporarily store necessary data in preset virtual viewpoint units.
- the image data can be directly acquired from the camera 12 via the communication control device 30 so that necessary image data can be selected to generate an image at an arbitrary virtual viewpoint.
- the captured image is transmitted from the camera 12 to the camera buffer device 14 via the analog line 20.
- the force melano buffer device 14 and later are temporarily transmitted to the notifier device 32 by packet communication in units of image data with ID. Since it is stored in the HD, it is possible to combine image data at the same time using HD information. Therefore, the viewpoint conversion composite image generation Z display device 16 includes an image alignment device 38 that arranges captured images (image data) from a plurality of cameras 12 in time series based on HD information, and the image data in time series. An image data storage device 40 is provided. Also, if the parameters of the acquired image data are not synchronized, the composite image will be far from the entity.
- the ID includes at least one of the time stamp 241, the imaging device position / orientation information 242, the imaging device internal parameter 243, and the exposure information 244, and image data to be pasted in the three-dimensional space as necessary. Try to make mutual adjustments.
- FIG. 4 is a principal block diagram of the viewpoint conversion composite image generation Z display device according to the second embodiment.
- the camera buffer device 14 attached to the camera 12 is also configured to transmit data to the viewpoint buffer device 32 (front or rear camera image buffer devices 32F and 32R) via the LAN line 18.
- video data may be transmitted directly from the camera 12 to the front camera group buffer device 32F or the rear camera group buffer device 32R through the analog line 20 such as NTSC, and stored.
- the front camera group buffer device 32F or the rear camera group buffer device 32R is a so-called temporary storage device and is connected to the LAN line 18.
- the front camera group buffer device 32F or the rear camera group buffer device 32R performs the AZD conversion processing on the temporarily stored image data and transmits the packet to the viewpoint conversion composite image generation Z display device 16 via the LAN line 18 through the control device 30.
- Image data can be sent. Therefore, when the image selection device 34 is called, the selected front camera group buffer device 32F or rear camera group buffer device 32R sends the image data to the viewpoint conversion composite image generation Z display device 16.
- images from a plurality of cameras such as the front camera group 12F can be taken into the buffer device in real time. This is more advantageous when connected to the viewing buffer device 32 via an analog line because images can be transferred via a 2-core NTSC video cable.
- the image can be transferred to the viewpoint buffer device 32 at the video rate. For this reason, the image can be transmitted separately from the LAN line 18, which is effective in increasing the transmission speed, reducing the wiring (mainly the cable thickness), and reducing the number of packets flowing through the entire LAN.
- the wiring length is relatively short between the viewpoint buffer device 32 and the analog buffer, there is an advantage that the influence of noise can be minimized.
- Each pixel of the image data selectively captured from the viewpoint buffer device 32 in this way is associated with a point in the three-dimensional space by the space reconstruction device 36 and reconstructed as space data. .
- This calculates the force where each object constituting the selected image exists in the three-dimensional space, and temporarily stores the spatial data as the calculation result in the spatial data storage device 42.
- the viewpoint conversion device 43 reads the spatial data created by the spatial reconstruction device 36 from the spatial data storage device 42 and reconstructs an image viewed from the designated virtual viewpoint. This is an inverse conversion process of the process performed by the space reconstruction device 36.
- a new converted viewpoint power image is generated from the data read from the spatial data storage device 42, temporarily stored in the viewpoint conversion image data storage device 44, and then displayed on the display device 46.
- the image data packet from the camera 12 as an imaging device required according to the movement of the virtual viewpoint in the viewpoint converted image is preferentially acquired from the viewpoint buffer device 32 or the camera buffer device 14. Therefore, unnecessary data processing is eliminated, and in the first and second embodiments, the image composition processing speed is increased, and it is highly effective when applied to a moving object such as a vehicle that requires immediacy. .
- a plurality of cameras 12 and the viewpoint conversion composite image generation Z display device 16 are set by LAN connection via the camera buffer device 14 and the viewpoint buffer device 32. Necessary image data uniquely determined for each virtual viewpoint is selected from the camera buffer device 14 and the viewpoint buffer device 32, quickly captured by packet transmission, and displayed after image synthesis. Increases speed and speeds up composite images Therefore, an extremely excellent image generating apparatus can be obtained.
- the application example to the vehicle 10 has been described.
- This is a monitoring space inside a building, for example, monitoring a store, monitoring an absent room, monitoring on a road, etc. It is also possible to install the building for the purpose. Furthermore, it can be applied to wheelchairs, etc., or it can be used for monitoring around the moving space by mounting on human clothes and other attachments.
- the images taken by the respective imaging devices are packetized to a temporary storage unit. Since the image generation unit can select and store image data in a temporary storage unit corresponding to a predetermined virtual viewpoint, the image generation unit can store the virtual viewpoint.
- the image data required for each virtual viewpoint stored in the temporary storage unit can be collectively selected by selecting the temporary storage unit, and the necessary images corresponding to the virtual viewpoint can be selected. Only the data can be selected and extracted, and the composite image can be reconstructed from the coordinate conversion image.
- FIG. 5 is a block diagram showing a configuration in which the image generation apparatus according to the third embodiment is mounted on a vehicle so that the surrounding situation can be monitored for assistance during vehicle operation.
- the image generation apparatus according to the third embodiment shown in FIG. 5 further includes an operation control device 54 and an air conditioning control device 56. Yes.
- FIG. 6 is a block diagram of a viewpoint conversion composite image generation Z display device according to the third embodiment.
- the viewpoint conversion composite image generation Z display device 16 has the priority readout control device 50.
- the camera 12 as the imaging device capable of displaying the obstacle 58 recognized on the viewpoint-converted image is specified, and the image data of the camera 12 is preferentially read to display the display device.
- the output to 46 is made.
- a plurality of cameras 12 and the viewpoint conversion composite image generation Z display apparatus 16 are set by LAN connection via the camera buffer apparatus 14 and the viewpoint buffer apparatus 32. Necessary image data that is uniquely determined for each virtual viewpoint is selected from the camera buffer device 14 and the viewpoint buffer device 32, is quickly captured by packet transmission, and is displayed after being synthesized. Since the display speed is increased and the composite image can be displayed quickly, an extremely excellent image generation apparatus can be obtained.
- the composite image generation unit that forms the viewpoint-converted composite image generation Z display device 16 has a force that can be configured by the image CPU 52.
- the image CPU 52 Based on the image output of at least one camera 12FR, 12FC, 12FL, 12RR, 12RC or 12R L, (1) designation of the imaging device to be used, (2) presence or absence of image compression, (3) image compression rate, (4 ) Number of output pixels, (5) Video frame rate, (6) Virtual viewpoint position and orientation, (7) Warning method, (8) Time trigger or event trigger communication method, (9) Time trigger In such a case, it is possible to make variable processing such as allocation time and order of each imaging device and (10) permission to transmit data other than images.
- FIG. 7 is a diagram for explaining the operation of the image generation apparatus according to the third embodiment.
- the third embodiment is an example in which a frame image is transmitted in a packet group.
- the image CPU 52 performs identification of the imaging device that images the obstacle 58, measurement of the obstacle 58, measurement, recognition, and the like based on information from the camera 12, which is at least one imaging device.
- the recognition of these moving objects for example, object recognition from the image based on the same technical means as the prior art, such as recognition of the obstacle 58 at the time of generating the spatial model described in Patent Document 3. It is sufficient to use this means.
- the power to image these recognition results (for example, a part of a space model showing a preceding vehicle) Adjust the camera 12 to calculate the camera parameter force, select it, adjust it to display the acquired image, and perform viewpoint conversion processing if necessary.
- FIG. 8 is a system block diagram of an image generation apparatus according to the fourth embodiment.
- the system of the image generation apparatus according to the fourth embodiment to which the present invention is applied is basically the same as that of the first to third embodiments described above, except that the space model generation apparatus 64 and the calibration apparatus 66 are provided.
- the system configuration provided is different.
- an ID is assigned to each captured image data unit of each camera 12, and the time stamp 241, calibration data as the imaging device position and orientation information 242, the imaging device internal parameters 243, and the exposure information 244 are stored. At least one is included.
- the image data sent from each camera 12FR, 12FC, 12FL, 12RR, 12RC, and 12RL is given an ID and includes the shooting information such as the time stamp 241.
- 14FC, 14FL, 14RR, 14RC, and 14RL are continuously sent to the viewpoint conversion composite image generation Z display device 16.
- the viewpoint conversion image generation Z display device 16 In the viewpoint conversion image generation Z display device 16 to which image data or the like is sent, a plurality of cameras 12F R, 12FC, 12FL, 12RR are selected according to the viewpoint conversion by the image selection device 34 serving as an imaging camera switching means. , 12RC, 12RL are switched.
- the above camera-captured images are temporarily stored in the camera buffer devices 14FR, 14FC, 14FL, 14RR, 14RC, and 14RL by packet communication in units of ID-attached image data. Can be combined. Therefore, the viewpoint conversion composite image generation Z display device 16 uses the image alignment device 38 based on the HD information to capture images (image data) from multiple cameras 12FR, 12FC, 12FL, 12RR, 12RC, and 12RL forces.
- An image data storage device 40 that arranges the image data in series and stores the image data in time series is provided. Also, if the parameters of the acquired image data are not synchronized, the composite image will be far from the actual situation. Therefore, as described above, the ID includes at least one of the time stamp 241, calibration data as the imaging device position / orientation information 242, imaging device internal parameters 243, and exposure information 244, and if necessary, three-dimensional Adjust the image data to be pasted in the space.
- a distance measuring device 60 for measuring the distance to the moving obstacle 58 is provided.
- the distance measuring device 60 may be configured to use distance measurement by laser radar, millimeter wave radar, or the like and distance measurement by stereo imaging.
- radar ranging a normal system that measures the distance based on the time difference between the transmitted signal and the reflected signal may be used.
- stereo imaging the same subject is photographed from a plurality of different viewpoints, the correspondence of the same point of the subject in these images is obtained, and the distance to the subject is calculated by the principle of triangulation. That's fine.
- the entire right image of an image captured by a stereo imaging device is divided into small areas to determine the range for stereo ranging calculation, and then the position of the image that is the same as the left image V, and calculate the positional difference between these images, and calculate the distance to the target object.
- the distance image data is generated based on the distance information obtained by the stereo distance measurement between two or more images captured by the stereo imaging device, and stored in the distance image data storage device 62.
- the viewpoint conversion image generation Z display device 16 is provided with a space model generation device 64.
- the spatial model generator 64 generates a spatial model using the image data, the distance image data from the distance measuring device 60, and the calibration data! /
- the calibration device 66 is for the imaging device 12 (stereo power camera unit) arranged in the three-dimensional real world. In the three-dimensional real world, the mounting position of the imaging device 12, the mounting angle, and lens distortion correction are performed. Determine and specify camera parameters that represent camera characteristics such as value and lens focal length. The camera parameters obtained by the calibration are stored in the calibration data storage device 48 as calibration data.
- the space model generation device 64 generates a space model using image data, distance image data, and calibration data.
- the generated spatial model is stored in the spatial model storage device 70.
- Each pixel of the image data that is selectively captured is associated with a point in the three-dimensional space by the space reconstruction device 36 and reconstructed as space data. This is to calculate the force where each object constituting the selected image exists in the three-dimensional space, and to store the spatial data as the calculation result in the spatial data storage device 42. Each of the calculations is This is performed for all the pixels of the image obtained from the image pickup device 12.
- the viewpoint conversion device 43 can convert the image data stored in the spatial data storage device 42 by the spatial reconstruction device 36 into an image in which an arbitrary viewpoint position force is expected, and an arbitrarily set viewpoint can be designated. That is, it specifies from what position in the three-dimensional coordinate system, at what angle, and at what magnification. As a result, an image viewed from a new conversion viewpoint is generated by the data read from the spatial data storage device 42, temporarily stored in the viewpoint conversion image data storage device 44, and then displayed on the display device 46. It will be displayed as a converted image.
- the viewpoint conversion image generation Z display device 16 is provided with an imaging device arrangement object model storage device 72 that stores and stores the vehicle model, and the vehicle model is stored when space reconstruction is performed. It can be displayed at the same time.
- a viewpoint selection device 74 is provided, and image data corresponding to a preset virtual viewpoint set in advance is stored in the virtual viewpoint data storage device 76, and immediately when the viewpoint selection processing is performed. The corresponding image is transmitted to the viewpoint conversion device 44, and the selection command is output from the image selection device 34 so that the converted image corresponding to the selected virtual viewpoint is displayed! /
- the system includes an object recognition device 78 for recognizing an object.
- the object recognition device 78 includes a spatial model (coordinate data of the generated three-dimensional space), and an imaging device arranged object. Recognize an object from the imaging device placement object model (including camera parameters), viewpoint conversion image data, distance image data, live-action image data, etc. stored in the model storage means 72, label it, and prioritize imaging Relative to the obstacle 58, it is calculated that it is approaching faster than the vector vector, and is set as the collision prediction degree. Based on this value, the virtual viewpoint to be photographed is selected, and the image selection device 34 is instructed. And the communication control device 30 may select an imaging device that preferentially transmits packets.
- the power of disclosing the implementation means by the device
- the priority of the recognized obstacle 58 that may be configured to be processed on a computer using a CPU or the like is shown.
- the image image CPU 52 sends a command to the communication control device 30 to determine the imaging device to be set and adopted for shooting the obstacle 58, and to transmit more images that increase the frequency of packet transmission.
- packet transmission The frequency may be kept low. That is, the number of packets transmitted from the image CPU 52 is variably controlled for each S camera 12 according to the priority of the obstacle 58.
- FIG. 9 is a system block diagram of an image generation apparatus according to the fifth embodiment.
- the system of the image generation apparatus includes an image compression rate control apparatus 80 in addition to the configuration shown in FIG.
- the image compression rate control device 80 includes an image pickup device that picks up an image with the obstacle 58 from the label of the obstacle 58 obtained by the object recognition device 78 based on the collision risk and the like. No, the image pickup device is picked up! /, And the image pickup device is determined, and whether or not to compress the image output from each image pickup device 12 is set for the communication control device 30. Further, as in the fourth embodiment, as an image area having no obstacle 58 to be recognized as a computer program composed of a CPU or the like, the image data is compressed and transmitted as a packet group. For image areas that have no compression! The image CPU 52 transmits an instruction to transmit the image data divided into packet groups to the communication control device 30. In other words, the number of packets is variably controlled depending on the presence or absence of compression.
- the compression rate is increased to reduce the number of packets required to transmit one image, and for the image of the area with the obstacle 58, more detailed processing is performed.
- the image CPU 52 transmits an instruction to transmit a large number of packets to the communication control device 30 so that the compression rate of the image data is kept low. As a result, the number of packets is variably controlled for each camera 12 depending on the compression level.
- FIG. 10 is a system block diagram of an image generation apparatus according to the sixth embodiment.
- FIG. 11 is a system block diagram of an image generation apparatus according to the seventh embodiment.
- the seventh embodiment shown in FIG. 11 is obtained by replacing the image compression rate control device 82 having the configuration shown in FIG. 10 with an image frame rate control device 84.
- the image frame rate control device 84 increases the imaging frame rate of the imaging device that detects the obstacle 58 for each imaging device, depending on the presence or absence of the obstacle 58, and the imaging device that does not detect the obstacle 58. Reduce the imaging frame rate to control the volume of image data and save the number of packets.
- the CPU instructs the camera 12 to set the video frame rate of the camera 12 as a computer program composed of CPUs, and sends packets to be allocated to communication. Increase or decrease.
- parameters related to increase / decrease of communication packets are mainly changed depending on the presence / absence of the obstacle 58, but the relative speed of the obstacle 58, that is, the obstacle 58 having a strong degree of approaching is included.
- the packet transmission assignment may be controlled so that the camera 12 is given priority, or the front camera group 12F in the traveling direction of the vehicle 10 is given priority. You can increase it.
- the image CPU 52 recognizes the position and driving situation of the obstacle 58 to be noticed, and changes the position and posture of the virtual viewpoint to a position that matches the position of the obstacle 58. Make instructions.
- Image of at least one camera 12FR, 12FC, 12FL, 12RR, 12RC, 12RL The image CPU52 implements the type of obstacle 58 determined by the camera and the route prediction. Controls the transmission frequency of the packet information for presenting instructions such as whether to issue an alarm, whether to issue a proximity warning, or not to issue a warning to the display device 46.
- an image obtained from at least one camera 12FR, 12FC, 12FL, 12RR, 12RC, 12RL is used to periodically transmit an image from the camera 12FR, 12FC, 12FL, 12RR, 12RC, 12RL.
- the mode to send images as a group of packets (time trigger)
- the change in the image of the camera 12 is detected and the mode to send in response to the detection (event trigger) is switched. carry out.
- the in-vehicle LAN line 18 includes information related to operation control of the vehicle 10, control of an air conditioner and an anti-fogging device, control information of car audio, streaming information, and sensors other than images.
- Input information flows as packets. For example, if the obstacle 58 is not detected, audio streaming may be transmitted as packet transmission permission information.
- Instructions such as switching to prioritize packet transmission of the camera 12 that captures the object 58 are sent from the image CPU 52 to the communication control devices of in-vehicle devices such as operation control devices 54 and air conditioning control devices 56 connected to various LANs. And control packet transmission (see Figure 5).
- a packet having the power of the camera 12FC capturing the obstacle 58 is preferentially transmitted.
- the power is described to increase the number of packets from one camera 12.
- a plurality of necessary cameras 12 are selected. Then, control is performed so that the amount of packets from them is equally adjusted. As a result, it is possible to efficiently transmit information to the viewpoint-converted image generation Z display device 16 by allocating information transmission to the LAN line 18 with reduced wiring and giving priority to necessary information.
- the force described in the application example to the vehicle 10 is used for monitoring space inside the building, for example, monitoring of a store, monitoring of an unoccupied room, monitoring of a road, etc. It is also possible to install in buildings. Furthermore, it can be applied to wheelchairs, etc., or it can be used to monitor the surroundings of a moving space by installing it on human clothes or other attachments.
- the connection cable is simplified. Even when multiple imaging devices are used, the cable length used is reduced and multiple Since it does not become a line, it can be set as the image generation apparatus which utilized space effectively. Also
- image data can be selected and combined in units of packets, and the necessary image data corresponding to the converted viewpoint It is possible to reconstruct a composite image using a coordinate conversion image by selecting and extracting only. This eliminates the need for wasteful data processing, and enables data conversion by selective use of only a portion where large image data is utilized, thereby shortening the processing time. In particular, when mounted on a high-speed moving body, the display delay of the composite image becomes fatal, whereas in the present invention, the processing capability can be greatly improved.
- the image generation method and the image generation apparatus according to the present invention The display device 46 installed in the driver's seat of the vehicle 10 can display the peripheral information outside the vehicle 10 as an image that also has a virtual viewpoint power different from the camera viewpoint, and as a monitoring device for safety guards. It can be used.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Mechanical Engineering (AREA)
- Computing Systems (AREA)
- Geometry (AREA)
- Computer Graphics (AREA)
- General Physics & Mathematics (AREA)
- Closed-Circuit Television Systems (AREA)
Abstract
Description
Claims
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2004238828A JP4608268B2 (ja) | 2004-08-18 | 2004-08-18 | 画像生成方法および装置 |
JP2004-238828 | 2004-08-18 | ||
JP2004333206A JP4647975B2 (ja) | 2004-11-17 | 2004-11-17 | 画像生成装置 |
JP2004-333206 | 2004-11-17 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2006018951A1 true WO2006018951A1 (ja) | 2006-02-23 |
Family
ID=35907342
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2005/013605 WO2006018951A1 (ja) | 2004-08-18 | 2005-07-25 | 画像生成方法および画像生成装置 |
Country Status (1)
Country | Link |
---|---|
WO (1) | WO2006018951A1 (ja) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2431226A1 (de) * | 2010-09-17 | 2012-03-21 | SMR Patents S.à.r.l. | Rückblickeinrichtung für ein Kraftfahrzeug |
CN104272715A (zh) * | 2012-05-01 | 2015-01-07 | 中央工程株式会社 | 立体相机及立体相机系统 |
EP2939877A4 (en) * | 2012-12-25 | 2016-08-24 | Kyocera Corp | CAMERA SYSTEM, CAMERA MODULE AND CAMERA CONTROL SYSTEM |
US10594989B2 (en) | 2011-09-16 | 2020-03-17 | SMR Patent S.à.r.l. | Safety mirror with telescoping head and motor vehicle |
US10638094B2 (en) | 2011-09-16 | 2020-04-28 | SMR PATENTS S.á.r.l. | Side rearview vision assembly with telescoping head |
EP3672226A3 (en) * | 2018-12-21 | 2020-08-26 | HERE Global B.V. | Method and apparatus for regulating resource consumption by one or more sensors of a sensor array |
US11445167B2 (en) | 2017-06-23 | 2022-09-13 | Canon Kabushiki Kaisha | Display control apparatus, display control method, and storage medium |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH1178692A (ja) * | 1997-09-03 | 1999-03-23 | Nissan Motor Co Ltd | 車両用映像提示装置 |
WO2000064175A1 (fr) * | 1999-04-16 | 2000-10-26 | Matsushita Electric Industrial Co., Ltd. | Dispositif de traitement d'images et systeme de surveillance |
JP2002324235A (ja) * | 2001-04-24 | 2002-11-08 | Matsushita Electric Ind Co Ltd | 車載カメラの画像合成表示方法及びその装置 |
-
2005
- 2005-07-25 WO PCT/JP2005/013605 patent/WO2006018951A1/ja active Application Filing
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH1178692A (ja) * | 1997-09-03 | 1999-03-23 | Nissan Motor Co Ltd | 車両用映像提示装置 |
WO2000064175A1 (fr) * | 1999-04-16 | 2000-10-26 | Matsushita Electric Industrial Co., Ltd. | Dispositif de traitement d'images et systeme de surveillance |
JP2002324235A (ja) * | 2001-04-24 | 2002-11-08 | Matsushita Electric Ind Co Ltd | 車載カメラの画像合成表示方法及びその装置 |
Cited By (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2431226A1 (de) * | 2010-09-17 | 2012-03-21 | SMR Patents S.à.r.l. | Rückblickeinrichtung für ein Kraftfahrzeug |
US10594989B2 (en) | 2011-09-16 | 2020-03-17 | SMR Patent S.à.r.l. | Safety mirror with telescoping head and motor vehicle |
US10638094B2 (en) | 2011-09-16 | 2020-04-28 | SMR PATENTS S.á.r.l. | Side rearview vision assembly with telescoping head |
CN104272715A (zh) * | 2012-05-01 | 2015-01-07 | 中央工程株式会社 | 立体相机及立体相机系统 |
EP2846531A4 (en) * | 2012-05-01 | 2015-12-02 | Central Engineering Co Ltd | STEREO CAMERA AND STEREO CAMERA SYSTEM |
EP2939877A4 (en) * | 2012-12-25 | 2016-08-24 | Kyocera Corp | CAMERA SYSTEM, CAMERA MODULE AND CAMERA CONTROL SYSTEM |
US10242254B2 (en) | 2012-12-25 | 2019-03-26 | Kyocera Corporation | Camera system for a vehicle, camera module, and method of controlling camera |
US11445167B2 (en) | 2017-06-23 | 2022-09-13 | Canon Kabushiki Kaisha | Display control apparatus, display control method, and storage medium |
EP3618429B1 (en) * | 2017-06-23 | 2023-09-20 | Canon Kabushiki Kaisha | Display control device, display control method, and program |
EP3672226A3 (en) * | 2018-12-21 | 2020-08-26 | HERE Global B.V. | Method and apparatus for regulating resource consumption by one or more sensors of a sensor array |
US10887169B2 (en) | 2018-12-21 | 2021-01-05 | Here Global B.V. | Method and apparatus for regulating resource consumption by one or more sensors of a sensor array |
US11290326B2 (en) | 2018-12-21 | 2022-03-29 | Here Global B.V. | Method and apparatus for regulating resource consumption by one or more sensors of a sensor array |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6633216B2 (ja) | 撮像装置、及び、電子機器 | |
US7457456B2 (en) | Image generation method and device | |
JP4744823B2 (ja) | 周辺監視装置および俯瞰画像表示方法 | |
US7386226B2 (en) | Stereo camera system and stereo optical module | |
WO2019192359A1 (zh) | 一种车载全景视频显示系统、方法及车载控制器 | |
US8155385B2 (en) | Image-processing system and image-processing method | |
WO2006018951A1 (ja) | 画像生成方法および画像生成装置 | |
WO2013047012A1 (ja) | 車両周辺監視装置 | |
WO2012172923A1 (ja) | 車両周辺監視装置 | |
WO2012169355A1 (ja) | 画像生成装置 | |
EP2631696B1 (en) | Image generator | |
EP1635138A1 (en) | Stereo optical module and stereo camera | |
JP4643860B2 (ja) | 車両用視覚支援装置及び支援方法 | |
JPWO2010137265A1 (ja) | 車両周囲監視装置 | |
JP5495071B2 (ja) | 車両周辺監視装置 | |
JP5516998B2 (ja) | 画像生成装置 | |
JP2010088096A (ja) | 車載用カメラユニット、車両外部ディスプレイ方法及びドライビングコリドーマーカー生成システム | |
EP3503531B1 (en) | Image display apparatus | |
JP4647975B2 (ja) | 画像生成装置 | |
JP2004064441A (ja) | 車載画像処理装置及び周囲モニタシステム | |
JP2006060425A (ja) | 画像生成方法および装置 | |
KR20160067507A (ko) | 차량 | |
KR20140080202A (ko) | 차량의 사각지대 감지 장치 및 방법 | |
JP2006033282A (ja) | 画像生成装置およびその方法 | |
WO2013094405A1 (ja) | 監視システム |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AK | Designated states |
Kind code of ref document: A1 Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BW BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE EG ES FI GB GD GE GH GM HR HU ID IL IN IS KE KG KM KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NA NG NI NO NZ OM PG PH PL PT RO RU SC SD SE SG SK SL SM SY TJ TM TN TR TT TZ UA UG US UZ VC VN YU ZA ZM ZW |
|
AL | Designated countries for regional patents |
Kind code of ref document: A1 Designated state(s): GM KE LS MW MZ NA SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LT LU LV MC NL PL PT RO SE SI SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application | ||
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |