JP4218307B2 - Omnidirectional image generation method and apparatus - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an omnidirectional image generation method. In particular, the present invention relates to an omnidirectional image generation apparatus that generates an omnidirectional image from images taken simultaneously by a plurality of cameras.
[0002]
[Prior art]
With the spread of systems that display high-resolution video and the spread of VR content that can move the viewpoint, the demand for high-resolution content is increasing. Since the output image size can be freely set for the CG image, it is easy to create a high-resolution image. However, the resolution of the obtained image is limited because there are restrictions on the photographing apparatus. In addition, in order to obtain high-resolution images at the same time, a very expensive photographing device is required, and film scanning for digitizing the captured data is not easy in terms of cost and time. I can't get an image.
[0003]
As a method of obtaining an image of the entire surroundings, a wide viewing angle can be taken at once using a fisheye lens without using multiple cameras, or all directions can be taken at once with a single camera using a convex mirror. There is a way to do it. However, the resolution with a single camera is limited, and it is not suitable for high-resolution content video.
[0004]
Therefore, as a method of obtaining a real high-resolution image using a general photographing apparatus, there are a method of joining images taken by a plurality of cameras and a method of joining images taken by a moving camera. However, many methods for creating a high-resolution panoramic image have been proposed, but there are still few methods for creating a seamless all-around image. In addition, although several proposals have been made at the still image level for the creation of an omnidirectional image, there are few that support moving images.
[0005]
For example, “Ambient camera system, method for generating an ambient image based on an image captured by an ambient camera, connection processing apparatus and method for an image captured by an adjacent camera, and a distance measuring device using an adjacent camera” "Method" describes a device that obtains a 360 ° image in both the vertical and horizontal directions by mounting a camera on each component surface of a polyhedron such as a regular dodecahedron. Describes how to do it. Connection of images is not easy because the projection centers of the cameras do not match, but here the boundary between images is adjusted by dynamically adjusting the connection position between adjacent captured images according to the perspective of the captured subject. It uses a method that eliminates discontinuities and joints in the vicinity and generates a smooth landscape. This makes it possible to connect adjacent images without a sense of incongruity, but this method only describes the continuity of geometric information such as line segments at joints, and does not mention color tone at all. Absent. This causes a seam due to a difference in color tone near the boundary between adjacent images.
[0006]
Further, for example, “Recording and Reproducing Method of Image Information” in Patent Document 2 describes a method of connecting and recording videos captured by a plurality of imaging devices to form and record and reproduce one moving image information. In particular, a method is described in which an image connected to one image is reduced and separated into an even field and an odd field in an interlace to generate data of two frames, and encode and record in the MPEG2 system. The data created here can be reproduced by using a reproduction apparatus that can reproduce a two-frame standard television image signal. This method is a method of creating an image specialized for a playback apparatus. Regarding the connection between adjacent images, only the method of pasting in a cylindrical shape is mentioned, and a smooth connection cannot be expected with this method alone.
[0007]
[Patent Document 1]
JP 2001-204015 A [Patent Document 2]
Japanese Patent Laid-Open No. 2002-112184
[Problems to be solved by the invention]
As described above, among the conventional omnidirectional image generation systems, among the image generation methods using the splicing of images taken by a plurality of imaging devices, a method in which a joint is not conspicuous at any position of 360 ° There were few proposals.
[0009]
[Means for Solving the Problems]
The present invention was devised in view of the above-mentioned problems, and its purpose is to make the joints conspicuous at any position of 360 ° when creating a surrounding image by connecting a plurality of images. It is to provide an apparatus and method that is not.
[0010]
Therefore, the present invention employs the following means.
[0011]
In an omnidirectional image generation device capable of capturing photographing data obtained from a plurality of photographing devices connected to a photographing control device, the photographing control device controls photographing timings of the plurality of photographing devices and photographs all surrounding images. Means for transforming the image by performing columnar or spherical mapping of the captured image data, means for selecting both-end images in order to rearrange the positions of the plurality of image data,
By having means for correcting the color tone of a plurality of image data, means for performing geometric correction of the plurality of image data, and means for connecting the corrected image data after performing the above deformation and correction,
Provided is an omnidirectional image generation device that can be connected together in a sense of incongruity at all connection portions around the entire periphery.
[0012]
The image data handled by the present invention may be a still image, a moving image, or both.
[0013]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 is a diagram showing a system configuration of the present embodiment.
[0014]
This system includes a CPU (1) that performs calculation and control based on a program, a main storage device (100) such as a memory that stores data and programs, and an auxiliary storage device (200) of a hard disk.
[0015]
In addition, the photographing device (10), the photographing control device (20) for controlling photographing timing, the D / A converter (30) for converting image data photographed in analog to digital data, and these components are mutually coupled. It consists of a bus (2).
[0016]
In order to capture an image of the entire periphery with high resolution, a plurality of imaging devices (10) are provided, and the plurality of imaging devices (10) are connected to the imaging control device (20). The photographing control device (20) controls the photographing timing of the photographing device (10).
[0017]
Moreover, when using the imaging device (10) which can obtain a digital image directly, it is not necessary to provide a D / A converter (30).
[0018]
The auxiliary storage device (200) includes data relating to the installation method of the photographing device (10), for example, internal parameters such as the number and positional relationship of the photographing devices (10), the focal length and the angle of view of each photographing device (10). , A camera parameter (210), a photographing data storage unit (220) for storing images photographed by a plurality of photographing devices (10), and a 360 ° jointed all-around image without any sense of incongruity. An image deformation data storage unit (230) for storing image data obtained by performing image deformation such as spherical or columnar mapping on the image data (220) captured in this manner, and a joint between adjacent images in all images A corrected image data storage unit (240) for storing image data that has been subjected to color tone correction or geometric deformation so that the image is not conspicuous, and the corrected image data are joined together Composed of all around the image data storage unit (250) which can be obtained by.
[0019]
The main storage device (100) includes a photographing unit (110) and an all-around image generation unit (120).
[0020]
In the photographing unit (110), a timing control program (111) for controlling timing related to photographing such as simultaneous photographing with a plurality of photographing devices (10), the number and positional relationship data of photographing devices (10) provided in plural, and photographing. Includes a camera parameter acquisition program (112) for acquiring internal parameters such as the focal length and angle of view of the device (10), and an all-around shooting program (113) for simultaneously shooting all-around video with a plurality of shooting devices (10). It is.
[0021]
The omnidirectional image generation unit (120) includes a captured image acquisition program (121) for capturing images captured by a plurality of imaging devices (10), and a 360 ° jointed omnidirectional image without any sense of incongruity. In order to create an image deformation program (122) for performing image deformation such as spherical or columnar mapping on the image data (220) captured by the image capturing apparatus, and to create a 360 ° all-around image without any sense of incongruity, Both-end image selection program (123) for selecting which image is to be joined at both ends, a color tone correction program (124) for correcting a difference in color tone of an acquired image due to individual film and lens differences of each photographing device (10), adjacent Correction program (125) for performing fine correction and performing fine adjustment so that the images to be joined can be joined together completely without any sense of incongruity, and the corrected data An omnidirectional image generation program (126) for generating an omnidirectional image by connecting the images is included.
[0022]
First, in the photographing unit (110), omnidirectional images are photographed using a plurality of photographing devices (10). At this time, the photographing timing is controlled by the photographing control device (20). Also, external parameters such as the number of the photographing devices (10) and installation positions, and internal parameters such as an angle of view and a focal length are acquired and stored as camera parameters (210). At this time, in order to perform shooting that completely covers the entire periphery using a plurality of imaging devices, the number and installation positions of the imaging are determined in consideration of the angle of view and internal parameters of the imaging device, the size of the imaging device, It is necessary to decide. Furthermore, in order to facilitate the connection of adjacent images, it is necessary to make adjustments so that the joining portions of the respective images obtained from adjacent photographing devices overlap. The data photographed by the photographing device (10) is converted from analog data to digital data through an A / D converter (30) and stored as photographing data (220). At this time, when photographing is performed using the photographing apparatus (10) capable of directly obtaining digital data, A / D conversion is not necessary and the photographing data (220) is stored as it is.
[0023]
Using the angle-of-view information of the photographing apparatus (10), which is one of the camera parameters (210), the photographing data (220) is deformed by the image deformation program (122) to generate the image deformation data (230). To do. As an image deformation method, a column surface mapping that produces an effect of pasting an image on a column surface, or a spherical mapping method that produces an effect of pasting an image on a spherical surface is used. Thereby, the uncomfortable feeling of continuity of the image connection portion is eliminated. When images are connected only in the horizontal direction, there is no problem with columnar mapping. However, when not only horizontal but also vertical connection is required, spherical mapping is preferable.
[0024]
When creating an all-around image by joining multiple images, not only adjacent images but also both ends of the image, that is, the right end of the image at the right end and the left end of the image at the left end of the image Must also be continuous. Otherwise, for example, when looking around the entire surrounding image, there may be one joint. Therefore, which image should be installed at both ends is determined by the both-end image selection program (123). The images placed at both ends should be the portion with the least change when connected later. For example, since the color tone of the image obtained by the individual difference of the lens and film of the imaging device (10) is different, the imaging device (10) having a film that is closest to the color tone of the obtained image is adjacent. The image may be installed as a double-ended image. For example, when shooting all around the outdoors outdoors during the day, there is a possibility that sunlight may enter any position. If image information cannot be obtained due to halation occurring and flying white, that portion may be installed as both-end images. Conversely, when the photographing apparatus (10) is installed, it may be installed so that the sunlight is intentionally at the connection portion of the image. When both end images are selected, the entire image is first rearranged. Then, geometric correction and color tone correction are performed so that the right end portion of the image at the first right end and the left end portion of the image at the first left end are completely joined together. Regarding the color tone correction, it is desirable to unify the color tone of the images at both ends to the average color tone of the entire image.
[0025]
Next, in the color correction program (124), the entire color tone is unified. Since the color tone of the images at both ends is fixed, the color tone is corrected so that the color tone of the adjacent images in order from the end image is the same. By connecting the right end portion of the image at the first right end and the left end portion of the image at the left end in advance, it is only necessary to worry about color correction of the image in the middle portion. In order to reduce the color tone difference between images, the automatic correction function of the photographing apparatus is not used during photographing, and the parameters of all photographing apparatuses are set to be constant manually.
[0026]
Next, in the geometric correction program (125), geometric correction is performed so that the connected portions of adjacent images are completely connected. With reference to the line segment information of the object existing in the connection part, the line segment is deformed so as to be continuously connected in all the places of the connection part.
[0027]
The corrected image data (240) is created and stored through the above three steps (123) (124) (125).
[0028]
The all-around image data (250) is created by connecting all the corrected image data (240) by the all-around image creation program (126).
[0029]
FIG. 3 is an example of installation of the photographing apparatus. This is an example in which nine photographing devices (10) are arranged at intervals of 40 ° on a cylinder. The photographing surface (12) is determined by the photographing range (11) and the focal length obtained from the angle of view of each photographing device (10). At this time, in order to facilitate joining later, the imaging surfaces (12) of the adjacent imaging devices (10) are slightly overlapped.
[0030]
FIG. 4 is an image modification example. Column surface mapping or spherical mapping is performed so that the connected portions are continuously connected. Taking columnar mapping as an example, A is a captured image as it is, but it is necessary to transform each image so that it becomes B. To do this, change image C to image D. May be made. Image D is a modification when the angle of view of the photographing apparatus (10) is 60 °.
[0031]
FIG. 5 is a flowchart showing the flow of processing in the present embodiment.
[0032]
First, in step 1000, the photographing control device (20) instructs the photographing start timing, and in step 1100, photographing using a plurality of photographing devices (10) is performed simultaneously. In step 1200, the photographing control apparatus (20) issues an instruction to end photographing. In step 1300, photographing data photographed by all photographing apparatuses is taken in.
[0033]
In step 2000, a joining method is selected. When performing stitching only in the horizontal direction with reference to the installation position data of the photographing apparatus, which is one of the camera parameters (210), in the case of performing stitching in both the horizontal direction and the vertical method in Step 2100. Advances to step 2000. In step 2100, image transformation by columnar mapping is performed on the captured image data, and in step 2200, image transformation by spherical mapping is performed on the captured image data. Thereafter, image processing is performed based on the image deformation data.
[0034]
In step 3000, images that become both ends when joining are selected in advance. The selection of both-end images will be described in detail with reference to FIG.
In step 4000, the entire color tone is unified by correcting the color tone of all the images. Since the color tone of the images at both ends is fixed, the color tone is corrected so that the color tone of the adjacent images in order from the end image is the same. By connecting the right end portion of the image at the first right end and the left end portion of the image at the left end in advance, it is only necessary to worry about color correction of the image in the middle portion.
[0035]
In step 5000, geometric correction is performed so that the connected portions of adjacent images are completely connected. With reference to the line segment information of the object existing in the connection part, the line segment is deformed so as to be continuously connected in all the places of the connection part.
[0036]
In step 6000, all the corrected images are connected to create an all-around image.
[0037]
FIG. 6 is a flowchart showing the flow of both-end image selection when performing stitching.
[0038]
Even if the all-around image is created by joining, the data will be possessed as a panoramic image, so there will always be images at both ends. For this reason, since image correction is facilitated by installing image pairs that are considered to have less correction processing at both ends, it is selected which adjacent image pair is installed at both ends.
[0039]
In step 3100, it is checked whether or not there is a connection portion with extremely small image information. For example, when taking a picture outdoors on a sunny day, the sun may enter any position and cause halation to fly white, or the shadow may become dark. If these phenomena occur in the connection portion, the process proceeds to step 3300. If not, the process proceeds to step 3200.
[0040]
In step 3200, it is confirmed whether or not there is a portion whose color tone does not change in the adjacent image. The color tone information is important because the color tone of the image obtained by the individual difference of the lens and film of the photographing apparatus (10) is different, and the color tone correction is applied to the entire image. If there is an image pair having an approximate color tone in the adjacent images, the process proceeds to step 3400, and if not, the process proceeds to step 3500.
[0041]
In step 3300, two images having a connection portion with little image information are set as both-end images. Even if the two images are separated and moved to both ends, the correction of the both end images can be reduced.
[0042]
In Step 3400, two adjacent image pairs whose color tone does not change are set as both-end images.
[0043]
In step 3500, a pair of adjacent images in which the change in color tone and the distortion of the object at the connected portion are minimized are searched for as both-end images.
[0044]
In step 3600, the average value of the color tones of all images is calculated.
[0045]
In step 3700, the right end of the rightmost image and the left end of the leftmost image are connected in advance. Based on the average color tone calculated in step 3600, color adjustment is performed.
[0046]
Finally, in step 3800, the images are rearranged. Assume that there are images arranged in the order of [1,2,3,4,5,6,7,8], assuming that images are taken by eight photographing devices (10). In order to move the image pair selected in Step 3300 or Step 3400 or Step 3500 to both ends, for example, if [5] [6] is selected, [6,7,8,1,2,3,4 , 5], and correction is performed while maintaining this order.
[0047]
【The invention's effect】
According to the present invention, it is possible to provide an apparatus and method in which a joint is not conspicuous at any position of 360 ° when creating an image of the entire periphery by connecting a plurality of images.
[Brief description of the drawings]
FIG. 1 is a system configuration diagram of an omnidirectional image generation apparatus according to a first embodiment.
FIG. 2 is an internal configuration diagram of the omnidirectional image generation apparatus according to the first embodiment.
FIG. 3 shows an installation example of the photographing apparatus according to the first embodiment.
FIG. 4 is an image modification example of the first embodiment.
FIG. 5 is a basic flowchart of the omnidirectional image generation apparatus according to the first embodiment.
FIG. 6 is a flowchart of both-end image selection according to the first embodiment.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... CPU, 2 ... Bus, 10 ... Imaging device, 11 ... Imaging range, 12 ... Imaging surface, 20 ... Imaging control device, 30 ... A / D converter , 100 ... Main storage device, 110 ... Shooting unit, 111 ... Timing control program, 112 ... Camera parameter acquisition program, 113 ... All-around shooting program, 120 ... All-around image generation , 121... Photographic image acquisition program, 122 ... image transformation program, 123 ... both-end image selection program, 124 ... color tone correction program, 125 ... geometric correction program, 126 ... all-around image Generating program, 200... Auxiliary storage device, 210... Camera parameter storage unit, 220 .. photographing data storage unit, 230... Image deformation data storage unit, 240. Rear image data storage section, 250 ... all around the image data storage unit

Claims (8)

In the omnidirectional image generation device capable of capturing image data obtained from a plurality of imaging devices connected to the imaging control device,
Means for controlling photographing timing of a plurality of photographing devices in the photographing control device and photographing images of the entire periphery;
Means for transforming the image by performing columnar or spherical mapping of the captured image data;
Means for selecting a pair of image data having a similar color tone among adjacent pairs of image data, and rearranging the positions of the plurality of image data using the pair of image data having a similar color tone as both end images;
Means for correcting color tone of a plurality of image data;
Means for geometric correction of a plurality of image data;
An omnidirectional image generating apparatus comprising means for stitching together corrected image data after performing the above deformation and correction. The omnidirectional image generating device according to claim 1,
There are multiple shooting devices,
An omnidirectional image generating apparatus characterized in that installation is performed so that all surrounding images can be taken. In an omnidirectional image generation method capable of capturing image data obtained from a plurality of imaging devices connected to an imaging control device,
The shooting control device controls shooting timings of a plurality of shooting devices to take images of the entire periphery,
Deform the image by performing columnar or spherical mapping of the captured image data,
Select an image data pair that approximates the color tone among adjacent image data pairs,
Rearranging the positions of the plurality of image data with the image data pairs having approximate color tones as both-end images,
Perform color correction on multiple image data,
Perform geometric correction of multiple image data,
An omnidirectional image generation method characterized by stitching together corrected image data after performing the above deformation and correction. The omnidirectional image generation method according to claim 3,
An omnidirectional image generation method characterized in that a plurality of imaging devices are provided and installation is performed so as to capture all images around the entire periphery.   Claim 1 Or an omnidirectional image generating apparatus according to 2, wherein
  Means for calculating an average color tone of the plurality of image data;
  The omnidirectional image generating apparatus according to claim 1, wherein the color tone correcting unit performs color tone correction of the plurality of image data based on an average value of the color tones.   Claim 1 Or an omnidirectional image generating apparatus according to 2 or 5,
  The omnidirectional image generating apparatus characterized in that the color tone correcting unit corrects the color tone of adjacent image data in order from the edge image data among the plurality of image data after rearrangement.   The omnidirectional image generation method according to claim 3 or 4,
  Calculating an average color tone of the plurality of image data;
  A color tone correction of the plurality of image data is performed based on the average value of the color tone. Direction image generation method.   The omnidirectional image generation method according to claim 3, 4 or 7,
  An omnidirectional image generation method characterized by correcting the color tone of adjacent image data in order from the edge image data among a plurality of image data after rearrangement.

Images