JP2009193221A - Omnidirectional image generation method, program and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a continuous omnidirectional image even from images wherein only partial areas overlap. <P>SOLUTION: A step for calculating presence probability of a photographic object at a point on each layer from similarity between corresponding points on the images corresponding to the point on each layer has steps: for performing division into an area wherein the presence probability of the photographic object can be obtained and an area wherein the presence probability of the photographic object cannot be obtained; for calculating the presence probability of the photographic object on each layer from the similarity between corresponding points on two or more images in the area wherein the presence probability of the photographic object can be obtained; for blending the presence probability of the photographic object in an area constant from a boundary with the area wherein the presence probability of the photographic object cannot be obtained in the area wherein the probability of the photographic object can be obtained; and for interpolating the presence probability of the photographic object on each layer of the area wherein the presence probability of the photographic object cannot be obtained base on the presence probability of the photographic object on each layer of the boundary with the area wherein the presence probability of the photographic object cannot be obtained in the area wherein the presence probability of the photographic object can be obtained. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an omnidirectional image generation method, a program, and a recording medium, and in particular, an omnidirectional image centered at an arbitrary viewpoint using a plurality of images shot in different directions so that parts of the images overlap. It relates to the technology to generate.

As one of the methods for generating an omnidirectional image, there is known a method for generating an image by estimating the probability that a subject exists at each point in a three-dimensional space and weighting color information according to the value. Yes. (See Patent Document 1 below)
Hereinafter, a conventional omnidirectional image generation method will be described.
FIG. 1 is a diagram showing a geometric configuration of a conventional omnidirectional image generation method.
In FIG. 1, 101 is the center of an omnidirectional image, 102 is the center of a camera lens, 103 is an image photographed by the camera, 104 is a reliability map indicating the probability that a subject exists at a point on the layer, and 105 is on the layer. It is a color map showing the color information in a point.
FIG. 2 is a flowchart for explaining a conventional omnidirectional image generation method.
The processing procedure for generating an omnidirectional image includes step 1 in which the position and orientation of the camera are changed so that there are overlapping areas, and a plurality of images are acquired. Step 2 for setting the position, number, and shape, Step 3 for determining the relationship between the corresponding points on each image corresponding to the points on the layer, and the points on the layer from the similarity of the corresponding points determined in Step 3 Step 4 of calculating the existence probability of the subject and blending the pixel values of the corresponding points on the corresponding image according to the positional relationship between the points on the layer, the corresponding points on the corresponding image, and the center of the omnidirectional image As a result, the color information of each layer in which the color information obtained in step 5 is rendered at the ratio of the existence probability of the subject obtained in step 5 and the step 5 in which the color information of the points on the layer is calculated is added. It consists Step 6 for output as the omnidirectional image.

As prior art documents related to the invention of the present application, there are the following.
JP 2007-257100 A

However, in the above-described conventional technology, when converting a camera image into an omnidirectional image, viewpoint conversion is performed using depth information of a subject to obtain a continuous omnidirectional image. Therefore, in order to generate an omnidirectional image, information on distance is indispensable.
As described above, in the conventional technique, when there is no corresponding point on the two camera images corresponding to the point on the layer, the probability that the subject exists cannot be obtained from the similarity of the corresponding point on the two camera images. Therefore, the existence right rate of the subject is set to 1 at all points. At these points, since the depth of the subject is not taken into consideration, it is difficult to generate a continuous omnidirectional image from a plurality of images obtained by extracting a subject having a depth.
Further, when the depth information is to be obtained at all points on the layer, a plurality of cameras or a plurality of images are required so that all the points of the subject are photographed with two or more images.
The present invention has been made in order to solve the above-described problems of the prior art, and the object of the present invention is to provide a continuous image generation method even in an omnidirectional image generation method, even from images that overlap only a part of regions. The object is to provide a technique capable of generating an omnidirectional image.
Another object of the present invention is to provide a program for causing a computer to execute the above-described omnidirectional image generation method.
Another object of the present invention is to provide a recording medium on which the aforementioned program is recorded.
The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings.

Of the inventions disclosed in this application, the outline of typical ones will be briefly described as follows.
The present invention creates a omnidirectional image centered on an arbitrary viewpoint from a plurality of images that are shot in different directions using a camera whose internal parameters are known and whose camera position and orientation are known at the time of shooting. A method for generating an omnidirectional image, in which the camera position / orientation is changed so that a part of the image overlaps to acquire a plurality of images, and a layer position necessary for generating the omnidirectional image, Step 2 for setting the number and shape, Step 3 for obtaining the relationship between the corresponding points on each image corresponding to the points on each layer, and the points on each layer from the similarity of the corresponding points obtained in Step 3 Step 4 for calculating the existence probability of the subject, and the corresponding point pixel on each image according to the positional relationship between the point on each layer, the corresponding point on each image, and the center of the omnidirectional image To blend values In step 5 for calculating the color information of the points on each layer, the color information of each layer obtained by rendering the color information obtained in step 5 is added at the ratio of the presence rate of the subject obtained in step 4; Output as an omnidirectional image, and the step 4 includes a region where the existence probability of the subject can be obtained based on the corresponding points on the image corresponding to the points on each layer, and the presence of the subject In the step 4-1 where the probability cannot be obtained and the region in which the existence probability of the subject can be obtained, the similarity between the corresponding points on the two or more images corresponding to the points on each layer is calculated. Step 4-2 for calculating the existence probability of the subject on each layer, and a region where the existence probability of the subject cannot be obtained in an area where the existence probability of the subject can be obtained. Step 4-3 for blending the existence probability of the subject in a certain region from the boundary between the subject and the subject in the region where the existence probability of the subject can be obtained in the region where the existence probability of the subject cannot be obtained Step 4-4 for interpolating the existence probabilities of the subjects on the plurality of layers based on the existence probabilities of the subjects on each layer at the boundary with the region where the existence probability cannot be obtained.

In the present invention, the region where the subject existence probability cannot be obtained is defined as the first region, the region where the subject existence probability can be obtained is defined as the second region, and the first region is the second region. The step 4-4 determines that the subject exists at the boundary of the second area with the first area from among the plurality of layers of the first area. Selecting a layer at an intermediate position between a layer having the highest probability and a layer having the highest probability of existence of the subject at the boundary between the third region and the first region; and each layer of the first region The above-mentioned subject existence probability includes a step of assigning a presence probability of 1 to the selected layer and a presence probability of 0 to the other layers.
In the invention, the area where the existence probability of the subject cannot be obtained is the first area, the areas where the existence probability of the subject can be obtained are the second and third areas, and the first area is the second area. When located between the third region and the third region, the step 4-4 determines the existence probability of the subject at the boundary between the second region and the first region from among the plurality of layers of the first region. Sequentially selecting a layer located between a layer having the highest probability of existence of the subject at a boundary between the third region and the first region of the third region; and And a step of giving a presence probability of 1 to the selected layer and a presence probability of 0 to the other layers as the existence probability of the subject on each layer.
In the present invention, the region where the subject existence probability cannot be obtained is defined as the first region, the region where the subject existence probability can be obtained is defined as the second region, and the first region is the second region. And the third area, the step 4-4 determines the existence probability of the subject in each layer at the boundary between the first area and the second area as the second area. The step of providing the existence probability of the subject in each layer at the boundary with the first area, and the existence probability of the subject in each layer at the boundary with the third area in the first area, Assigning the existence probability of the subject in each layer at the boundary with the first region; and in each layer between the boundary with the second region and the boundary with the third region in the first region As the subject's existence probability, The subject is continuously changed from the existence probability of the subject in each layer at the boundary between the first region and the second region to the existence probability of the subject at each layer in the boundary between the first region and the third region. Providing existence probability.

In the present invention, the region where the subject existence probability cannot be obtained is defined as a first region, the region where the subject existence probability can be obtained is defined as a second region, and the first region is adjacent to the two regions. If it is, the step 4-4 selects a layer having the highest probability of existence of the subject at the boundary of the second region with the first region from the plurality of layers of the first region. And, as the existence probability of the subject on each layer of the first region, 1 is given to the selected layer, and 0 is given to the other layers.
In the present invention, the region where the subject existence probability cannot be obtained is defined as a first region, the region where the subject existence probability can be obtained is defined as a second region, and the first region is adjacent to the two regions. In the step 4-4, the existence probability of the subject in each layer at the boundary between the second region and the first region is set as the existence probability of the subject in each layer in the first region. Give.
In the present invention, a region where the existence probability of the subject cannot be obtained is a first region, and a certain region from a boundary with the first region in a region where the existence probability of the subject can be obtained is a fourth region, When the other area is the fifth area, the step 4-3 selects a predetermined number of layers from the plurality of layers of the first area in descending order of the existence probability of the subject before blending. The layer selected in the first step based on the existence probability of the subject before blending the layer selected in the first step as the existence probability of the subject on each layer of the first step and the first region The second step of assigning a new existence probability of the subject to the object and assigning an existence probability of 0 to a layer other than the layer selected in the first step. DOO has, in the first step, reducing the number of layers for sequentially selecting toward a boundary between the first region of the fourth region from the boundary between the fourth region and the fifth region.
In the present invention, a region where the existence probability of the subject cannot be obtained is a first region, and a certain region from a boundary with the first region in a region where the existence probability of the subject can be obtained is a fourth region, When the other region is the fifth region, in step 4-3, the existence probability of the subject of each layer in the fourth region is determined from the existence probability of the boundary of the fourth region with the fifth region. An existence probability that is continuously changed up to the existence probability of the boundary between the fourth area and the first area is given.
The present invention is also a program for causing a computer to execute the above-described omnidirectional image generation method.
The present invention is also a computer-readable recording medium on which the program is recorded.

The effects obtained by the representative ones of the inventions disclosed in the present application will be briefly described as follows.
According to the omnidirectional image generation method of the present invention, it is possible to generate a continuous omnidirectional image even from an image in which only a part of regions overlap.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
In all the drawings for explaining the embodiments, parts having the same functions are given the same reference numerals, and repeated explanation thereof is omitted.
[Example 1]
The overall method for generating an omnidirectional image according to the first embodiment of the present invention is basically the same as that shown in FIG.
Hereinafter, the omnidirectional image generation method of the present embodiment will be described with reference to FIG.
In the first embodiment, as shown in FIG. 2, step 1 in which the position and orientation of the camera are changed so that a part of the images overlap to acquire a plurality of images, and layers necessary for generating an omnidirectional image Step 2 for setting the position, number, and shape of the image, Step 3 for determining the relationship between the corresponding points on each image corresponding to the points on the layer, and the points on the layer from the similarity of the corresponding points determined in Step The pixel value of the corresponding point on the corresponding image is calculated according to the positional relationship between the point on the layer, the corresponding point on the layer, the corresponding point on the image, and the center of the omnidirectional image. By blending, the color information of each layer in which the color information obtained in step 5 is added at the ratio of the existence probability of the subject obtained in step 5 and step 4 where the color information of the points on the layer is calculated is added. ,Every direction It consisted of the steps 6 and outputs it as an image.
Hereinafter, detailed processing in each step will be described.

(4-4) Step 4-4
In the region V _k , there is only one image having a corresponding point corresponding to a point on the layer. Therefore, the subject existence probability cannot be obtained from the similarity of the corresponding point.
Therefore, the existence probability of the subject in the region V _k is interpolated using the existence probability of the subject at the boundary between the region U _k−1 and the region U _k .
In this embodiment, as shown in Method 1 in FIG. 8, it is assumed that the subject exists in the middle of the layer having the highest subject existence probability at the boundary between the region U _k−1 and the region U _k . The existence probability of all points on the layer at the intermediate position is set to 1, and the existence probability of points on the other layers is set to 0.
In the method 1 of FIG. 8, the existence probability (here, 1) of the third layer from the front is the largest at the left end boundary (the right end of the region U _k-1 ), and the right end boundary (the left end of the region U _k ), The existence probability (here, 1) of the sixth layer from the front (second from the back) is the largest.
The layers at the intermediate positions are the fourth and fifth layers. In Method 1 in FIG. 8, the existence probabilities of all points on the fourth layer with a small difference in the existence probabilities at both ends are set to 1. The existence probability of points on other layers is set to zero. In the method 1 of FIG. 8, the existence probability of all points on the fifth layer may be 1, and the existence probability of points on the other layers may be 0.
However, since the regions V ₁ and V _N are regions at the ends of the omnidirectional image that do not exist between the regions U _k , as shown in the method 1 in FIG. Here, it is assumed that a subject exists in the third layer from the front), and the existence probabilities of subjects at all points on the layer are set to 1, and the existence probabilities of subjects on other layers are set to 0.

[Example 2]
Since the overall method for generating an omnidirectional image according to the second embodiment is the same as the method described in the first embodiment, detailed description thereof is omitted.
The omnidirectional image generation of the second embodiment is different from the first embodiment in the method of calculating the existence probability of the subject at the point on the layer in step 4.
In the first embodiment, only when there are two or more images having corresponding points corresponding to all points on the layer in a straight line extending from the center of the omnidirectional image to the layer, the similarity between the corresponding points on the image is obtained. The existence probability of the subject is obtained. Therefore, although the degree of similarity is obtained at some points on the straight line, the value is not used.
Thus, in the second embodiment, the existence probability of the subject is interpolated using the similarity obtained at some points on the layer in the straight line extending from the center of the omnidirectional image to the layer.
Hereinafter, the process of step 4 in Example 2 is demonstrated.

(4) Step 4
In the second embodiment, step 4 is a step of dividing the area according to the number of images having corresponding points corresponding to points on a straight line extending from the center of the omnidirectional image toward the layer as shown in FIG. 4-1 and in a region where two or more images having corresponding points exist at all points on the straight line, step 4 for calculating the existence probability of the subject from the similarity of the corresponding points of the two or more images 2 and step 4-3 for calculating the existence probability of the subject from the similarity of the corresponding points of the two or more images in an area where there are two or more images having corresponding points at some points on the straight line; In a region where there are two or more images having corresponding points at some points on the straight line, a step 4-4 for performing a blending process to continuously connect the boundaries of the regions, and all the points on the straight line Step 4-5 for interpolating the existence probability of a subject, assuming that the subject exists in the middle position of the layer where the probability that the subject exists at the boundary is highest in a region where only one image having a corresponding point exists. Consists of
FIG. 5 is a flowchart showing the processing procedure of the reliability map calculation method of the omnidirectional image generation method according to the second embodiment of the present invention. Hereinafter, detailed processing of each step will be described.

(4-5) Step 4-5
In the region V _k , there is only one image having a corresponding point corresponding to a point on the layer. Therefore, the existence probability of the subject cannot be obtained from the similarity of the corresponding point. Therefore, the existence probability of the subject in the region V _k is interpolated using the existence probability of the subject at the boundary between the region W _k−1 and the region W _k .
In this embodiment, as shown in Method 1 in FIG. 8, it is assumed that the subject exists in the middle of the layer having the highest subject existence probability at the boundary between the region W _k−1 and the region W _k . The existence probability of all points on the layer at the intermediate position is set to 1, and the existence probability of points on the other layers is set to 0.
However, since the regions V ₁ and V _N are regions at the ends of the omnidirectional image that do not exist between the regions U _k , as shown in the method 1 in FIG. Here, it is assumed that a subject exists in the third layer from the front), and the existence probabilities of subjects at all points on the layer are set to 1, and the existence probabilities of subjects on other layers are set to 0.

Note that steps 4-4 of Example 1, the interpolation method for the presence probability of an object region V _k in Step 4-5 of Example 2, it is also possible to use the procedure of Example 4.
As described above, according to the present embodiment, surrounding depth information can be estimated in a region where depth information cannot be estimated from a plurality of camera images captured so that a part of the subject overlaps. By interpolating the depth information from the region, it is possible to generate a continuous omnidirectional image.
In addition, since it is only necessary to shoot with a plurality of cameras so that a part of the area of the subject overlaps, all points of the subject are always omnidirectional compared to a case where shooting is performed so that two or more cameras always overlap. The number of cameras or the number of images necessary for generating an image can be reduced.
The omnidirectional image generation method described above can also be executed by a computer. In that case, the omnidirectional image generation method can be executed by a computer executing a program stored in a hard disk or the like in the computer. Done. This program is supplied by downloading via a CD-ROM or a network.
As mentioned above, the invention made by the present inventor has been specifically described based on the above embodiments. However, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the scope of the invention. Of course.

It is a schematic diagram for demonstrating the conventional omnidirectional image generation method. It is a flowchart which shows the process sequence of the omnidirectional image generation method of the Example of this invention, and the conventional omnidirectional image generation method. It is a flowchart which shows the process sequence of the calculation method of the reliability map of the omnidirectional image generation method of Example 1 of this invention. It is a schematic diagram for demonstrating the area division | segmentation of the reliability map in step 4-1 of FIG. It is a flowchart which shows the process sequence of the calculation method of the reliability map of the omnidirectional image generation method of Example 2 of this invention. FIG. 6 is a schematic diagram for explaining region division of a reliability map in step 4-1 in FIG. 5. In each example of the present invention, it is a schematic view for explaining a blending method of the existence probability of an object in the region U _k and the region W _k. It is a schematic diagram for demonstrating the interpolation method of the subject's existence probability in the area _| region Vk in each Example of this invention. In each example of the present invention, it is a schematic diagram for explaining an interpolation method of the existence probability of an object in the region V ₁ and V _n.

Explanation of symbols

101 Center of Omnidirectional Image 102 Center of Camera Lens 103 Image 104 Reliability Map 105 Color Map

Claims (10)

An omnidirectional image that creates an omnidirectional image centered on an arbitrary viewpoint from a plurality of images that have been shot with different internal parameters and the camera position and orientation at the time of shooting. A generation method,
Step 1 for obtaining a plurality of images by changing the position / orientation of the camera so that a part of the images overlaps;
Step 2 for setting the position, number, and shape of layers necessary for generating an omnidirectional image;
Step 3 for obtaining a relationship with a corresponding point on each image corresponding to a point on each layer;
Step 4 for calculating the existence probability of the subject at a point on each layer from the similarity of the corresponding points obtained in Step 3;
By blending the pixel value of the corresponding point on each image according to the positional relationship with the point on each layer, the corresponding point on each image, and the center of the omnidirectional image, Step 5 for calculating point color information;
Step 6 adds the color information of each layer obtained by rendering the color information obtained in Step 5 at the ratio of the subject presence rate obtained in Step 4 and outputs it as an omnidirectional image.
The step 4 is divided into a region where the existence probability of the subject can be obtained and a region where the existence probability of the subject cannot be obtained based on the corresponding points on the image corresponding to the points on each layer. -1,
Step 4-2 for calculating the existence probability of the subject on each layer from the similarity of corresponding points on two or more images corresponding to the points on each layer in the region where the existence probability of the subject can be obtained When,
Blending the existence probability of the subject in a certain area from the boundary with the area where the existence probability of the subject cannot be obtained in the area where the existence probability of the subject can be obtained; and
In the region where the existence probability of the subject cannot be obtained, the existence probability of the subject on each layer at the boundary with the region where the existence probability of the subject cannot be obtained in the region where the existence probability of the subject can be obtained. And a step 4-4 for interpolating the existence probabilities of subjects on a plurality of layers. The area where the existence probability of the subject cannot be obtained is a first area, the areas where the existence probability of the subject can be obtained are second and third areas, and the first area is the second area and the third area. If it is located between
In the step 4-4, the layer having the highest probability of existence of the subject at the boundary between the second region and the first region among the plurality of layers of the first region; Selecting a layer at an intermediate position between the layer having the highest probability of existence of the subject at the boundary with one region;
Adding the existence probability of 1 to the selected layer and the existence probability of 0 to the other layers as the existence probability of the subject on each layer of the first region. The omnidirectional image generation method according to claim 1. The area where the existence probability of the subject cannot be obtained is a first area, the areas where the existence probability of the subject can be obtained are second and third areas, and the first area is the second area and the third area. If it is located between
In the step 4-4, the layer having the highest probability of existence of the subject at the boundary between the second region and the first region among the plurality of layers of the first region; Sequentially and successively selecting layers positioned between the layer having the highest probability of existence of the subject at the boundary with one region;
Adding the existence probability of 1 to the selected layer and the existence probability of 0 to the other layers as the existence probability of the subject on each layer of the first region. The omnidirectional image generation method according to claim 1. The area where the existence probability of the subject cannot be obtained is a first area, the areas where the existence probability of the subject can be obtained are second and third areas, and the first area is the second area and the third area. If it is located between
In the step 4-4, the existence of the subject in each layer at the boundary between the first region and the first region is determined as the existence probability of the subject at each layer at the boundary between the first region and the second region. Assigning a probability;
Assigning the existence probability of the subject in each layer at the boundary between the first area and the first area as the existence probability of the subject at each layer at the boundary between the first area and the third area;
Each layer at the boundary between the first region and the second region as the probability of existence of the subject in each layer between the boundary between the second region and the boundary between the third region in the first region Providing the existence probability that is continuously changed from the existence probability of the subject to the existence probability of the subject in each layer at the boundary between the first area and the third area. The omnidirectional image generation method according to claim 1. When the region where the existence probability of the subject cannot be obtained is the first region, the region where the existence probability of the subject can be obtained is the second region, and the first region is located next to the two regions,
The step 4-4 selecting a layer having the highest probability of existence of the subject at the boundary between the second region and the first region from the plurality of layers of the first region;
Adding the existence probability of 1 to the selected layer and the existence probability of 0 to the other layers as the existence probability of the subject on each layer of the first region. The omnidirectional image generation method according to claim 1. When the region where the existence probability of the subject cannot be obtained is the first region, the region where the existence probability of the subject can be obtained is the second region, and the first region is located next to the two regions,
In step 4-4, the existence probability of the subject of each layer at the boundary between the second region and the first region is given as the existence probability of the subject of each layer in the first region. The omnidirectional image generation method according to claim 1. The region where the existence probability of the subject cannot be obtained is the first region, the constant region from the boundary with the first region in the region where the existence probability of the subject can be obtained is the fourth region, and the other regions are the first region. When making 5 areas,
The step 4-3 selects a predetermined number of layers from the plurality of layers of the first region in descending order of the existence probability of the subject before blending;
The existence probability of the subject on each layer of the first region is a new probability for the layer selected in the first step based on the existence probability of the subject before blending of the layer selected in the first step. A second step of giving a presence probability of the subject and giving a zero existence probability to a layer other than the layer selected in the first step;
2. The number of layers sequentially selected from the boundary between the fourth area and the fifth area toward the boundary between the fourth area and the first area is decreased in the first step. The omnidirectional image generation method described in 1. The region where the existence probability of the subject cannot be obtained is the first region, the constant region from the boundary with the first region in the region where the existence probability of the subject can be obtained is the fourth region, and the other regions are the first region. When making 5 areas,
In step 4-3, as the existence probability of the subject in each layer in the fourth area, the boundary between the fourth area and the first area is determined based on the existence probability of the boundary between the fourth area and the fifth area. The omnidirectional image generation method according to claim 1, wherein an existence probability that is continuously changed to an existence probability is assigned.   A program for causing a computer to execute the omnidirectional image generation method according to any one of claims 1 to 8.   A computer-readable recording medium on which the program according to claim 9 is recorded.

Images