JP2012173858A - Omnidirectional image generation method, image generation device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate an omnidirectional image of deep scene from multiple images.SOLUTION: An omnidirectional image generation method for generating an omnidirectional image of deep scene from multiple images includes: a step for acquiring images of a subject taken from multiple viewing locations; a step for projecting each of the acquired images on multiple predefined layer surfaces; a step for calculating color information or brightness information of the subject on the layer surfaces where the images are projected; a step for calculating reliability, which shows an existence probability of the subject at each point on the layer surfaces, by obtaining a similarity between each point of the subject on the acquired images and each point of images projected on the layer surfaces; a step for selecting multiple layers with a central focus on a layer having the highest reliability of the subject, based on the reliability; and a step for generating an omnidirectional image of deep scene based on the color information or brightness information of the subject on the surfaces of the selected layers and on reliability information.

Description

  The present invention relates to an omnidirectional image generation method, an image generation apparatus, and a program, and more particularly, to a technique for generating an omnidirectional image centered on an arbitrary viewpoint from a plurality of images captured by changing directions so as to overlap each other.

  In the case of generating omnidirectional images of a scene with a depth where a subject such as a person exists near the camera from a plurality of images, in order to continuously synthesize a plurality of images, the difference in appearance between the images, That is, it is necessary to synthesize them in consideration of subject depth information (three-dimensional information).

  However, it is difficult to estimate depth information (three-dimensional information) from a two-dimensional image, and various depth estimation methods have been proposed. One of them is a method of generating a high-quality three-dimensional image by expressing the captured scene as a multilayer layer and expressing the position where the subject exists on the layer as the reliability (for example, Patent Document 1). reference).

Japanese Patent No. 4052331

  However, in the method described in Patent Literature 1, since the depth of the subject is made ambiguous, unnecessary information is added even to pixels that are well estimated, and the quality of the generated image deteriorates. There is. Further, when the number of layers covering the subject increases, there is a problem that the calculation time increases at the same time that a large amount of computer resources such as memory are consumed.

  The present invention has been made in view of such circumstances, and can appropriately generate an image while saving computer resources when generating an omnidirectional image from images captured so as to overlap each other. An object is to provide an omnidirectional image generation method, an image generation apparatus, and a program.

  The present invention is an omnidirectional image generation method for generating an omnidirectional image of a subject from a plurality of images, an image acquisition step of acquiring an image of the subject from a plurality of viewpoint positions, and the acquired respective images. An image projecting step for projecting onto a plurality of predefined layer surfaces, a color luminance information calculating step for calculating color information or luminance information of the subject on the layer surface on which the image is projected, and an image acquiring step A reliability calculation for calculating a reliability representing the existence probability of the subject at each point on the layer surface by obtaining a similarity between the point of the subject on the acquired image and a point of the image projected on the layer surface And a layer selection step of selecting a plurality of layers around the layer having the highest reliability of the subject based on the reliability, and Color information or brightness information of the subject in-option by Layer plane and on the basis of the reliability information, and having an image generating step of generating an omnidirectional image of the object.

  The present invention is an omnidirectional image generation method for generating an omnidirectional image of a subject from a plurality of images, an image acquisition step of acquiring an image of the subject from a plurality of viewpoint positions, and the acquired respective images. An image projecting step for projecting onto a plurality of predefined layer surfaces, a color luminance information calculating step for calculating color information or luminance information of the subject on the layer surface on which the image is projected, and an image acquiring step Similarity between a region / representative point detection step of extracting a representative point in a region of a subject on the acquired image and a point of an image obtained by projecting the point of the subject on the image acquired by the image acquisition step on the layer surface A reliability calculation step for calculating a reliability indicating the existence probability of the subject at each point on the layer surface by obtaining a degree; and the reliability of the representative point Based on the layer selection step for selecting the layer range in which the reliability of the subject is high in the region, and based on the subject color information or luminance information and reliability information on the selected layer surface And an image generation step of generating an omnidirectional image.

  The present invention is an omnidirectional image generation method for generating an omnidirectional image of a subject from a plurality of images, an image acquisition step of acquiring an image of the subject from a plurality of viewpoint positions, and the acquired respective images. An image projecting step for projecting onto a plurality of predefined layer surfaces, a color luminance information calculating step for calculating color information or luminance information of the subject on the layer surface on which the image is projected, and an image acquiring step A reliability calculation for calculating a reliability representing the existence probability of the subject at each point on the layer surface by obtaining a similarity between the point of the subject on the acquired image and a point of the image projected on the layer surface A layer selection step for selecting the layer whose reliability of the subject is greater than or equal to a predetermined value based on the reliability, and the selected layer An image generation step of generating an omnidirectional image of the subject based on the color information or luminance information of the subject and reliability information, wherein the reliability of the subject satisfies a predetermined value. If not, the reliability in the reliability calculation step is set as the reliability.

  The present invention is an omnidirectional image generation method for generating an omnidirectional image of a subject from a plurality of images, an image acquisition step of acquiring an image of the subject from a plurality of viewpoint positions, and the acquired respective images. An image projecting step for projecting onto a plurality of predefined layer surfaces, a color luminance information calculating step for calculating color information or luminance information of the subject on the layer surface on which the image is projected, and an image acquiring step Similarity between a region / representative point detection step of extracting a representative point in a region of a subject on the acquired image and a point of an image obtained by projecting the point of the subject on the image acquired by the image acquisition step on the layer surface A reliability calculation step for calculating a reliability indicating the existence probability of the subject at each point on the layer surface by obtaining a degree; and the reliability of the representative point And a layer selection step of selecting a plurality of layers based on the reliability only when the layer having the highest reliability of the subject changes, and color information of the subject on the selected layer surface or And an image generation step of generating an omnidirectional image of the subject based on the luminance information and the reliability information.

  The present invention is an omnidirectional image generation method for generating an omnidirectional image of a subject from a plurality of images, an image acquisition step of acquiring an image of the subject from a plurality of viewpoint positions, and the acquired respective images. An image projecting step for projecting onto a plurality of predefined layer surfaces, a color luminance information calculating step for calculating color information or luminance information of the subject on the layer surface on which the image is projected, and a position of the subject In the changing area, the object existence probability at each point on the layer surface is obtained by calculating the similarity between the point of the object on the image acquired by the image acquisition step and the point of the image projected on the layer surface. A reliability calculation step for calculating a reliability representing the level, a layer selection step for selecting a plurality of layers based on the reliability, and the selected level An image generation step of generating an omnidirectional image of the subject based on color information or luminance information of the subject on the surface and reliability information, and the reliability calculation step does not change the position of the subject The region is characterized in that the reliability of each frame of the acquired image is the reliability calculated in the initial frame.

  The present invention is an image generation apparatus that generates an omnidirectional image of a subject from a plurality of images, and includes an image acquisition unit that acquires an image of the subject from a plurality of viewpoint positions, and the acquired images are defined in advance. Acquired by the image acquisition means for projecting onto the plurality of layer surfaces, color luminance information calculation means for calculating color information or luminance information of the subject on the layer surface on which the image is projected, and acquired by the image acquisition means Reliability calculating means for calculating a reliability indicating the existence probability of the subject at each point on the layer surface by obtaining a similarity to the image point obtained by projecting the point of the subject on the image onto the layer surface; , A layer selection unit that selects a plurality of layers around the layer having the highest probability of existence of the subject based on the reliability, and on the selected layer surface Based on the serial color information or brightness information and the reliability information of the object, characterized in that it comprises an image producing means for producing an omnidirectional image of the object.

  The present invention is an omnidirectional image generation program that causes a computer on an image generation apparatus that generates an omnidirectional image of a subject from a plurality of images to generate an omnidirectional image, and that captures an image of a subject from a plurality of viewpoint positions. An image acquisition step of acquiring, an image projecting step of projecting each of the acquired images onto a plurality of predefined layer surfaces, and color information or luminance information of the subject on the layer surface on which the image is projected By calculating the color luminance information calculation step to be calculated, and projecting the point of the subject on the image acquired by the image acquisition step on the layer surface, and calculating the similarity with the point of the image, each point on the layer surface A reliability calculating step for calculating a reliability indicating the existence probability of the subject, and the layer having the highest existence probability of the subject based on the reliability A layer selection step of selecting a plurality of layers as the center, and an image generation step of generating an omnidirectional image of the subject based on color information or luminance information and reliability information of the subject on the selected layer surface The computer is made to perform.

  According to the present invention, when generating an omnidirectional image captured from an arbitrary viewpoint position from images captured so as to overlap each other, it is possible to appropriately generate an image while saving computer resources. The effect is obtained.

It is a block diagram which shows the structure of one Embodiment of this invention. It is a flowchart which shows operation | movement of the apparatus shown in FIG. It is explanatory drawing which shows the arrangement | positioning relationship of the camera 1 shown in FIG. It is explanatory drawing which shows the mode of the correction | amendment of the reliability in 1st Embodiment. It is a flowchart which shows the process sequence in 2nd Embodiment. It is explanatory drawing which shows the mode of the correction | amendment of the reliability in 2nd Embodiment. It is explanatory drawing which shows the mode of the correction | amendment of the reliability in 3rd Embodiment. It is a flowchart which shows the process sequence in 4th Embodiment. It is explanatory drawing which shows the mode of the area division | segmentation in 5th Embodiment.

<First Embodiment>
Hereinafter, an image generation apparatus according to a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. In this figure, reference numeral 1 denotes a camera that captures an image of a subject A, and outputs the captured image as digital image data. The camera 1 is a camera that outputs an analog signal, and may convert the analog signal into digital data and output the digital data. Reference numeral 2 denotes an image processing unit that is configured by a computer device, inputs a plurality of image data output from the camera 1, generates an omnidirectional image of the subject A, and outputs the omnidirectional image. Reference numeral 3 denotes an input unit where an operator performs an input operation. Reference numeral 4 denotes a display unit that displays an image.

  Reference numeral 5 denotes a camera moving unit that moves the viewpoint of the camera 1 in order to capture an omnidirectional image of the subject A with the camera 1, and is configured by an actuator or the like. Reference numeral 6 denotes a camera position control unit that controls the amount of movement of the camera moving unit 5 based on an instruction from the image processing unit 2. In FIG. 1, the configuration in which the viewpoint of the camera 1 is moved by the camera moving unit 5 is shown. However, when the subject A can be moved, the orientation of the subject A is controlled to control the orientation of the subject A from all directions. The structure which images an image may be sufficient. Alternatively, a configuration may be employed in which a large number of cameras are arranged around the subject in advance and image data of images captured by the respective cameras is input to the image processing unit.

Next, the positional relationship of the camera 1 will be described with reference to FIG. In FIG. 3, a set of cameras 1 is C _i (iεN), and an original image I _i (m) captured by the camera 1 is assumed. Here, N represents a finite set of natural numbers. M represents the position of the pixel of the image in the image coordinate system with the upper left of the image i as the origin.

Next, the processing operation of the image processing unit 2 shown in FIG. 1 will be described with reference to FIG. First, the image processing unit 2 gives an instruction to the camera position control unit 6 to move the camera 1 and obtain an image I _i (m) (i∈N) obtained by imaging the subject A from a plurality of viewpoint positions. (Step S1). At this time, image data obtained by imaging so that at least two or more images overlap in all regions is acquired.

Next, the image processing unit 2 sets the virtual camera C _v to the viewpoint position as a reference of the omnidirectional image, sets a multi-layer L _{j (m) (j∈M)} from the virtual camera C _v in the depth direction . Here, M represents a finite set of natural numbers. At this time, an appropriate layer shape such as a plane, a cylindrical surface, or a spherical surface is selected according to the method of expressing the subject A. Next, the image I _i (m) is projected on each layer (step S2). When the shape of the layer is a plane, the transformation from the image I _i to the layer L _j is a projective transformation, and is generally associated by a matrix called homography. That is, conversion from m point on the image _{I i} of the camera _{C i,} to a point _{m v} on the image ^{I proj} _ij via the projection plane _{L j} is (1) is represented by equation (2).

Here, in a case where the pixel position of the image is expressed as m = [u, v] ^t in the image coordinate system with the upper left of the image i as the origin, m˜ [u, v, 1] ^t (˜ is m Represents the extension vector. R _{v ← i} and t _{v ← i} are rotation matrices and translation vectors for conversion from the camera coordinate system of the camera C _i to the virtual camera coordinate system, A _v is the internal matrix of the virtual camera, and A _i is the internal of the camera C _i Represents a matrix.

Note that the internal matrix and external matrix of the camera are obtained by a known calibration method. As a calibration method, a “Tsai method” or a “Zhang method” can be used. d _j is a distance from the origin of the virtual camera coordinate system to the layer L _j , and n is a normal vector of the plane layer. When the shape of the layer is a plane, the homography matrix H ^{v ← i} _j can be obtained from the feature points of the image I _i and the corresponding relationship.

Next, the image processing unit 2 calculates a color map representing the color information or luminance information of the subject at each point on the layer (step S3). The color map I ^color _j is generated by the equation (3) by blending the image projected on each layer in step S2.

As a representative example of the weighting factor w _ij (m), it can be set as shown in equation (4) on the assumption that the contribution of each image is equal.

Here, Num () is a function that takes a set as an argument and returns the number of elements. Further, the weighting factor w _ij (m) can be set to be weighted according to the positional relationship of the images. For example, when the positional relationship is represented by an angle, the weighting coefficient is calculated by equation (5).

Here, φ _ij (m) is a straight line connecting the center of the camera C _i that captured the image I _i (m) and the point m on the layer, and a straight line connecting the center of the omnidirectional image and the point m on the layer. Represents a corner. Ε is a minute value for preventing the denominator from becoming zero.

Next, the image processing unit 2 calculates a reliability map representing the existence probability of the subject at each point on the layer (step S4). The reliability map I ^prob _j represents the probability that a subject exists on each layer, and the larger this value, the higher the possibility that the subject exists at that position. Specifically, it is obtained by the equation (6) by the similarity S _j to the point of the image I _i corresponding to the point on the reliability map I ^prob _j , that is, the point on the projection image I ^proj _ij .

The similarity S _j (m) represents the similarity with the point m on the projection image corresponding to the point on the layer, and is calculated by the pixel value of the corresponding point and the neighboring pixel value. For example, using the correlation value of the pixel value, it is obtained by Equations (7) and (8).

  Here, B represents a neighborhood region of the pixel m, and is a value determined according to the characteristics of the image. ε is a minute value for preventing the denominator from becoming 0, and n is a parameter for adjusting the peak of the correlation value.

Next, the image processing unit 2 selects a layer having a high possibility that the subject exists from the information of the reliability map I ^prob _j calculated in step S4. The number of layers that may have a point m on the layer is less than the number of layers necessary to represent the entire scene, and estimation errors in other layers lower the synthesis quality. Here, as shown in FIG. 4, for each point m on the layer, information in the vicinity of the layer where the subject A is likely to exist is employed. The layers are scanned from the front to the back (or vice versa, from the back to the front), and values of a plurality of layers in the vicinity of the layer having the maximum reliability I ^prob _j (m) are adopted. Is set to 0 (expression (9), expression (10), expression (11)).

Here, α represents the number of layers to be selected, and takes an integer value of 0 or more. When α = 0, only one layer is selected, which is synonymous with depth map. When α ≠ 0, a plurality of layers are selected around the layer having the maximum reliability. After this process is performed on all points on the layer, the information of the reliability map I ^prob _j is renormalized by the equation (12) (step S5).

Next, the image processing unit 2 uses the reliability map I ^{prob ′} _j (m) representing the possibility that the subject obtained in step S5 exists in each layer, and the color information or luminance information of the subject obtained in step S3. The color map I ^color _j (m) to be represented is blended, and the information of all layers is added to generate an omnidirectional image I _v (m) according to the equation (13) (step S6).

<Second Embodiment>
Next, an image generation apparatus according to a second embodiment of the present invention will be described. In step S5 described above, in the first embodiment, a layer that employs reliability is determined for each point on the layer. However, since processing is performed for each pixel, the computer load is high. Therefore, after detecting the region of the subject from the original image, the reliability of the layer in the range where the representative points of the region exist is adopted. When there are a plurality of subjects, the region is divided for each subject, and the optimum layer range is determined in each region.

The processing operation in this embodiment is shown in FIG. In FIG. 5, the same parts as those in the processing operation shown in FIG. First, a region f (fεF) of the subject A that exists in common between the images is detected from the original image (step S7). Here, F is a set of regions. As a method of detecting the subject A, for example, a method using a background difference, a method using a difference in color from a known background color, a method of assuming a shape and a color of the subject, and searching for a matching area can be applied. is there. Subsequently, a plurality of representative points that exist in common between the images are extracted from the area of the subject A (step S8). As a representative point determination method, for example, a method using a point where a plurality of edges intersect, a method using a point where the hue changes abruptly, a method using a point where the texture changes can be applied. And the point which projected the typical point of the original image from the front layer to the back in order from the foremost layer (or the last layer) using the formulas (1) and (2) (or vice versa) For m ^f _k (kεP), the layer l _k having the maximum reliability I ^prob _j (m ^f _k ) is searched (step S5a; equation (14)). Here, P is a set of indexes for identifying representative points.

As shown in FIG. 6, the subject A is likely to exist between min _{k ∈ p lk} <j <max _{k ∈ p lk, and} therefore, at all points in the region f where the subject A exists, The values of the layers in this range are adopted, and the values of the other layers are set to 0 (Equations (15) and (16)).

After this processing is performed on all the regions extracted from the original image, the information of the reliability map I ^prob _j is renormalized by the equation (17) (step S5b). Note that the reliability value calculated in step S4 is adopted for regions other than the subject not extracted from the original image.

<Third Embodiment>
Next, an image generating apparatus according to a third embodiment of the present invention will be described. In the first embodiment and the second embodiment, the reliability value of a plurality of layers is adopted around the layer having the maximum reliability. However, when there are a plurality of reliability peaks, When there is no peak, it is difficult to determine the maximum value. Therefore, in the third embodiment, as shown in FIG. 7, when a point m on the layer is scanned from the front to the back (or vice versa), there is a reliability of one layer or more above the threshold Th. Adopts information of a layer having a threshold value Th (equations (18) and (19)). When the point m on the layer is scanned from the front to the back (or vice versa), if there is no layer that satisfies the reliability equal to or greater than the threshold value Th, the reliability value of step S4 is set.

After this processing is applied to the entire area, the information of the reliability map I ^prob _j is renormalized by the equation (20).

<Fourth Embodiment>
Next, an image generating apparatus according to a fourth embodiment of the present invention will be described. In the fourth embodiment, the layer that adopts the reliability is determined for each frame of the video because the computer load is large, so that the process of step S5 is performed only when the position of the subject greatly fluctuates.

The processing operation in this embodiment is shown in FIG. In FIG. 8, the same parts as those in the processing operation shown in FIG. First, a subject area is detected from the original image (step S7). As a subject detection method, for example, a method using a background difference, a method using a difference in color from a known background color, a method of searching for a matching region by assuming an object shape or color, and the like can be applied. Subsequently, a plurality of representative points are extracted from the subject area (step S8). As a representative point determination method, for example, a method using a point where a plurality of edges intersect, a method using a point where the hue changes abruptly, a method using a point where the texture changes can be applied. And the point which projected the typical point of the original image from the front layer to the back in order from the foremost layer (or the last layer) using the formulas (1) and (2) (or vice versa) For m ^f _k (kεP), the layer position where the reliability I ^prob _j (m _k ) is maximum is stored. When the difference between the layer position of the previous frame and the layer position where the reliability is maximum is equal to or greater than β, the processes of steps S5c and S5d are performed. β is a parameter representing the change width of the layer position between frames, and is a value determined according to the amount of movement of the subject. In step S5, an appropriate process may be selected from the first, second, and third embodiments.
Since subject A moves in directions other than the depth direction, subject region detection and representative point extraction are repeatedly performed. When the area of the subject and the representative point have changed from the previous frame, those values are stored, and the area for correcting the reliability is changed.

<Fifth Embodiment>
Next, an image generating apparatus according to a fifth embodiment of the present invention will be described. In the fifth embodiment, the layer that adopts the reliability is determined for each frame of the video because the computer load is large. Therefore, as shown in FIG. In step S4, the reliability is calculated for each frame of the former area and only for the first frame of the latter area. In step S5, an appropriate process may be selected from the first, second, and third embodiments.

  As described above, it is generated by correcting information with low reliability, that is, with low depth estimation accuracy, among the reliability information indicating the probability that the subject A exists at each point in the three-dimensional space. Image quality degradation can be solved. Also, by calculating only the necessary layers for each subject and scene, computer resources such as memory and processing time can be reduced.

  Note that a program for realizing the function of the image processing unit 2 in FIG. 1 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed, so that the omnidirectional Image generation processing may be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer system” includes a WWW system having a homepage providing environment (or display environment). The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  The present invention can also be applied to applications where it is indispensable to generate an omnidirectional image centered on an arbitrary viewpoint from a plurality of images picked up by changing directions so as to overlap each other.

  A ... Subject, 1 ... Camera, 2 ... Image processing unit, 3 ... Input unit, 4 ... Display unit, 5 ... Camera moving unit, 6 ... Camera position control unit , 7 ... Image acquisition unit

Claims (7)

An omnidirectional image generation method for generating an omnidirectional image of a subject from a plurality of images,
An image acquisition step of acquiring an image of a subject from a plurality of viewpoint positions;
An image projecting step of projecting each acquired image onto a plurality of predefined layer surfaces;
A color luminance information calculating step for calculating color information or luminance information of the subject on the layer surface on which the image is projected;
By calculating the similarity between the point of the subject on the image acquired by the image acquisition step and the point of the image projected on the layer surface, the reliability representing the existence probability of the subject at each point on the layer surface is obtained. A reliability calculation step to calculate,
A layer selection step of selecting a plurality of layers around the layer having the highest reliability of the subject based on the reliability;
An omnidirectional image generation method comprising: an image generation step of generating an omnidirectional image of the subject based on color information or luminance information of the subject on the selected layer surface and reliability information. An omnidirectional image generation method for generating an omnidirectional image of a subject from a plurality of images,
An image acquisition step of acquiring an image of a subject from a plurality of viewpoint positions;
An image projecting step of projecting each acquired image onto a plurality of predefined layer surfaces;
A color luminance information calculating step for calculating color information or luminance information of the subject on the layer surface on which the image is projected;
A region / representative point detecting step for extracting a representative point in the region of the subject on the image obtained by the image obtaining step;
By calculating the similarity between the point of the subject on the image acquired by the image acquisition step and the point of the image projected on the layer surface, the reliability representing the existence probability of the subject at each point on the layer surface is obtained. A reliability calculation step for calculating, and a layer selection step for selecting the layer range in which the reliability of the subject is high in the region based on the reliability,
An omnidirectional image generation method comprising: an image generation step of generating an omnidirectional image of the subject based on color information or luminance information of the subject on the selected layer surface and reliability information. An omnidirectional image generation method for generating an omnidirectional image of a subject from a plurality of images,
An image acquisition step of acquiring an image of a subject from a plurality of viewpoint positions;
An image projecting step of projecting each acquired image onto a plurality of predefined layer surfaces;
A color luminance information calculating step for calculating color information or luminance information of the subject on the layer surface on which the image is projected;
By calculating the similarity between the point of the subject on the image acquired by the image acquisition step and the point of the image projected on the layer surface, the reliability representing the existence probability of the subject at each point on the layer surface is obtained. A reliability calculation step to calculate,
A layer selection step of selecting the layer based on the reliability, the reliability of the subject being a predetermined value or more;
An image generation step of generating an omnidirectional image of the subject based on color information or luminance information and reliability information of the subject on the selected layer surface;
The omnidirectional image generation method, wherein the layer selection step uses the reliability of the reliability calculation step when the reliability of the subject does not satisfy a predetermined value. An omnidirectional image generation method for generating an omnidirectional image of a subject from a plurality of images,
An image acquisition step of acquiring an image of a subject from a plurality of viewpoint positions;
An image projecting step of projecting each acquired image onto a plurality of predefined layer surfaces;
A color luminance information calculating step for calculating color information or luminance information of the subject on the layer surface on which the image is projected;
A region / representative point detecting step for extracting a representative point in the region of the subject on the image obtained by the image obtaining step;
By calculating the similarity between the point of the subject on the image acquired by the image acquisition step and the point of the image projected on the layer surface, the reliability representing the existence probability of the subject at each point on the layer surface is obtained. A reliability calculation step to calculate,
A layer selection step of selecting a plurality of layers based on the reliability only when the layer having the highest reliability of the subject changes based on the reliability of the representative point;
An omnidirectional image generation method comprising: an image generation step of generating an omnidirectional image of the subject based on color information or luminance information of the subject on the selected layer surface and reliability information. An omnidirectional image generation method for generating an omnidirectional image of a subject from a plurality of images,
An image acquisition step of acquiring an image of a subject from a plurality of viewpoint positions;
An image projecting step of projecting each acquired image onto a plurality of predefined layer surfaces;
A color luminance information calculating step for calculating color information or luminance information of the subject on the layer surface on which the image is projected;
In the region where the position of the subject changes, each point on the layer surface is obtained by obtaining a similarity to the point of the image on the layer surface obtained by projecting the point of the subject on the image acquired by the image acquisition step. A reliability calculating step for calculating a reliability representing the reliability of the subject in
A layer selection step of selecting a plurality of layers based on the reliability based on the reliability;
An image generation step of generating an omnidirectional image of the subject based on color information or luminance information and reliability information of the subject on the selected layer surface;
The omnidirectional image generation method characterized in that the reliability calculation step sets the reliability of each frame of the acquired image to the reliability calculated in the initial frame in an area where the position of the subject does not change. An image generation device that generates an omnidirectional image of a subject from a plurality of images,
Image acquisition means for acquiring an image of a subject from a plurality of viewpoint positions;
Image projecting means for projecting each acquired image onto a plurality of predefined layer surfaces;
Color luminance information calculating means for calculating color information or luminance information of the subject on the layer surface on which the image is projected;
By calculating the similarity between the point of the subject on the image acquired by the image acquisition means and the point of the image projected on the layer surface, the reliability representing the existence probability of the subject at each point on the layer surface is obtained. A reliability calculation means for calculating;
Based on the reliability, a layer selection means for selecting a plurality of layers around the layer having the highest reliability of the subject;
An image generation apparatus comprising: an image generation unit configured to generate an omnidirectional image of the subject based on color information or luminance information of the subject on the selected layer surface and reliability information. An omnidirectional image generation program for causing a computer on an image generation device that generates an omnidirectional image of a subject from a plurality of images to generate an omnidirectional image,
An image acquisition step of acquiring an image of a subject from a plurality of viewpoint positions;
An image projecting step of projecting each acquired image onto a plurality of predefined layer surfaces;
A color luminance information calculating step for calculating color information or luminance information of the subject on the layer surface on which the image is projected;
By calculating the similarity between the point of the subject on the image acquired by the image acquisition step and the point of the image projected on the layer surface, the reliability representing the existence probability of the subject at each point on the layer surface is obtained. A reliability calculation step to calculate,
A layer selection step of selecting a plurality of layers around the layer having the highest probability of existence of the subject based on the reliability;
An omnidirectional image that causes the computer to perform an image generation step of generating an omnidirectional image of the subject based on color information or luminance information and reliability information of the subject on the selected layer surface. Generation program.

Images