JP5809607B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program that generate an image of an arbitrary viewpoint from a multi-viewpoint image.

  2. Description of the Related Art Conventionally, there has been known an arbitrary viewpoint image synthesis technique that fuses image data captured by a multi-viewpoint camera inside a computer and presents the appearance from an arbitrary viewpoint (see, for example, Non-Patent Document 1). For arbitrary viewpoint image synthesis, it is common to use Visual Hull, which is the product set space of cones formed by the silhouette of an object photographed from multiple viewpoints and the optical center of the camera. Arbitrary viewpoint image synthesis is a process of obtaining a 3D (three-dimensional) shape of an object using Visual Hull, and then performing a process for refining the shape, and in accordance with the line-of-sight information (viewpoint position and line-of-sight direction) of the virtual camera. This is done by calculating.

  Arbitrary viewpoint image synthesis is a series of calculation processes in which multi-viewpoint images from a multi-camera system are input, the 3D shape to be synthesized is obtained, and the texture of the shape surface is calculated. Calculation indices to include include 3D shape accuracy and spatial resolution.

  FIG. 4 is a flowchart showing the processing operation of arbitrary viewpoint image synthesis according to the prior art. With reference to FIG. 4, the processing operation of arbitrary viewpoint image synthesis according to the prior art will be described. First, a 3D shape of an object using Visual Hull is calculated from an image photographed from multiple viewpoints using a multi-camera system (step S11). The shape obtained in this step is inaccurate and is in a state approximating the shape of the object. Next, calculation is performed to approximate the Visual Hull to an accurate shape by a known method (step S12). As a result, target silhouettes corresponding to a plurality of viewpoints are obtained.

  Next, information (virtual camera angle-of-view information) such as the viewpoint position and line-of-sight direction of the virtual camera for combining the arbitrary viewpoint images is determined (step S13). Then, based on the determined field angle information of the virtual camera, the texture of the 3D shape surface is calculated and texture mapping is performed (step S14). Finally, the obtained virtual camera image is drawn (step S15). As a result, an arbitrary viewpoint image is synthesized and displayed.

Matsuyama T. and Takai T. and Wu X. and Nobuhara S .: "Gener-ation, Editing, and Visualization of 3D Video", "The Transactions of The Virtual Reality Society of Japan", No.4 Vol.7, pp . "521-532" (2002).

  However, in the arbitrary viewpoint image synthesis according to the prior art, the angle of view of the virtual camera is determined after the 3D shape refinement processing (Visual Hull refinement: step S12), and the 3D shape refinement is performed. Since the processing is performed regardless of the angle of view information of the virtual camera, there is a problem that the object to be refined is the entire 3D shape, the calculation load is high, and the calculation processing efficiency is poor.

  The present invention has been made in view of such circumstances, and can reduce calculation load and increase calculation processing efficiency when generating an image of an arbitrary viewpoint, and improve the image quality of the generated image. An object of the present invention is to provide an image processing apparatus, an image processing method, and an image processing program.

The present invention is an image processing apparatus for generating an image of a virtual camera arranged at an arbitrary viewpoint position from images of a plurality of input cameras taken from different positions of an object, and from the images of the plurality of input cameras. A three-dimensional shape calculating means for calculating and obtaining a three-dimensional shape of the object; an angle-of-view information determining means for determining angle-of-view information of the virtual camera; and the virtual camera based on the angle-of-view information. 3D shape accuracy with respect to the projection means for performing projection processing on the image, depth calculation means for calculating the depth value in the virtual camera coordinate system for each pixel of the virtual camera image, and the set of depth values and optimization processing means for performing optimization to maximize an indicator representing the photo-consistenc y is the optimized depth values of all pixels of the virtual camera Based, characterized in that it comprises an image producing means for producing an image of the virtual camera by obtaining by calculating the pixel values of the image of the virtual camera.

The present invention is an image processing method performed by an image processing apparatus that generates an image of a virtual camera arranged at an arbitrary viewpoint position from images of a plurality of input cameras obtained by imaging an object from different positions, A three-dimensional shape calculating step for calculating a three-dimensional shape of the object from an input camera image; an angle-of-view information determining step for determining angle-of-view information of the virtual camera; and the three-dimensional shape as the angle-of-view information. a projection step of projecting process to the virtual camera on the image based on, for each pixel of the virtual camera image, the depth calculation step of obtaining by calculating the depth values in the virtual camera coordinate system, the set of the depth value, 3 and optimization process steps photo-consistenc y performs optimization so as to maximize an indicator indicating the accuracy of the dimension shape, the optimization Based on the depth values of all pixels of the virtual camera, and having an image generating step of generating an image of the virtual camera by obtaining by calculating the pixel values of the image of the virtual camera.

  The present invention is an image processing program for causing a computer to function as the image processing apparatus.

  According to the present invention, since the 3D shape refinement is performed based on the angle-of-view information of the virtual camera in generating an arbitrary viewpoint image, it is possible to limit the target area for the refinement. As a result, the image quality of the generated image can be improved.

It is a block diagram which shows the structure of one Embodiment of this invention. It is a flowchart which shows operation | movement of the image process part 4 shown in FIG. FIG. 3 is a flowchart showing details of a processing operation for refining the Visual Hull in the visible region shown in FIG. 2 (step S <b> 16). It is a flowchart which shows the processing operation | movement of the arbitrary viewpoint image composition by a prior art.

  Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. In this figure, reference numerals 11 to 13 denote a plurality of imaging units constituting a multi-camera. The imaging units 11 to 13 are configured by cameras equipped with zoom lenses having different settings. Although three imaging units are illustrated in FIG. 1, four or more imaging units may be provided, and it is usually desirable to have about 12 units. Reference numeral 2 denotes an image storage unit that stores image data output from the plurality of imaging units 11 to 13. Reference numeral 3 denotes a viewpoint input unit that inputs a desired arbitrary viewpoint position. The viewpoint input unit 3 has a function of performing an input operation for the user to specify a desired viewpoint position.

  Reference numeral 4 denotes an image processing unit that synthesizes an image viewed from an arbitrary viewpoint position input by the viewpoint input unit 3. Reference numeral 5 denotes an image display unit that displays an image synthesized by the image processing unit 4, and is configured by a display device or the like. When the configuration of the image processing apparatus is described with reference to FIG. 1, known functions and configurations that are commonly possessed by an image processing apparatus that synthesizes an arbitrary viewpoint image from a multi-viewpoint image are directly described in the description of the present invention. As long as there is no relation, the description and illustration of the configuration are omitted. In the description of the present specification, one frame of a moving image (video) is referred to as an image.

  Next, the operation of the image processing unit 4 shown in FIG. 1 will be described with reference to FIG. FIG. 2 is a flowchart showing the operation of the image processing unit 4 shown in FIG. In FIG. 2, the same parts as those in the processing operation according to the prior art shown in FIG. The processing operation shown in this figure is different from the processing operation according to the prior art in that the Visual Hull refining process (step S12) is not performed, and the virtual camera field angle determination process (step S13) is performed only for the visible region. This is the point that the Hull refinement process (step S16) is performed.

  When the image processing unit 4 determines the viewing angle information (viewing point position, viewing direction, and viewing angle) of the virtual camera based on the viewpoint position information from the viewpoint input unit 3 (step S13), the image processing unit 4 responds to the viewing angle information of the virtual camera. Thus, the region to be refined is limited, and refinement of the 3D shape is performed so as to improve the evaluation index called photo-consistency (step S16). Then, based on the determined field angle information of the virtual camera, the texture of the 3D shape surface is calculated and texture mapping is performed (step S14).

  Next, the details of the Visual Hull refinement process (step S16) shown in FIG. 2 will be described with reference to FIG. FIG. 3 is a flowchart showing details of the processing operation of visual hull refinement (step S16) in the visible region shown in FIG. To simplify the explanation, the following notation is defined. Assume that the 3D shape is S, the virtual camera is {circumflex over (c)} (where ^ is attached to c, and so on), and the input camera is ciεC (i = 1,..., N).

  First, when the virtual camera angle of view is determined, the image processing unit 4 projects the 3D shape S onto the screen (image to be synthesized) of the virtual camera ^ c (step S161). Subsequently, the image processing unit 4 selects one pixel p on the virtual camera ^ c screen (step S162), and for the selected pixel p, the depth (from the viewpoint to the object) of the camera coordinate system of the virtual camera ^ c. (Distance) dp is calculated (step S163). The image processing unit 4 repeats the processes of steps S162 and S163 for all the pixels of the virtual camera ^ c. As a result, a virtual distance image based on the virtual camera ^ c is obtained.

  Next, the image processing unit 4 optimizes the photo-consistency of the obtained virtual distance image (step S164). “Photo-consistency” is one of indices indicating the accuracy of the 3D shape. When one point in the 3D space is photographed by a plurality of cameras, pixel values on the respective cameras are compared, and the degree of coincidence between the cameras can be considered. Actually, an error function that represents a pixel difference between cameras is defined, and a 3D shape is considered as a set of points, and a combination of error functions at all points is defined as an inverse of photo-consistency. When the 3D shape S is accurately obtained, the error value becomes 0 at all points on the surface.

ただし、Ｃ_ｑは全入力カメラＣのうち、画素ｐに対応する３Ｄ形状Ｓ表面の点が撮影されるカメラの集合を表す。ＺＮＣＣ（π_δｐ，ｃ_ｉ，ｃ_ｊ）は、微小平面π_δｐをカメラｃ_ｉとｃ_ｊの画像上に投影した位置での画素値の間のＺＮＣＣ（zero-mean-cross-correlation）である。また、｜Ｃ_ｑ｜Ｃ_２はカメラ集合Ｃ_ｑから２台のカメラを選ぶ組み合わせの数である。 Here, for one pixel p on the virtual distance image, the true distance is defined as d _p + δ _p, and the error function E _p (δ _p ) is defined as in equation (1).

Here, C _q represents a set of cameras in which the points on the surface of the 3D shape S corresponding to the pixel p among all the input cameras C are captured. ZNCC (π _δp , c _i , c _j ) is a ZNCC (zero-mean-cross-correlation) between pixel values at a position where the minute plane π _δp is projected on the images of the cameras c _i and c _j. . | C _q | C ₂ is the number of combinations for selecting two cameras from the camera set C _q .

Here, in addition to E _p (δ _p ), error functions E _d (δ _p ) and E _c (δ _p , δ _{p ′} ) are combined.
E _d (δ _p ) = | δ _p | (2)
E _c (δ _p , δ _{p ′} ) = | δ _p −δ _{p ′} | (3)

ただし、Ｐは全画素の集合、Ｎは画素間の連結関係の集合であり、λ_ｐ，λ_ｄ，λ_ｃは各項の重み係数である。 Therefore, the optimization objective function is defined as follows.

Here, P is a set of all pixels, N is a set of connection relations between pixels, and λ _p , λ _d , and λ _c are weighting factors of each term.

  Next, the image processing unit 4 performs texture mapping (step S14). Subsequent processing operations are the same as those of the prior art (processing operation shown in FIG. 4).

  As described above, since the 3D shape is refined after the virtual camera field angle determination process, only the visible region can be refined based on the virtual camera field angle information, thus reducing the processing load. It is possible to improve the calculation processing efficiency. In addition, in the synthesis of an arbitrary viewpoint image, a multi-viewpoint image by a multi-camera system is input, a 3D shape to be synthesized is obtained, and the texture of the shape surface is calculated. 3D shape accuracy and spatial resolution are included in the calculation index that greatly affects. In the present embodiment, it is possible to improve the image quality of the composite image by improving the 3D-shaped photo-consistency based on the view angle information of the virtual camera.

  The program for realizing the functions of the processing unit in FIG. 1 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into the computer system and executed to execute arbitrary viewpoint image synthesis. Processing may be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer system” includes a WWW system having a homepage providing environment (or display environment). The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Accordingly, additions, omissions, substitutions, and other changes of the components may be made without departing from the technical idea and scope of the present invention.

  Arbitrary viewpoint image composition technology based on the target 3D shape increases the added value of image distribution and realizes an interactive viewing experience. In addition, it can be expected to contribute greatly to the creation of killer apps for image communication by enabling remote teleconferencing and teleworking with a high sense of reality.

  DESCRIPTION OF SYMBOLS 11, 12, 13 ... Imaging part, 2 ... Image storage part, 3 ... Viewpoint input part, 4 ... Image processing part, 5 ... Image display part

Claims (3)

An image processing device that generates an image of a virtual camera arranged at an arbitrary viewpoint position from images of a plurality of input cameras obtained by capturing an object from different positions,
Three-dimensional shape calculation means for calculating and calculating a three-dimensional shape of the object from the images of the plurality of input cameras;
Angle-of-view information determining means for determining angle-of-view information of the virtual camera;
Projection means for projecting the three-dimensional shape onto the image of the virtual camera based on the angle-of-view information;
Depth calculation means for calculating a depth value in a virtual camera coordinate system for each pixel of the virtual camera image;
An optimization processing means for optimizing the set of depth values so that a photo-consistency that is an index representing the accuracy of a three-dimensional shape is maximized;
Image generation means for generating an image of the virtual camera by calculating and obtaining a pixel value of the image of the virtual camera based on the optimized depth values of all the pixels of the virtual camera. An image processing apparatus. An image processing method performed by an image processing apparatus that generates an image of a virtual camera arranged at an arbitrary viewpoint position from images of a plurality of input cameras obtained by imaging an object from different positions,
A three-dimensional shape calculation step for calculating a three-dimensional shape of the object from images of the plurality of input cameras;
An angle-of-view information determining step for determining angle-of-view information of the virtual camera;
A projecting step of projecting the three-dimensional shape onto the image of the virtual camera based on the angle-of-view information;
Depth calculation step for calculating a depth value in a virtual camera coordinate system for each pixel of the virtual camera image;
An optimization processing step for optimizing the set of depth values so that a photo-consistency that is an index representing the accuracy of a three-dimensional shape is maximized;
An image generation step of generating an image of the virtual camera by calculating and obtaining a pixel value of the image of the virtual camera based on the optimized depth value of all the pixels of the virtual camera. Image processing method.   An image processing program for causing a computer to function as the image processing apparatus according to claim 1.

Images