JP2011210202A - Apparatus for generating free viewpoint image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a compact and inexpensive apparatus for generating free-viewpoint image, which is easily installed and images all subjects including a stationary one.SOLUTION: An image processing computer 14 generates a virtual-view-point image GK on the basis of a pair of direct subject images GL and GR directly captured by a left camera CL and a right camera CR, respectively, two pairs of indirect subject images GLML/GRML and GLMR/GRMR, and a depth value Z determined by the direct and indirect subject images. The direct and indirect subject images are simultaneously formed by a single shooting operation of the left and right cameras CL and CR. Thus, every subject 12 including the stationary one can be imaged. Only by installing the pair of left camera CL and the right camera CR between the computer and the left and right mirrors ML and MR, respectively, thereby providing the easy-to-install, compact and inexpensive apparatus 10 for generating free-viewpoint image.

Description

  The present invention relates to a free viewpoint image generation apparatus for converting and outputting an image of a subject viewed from a desired virtual viewpoint from the imaging data of the subject.

  As an apparatus that can generate an image of a subject at an arbitrary virtual viewpoint, as shown in paragraphs 0007, 0048, 0049, FIG. 14, and patent document 2 of Patent Document 1, a large number of cameras are arranged so as to surround the subject. An apparatus for synthesizing video in a virtual camera assumed at a predetermined position from videos from these cameras has been proposed. However, in such an apparatus, since a large number of cameras are arranged, the apparatus becomes large-scale, and positioning of each camera and color correction and synchronization between cameras are necessary, so that signal processing becomes complicated. As a result, the heat generated by the power consumption is large and the apparatus is expensive.

JP 2006-352539 A WO 2007/026440 A1

  On the other hand, multiple images of the subject were acquired in the process of moving the camera along an arc surrounding the subject while keeping the camera facing the subject, and assumed to be at a predetermined position from the acquired multiple images An apparatus for synthesizing video in a virtual camera has been proposed. According to this, since the number of cameras is reduced, heat generation due to power consumption is reduced, color correction and synchronization between cameras is unnecessary, signal processing is simplified, and the apparatus is inexpensive. A device that accurately moves while maintaining the distance to the subject in a posture directed toward the camera, and the device is still large and complicated to install, and the subject is limited to immovable objects was there.

  The present invention has been made against the background of the above circumstances, and an object of the present invention is to provide a free viewpoint image generation apparatus that is not limited to a stationary object and that is easy to install and that is small and inexpensive. There is.

  As a result of various studies on the background of the above circumstances, the present inventors have made a virtual viewpoint of a subject using a virtual camera at the mirror image position of the actual camera formed using a mirror in addition to the actual camera. It has been found that a virtual viewpoint image of a subject viewed from a viewpoint arbitrarily assumed at a predetermined position can be obtained by acquiring each image and using each acquired image. The present invention has been made based on such knowledge.

  That is, the gist of the invention according to claim 1 for achieving the above object is to generate a free viewpoint image for converting the imaging data of the subject into an image viewed from a desired virtual viewpoint and outputting the image. An apparatus, at least one camera that images the subject, at least one mirror that is located within the imaging field of view of the camera and reflects light from the subject toward the camera, and is directly imaged by the camera Based on the at least one direct subject image, at least one indirect subject image indirectly captured by the camera via the mirror, and a depth value obtained from the direct subject image and the indirect subject image And an image processing calculation control device that generates the image of the virtual viewpoint.

  According to a second aspect of the present invention, in the first aspect of the invention, the image processing calculation control device represents the subject directly captured by the camera from the captured image of the camera. A subject image separation unit that separates a direct subject image from the indirect subject image representing the subject indirectly captured by the camera via the mirror, and a virtual camera based on the position of the camera and the position of the mirror A pair of cameras having a high correlation among a plurality of subject images including the at least one direct subject image and at least one indirect subject image separated by the subject image separation unit. Depth value calculation for calculating each of the depth values based on the relative positions of locally related portions between the subject images formed A virtual viewpoint image for generating a virtual viewpoint image of the subject viewed from the virtual viewpoint based on the pair of subject images for which the depth value is calculated by the depth value calculation unit, and the depth value A generation unit.

  The gist of the invention according to claim 3 is that, in the invention of claim 2, the subject image separation unit includes an image including the subject imaged by the camera and the subject imaged by the camera. By calculating a difference image from the background image not included, a subject image of only the subject is generated, and a relatively large one of the subject images is extracted as the direct subject image, and a relatively small one Is extracted as the indirect subject image.

  The gist of the invention according to claim 4 is that, in the invention of claim 2 or 3, the camera position calculation unit is configured to determine the mirror of the camera based on the position of the camera and the position of the mirror. The position of the virtual camera in the mirror image position with respect to is calculated.

  The gist of the invention according to claim 5 is that, in the invention of claim 4, the camera position calculation unit calculates a three-dimensional vector from the position of the camera to the center of the mirror, and A symmetric vector that is symmetric with respect to the mirror surface of the vector and the mirror is calculated, and the tip position of the symmetric vector is calculated as the position of the virtual camera.

  According to a sixth aspect of the present invention, there is provided the gist of the invention according to any one of the second to fifth aspects, wherein the depth value calculation unit is the at least one direct subject separated by the subject image separation unit. Among a plurality of subject images including an image and at least one indirect subject image, one or a plurality of pixels constituting the local area of one subject image and the other subject image between subject images that form a highly correlated pair Calculating a parallax that is a relative position between a pair of one pixel or a plurality of pixels having the highest correlation with one pixel or a plurality of pixels constituting the local area, and the focal point of the camera The depth value between the paired subject images is calculated based on the distance and the inter-camera distance.

  A gist of the invention according to claim 7 is that, in the invention according to any one of claims 2 to 6, the virtual viewpoint image generation unit generates the direct subject image, the indirect subject image, and the depth value. Based on the three-dimensional position of the pixel, the three-dimensional position of the pixel representing the subject in three dimensions or the three-dimensional position of the pixel representing the subject in a three-dimensional figure using a plurality of polygons is determined. The object is to generate a virtual viewpoint image of the subject viewed from a predetermined virtual viewpoint.

  According to the free viewpoint image generation device of the invention according to claim 1, the image processing calculation control device passes at least one direct subject image directly captured by at least one camera and at least one mirror by the camera. A virtual viewpoint image is generated based on at least one indirect subject image captured indirectly and a depth value obtained from the direct subject image and the indirect subject image. For this reason, since the direct subject image and the indirect subject image can be instantaneously obtained by at least one imaging with at least one camera, the subject is not limited to an immovable object, and at least one camera is located in front of the mirror. Therefore, a free viewpoint image generating apparatus that is easy to install, small and inexpensive can be obtained.

  According to the free viewpoint image generation device of the invention according to claim 2, the image processing calculation control device includes the direct subject image representing the subject directly captured by the camera from the captured image of the camera, and the camera. A subject image separation unit that separates the indirect subject image representing the subject indirectly captured through the mirror, and a camera that calculates the position of the camera and the position of the virtual camera based on the position of the mirror A local portion between subject images forming a highly correlated pair from among a plurality of subject images including the position calculating unit and the at least one direct subject image and at least one indirect subject image separated by the subject image separating unit A depth value calculation unit for calculating the depth value based on relative positions of mutually related portions, and the depth value calculation And the object image forming a pair mutually 該奥 go values calculated by the section, and a virtual viewpoint image generation unit for generating a virtual viewpoint image of the object viewed from the virtual viewpoint based on the 該奥 go values, including. Accordingly, the image processing calculation control device forms a highly correlated pair from among a plurality of subject images including the at least one direct subject image and at least one indirect subject image separated by the subject image separation unit. The virtual viewpoint image of the subject viewed from the virtual viewpoint based on the subject image and the depth value calculated by the depth value calculation unit based on the relative position of the locally interrelated portions between them. Since the direct subject image and the indirect subject image can be obtained instantly, the subject is not limited to an immovable object, and it is only necessary to install at least one camera in front of the mirror, making installation easy and compact. In addition, an inexpensive free viewpoint image generation apparatus can be obtained.

  According to the free viewpoint image generation device of the invention according to claim 3, the subject image separation unit is configured to obtain an image including the subject captured by the camera and a background image not including the subject captured by the camera. By calculating a difference image, a subject image of only the subject is generated, a relatively large one of the subject images is extracted as the direct subject image, and a relatively small one is used as the indirect subject image. Since it is what is extracted, the direct subject image and the indirect subject image are easily separated.

  According to the free viewpoint image generation device of the invention according to claim 4, the camera position calculation unit is based on the position of the camera and the position of the mirror, and the virtual camera is located at the mirror image position of the camera Therefore, the position of the virtual camera is easily calculated.

  According to the free viewpoint image generation device of the invention according to claim 5, the camera position calculation unit calculates a three-dimensional vector from the position of the camera to the center of the mirror, and the three-dimensional vector and the mirror surface of the mirror Therefore, the position of the virtual camera can be easily calculated.

  According to the free viewpoint image generation device of the invention according to claim 6, the depth value calculation unit includes a plurality of the at least one direct subject image and at least one indirect subject image separated by the subject image separation unit. Between subject images that form a highly correlated pair from among the subject images, one pixel or a plurality of pixels constituting the locality of one subject image and one or a plurality of pixels constituting the locality of the other subject image Calculating a parallax that is a relative position between a pair of one or a plurality of pixels having the highest correlation between the pair of pixels, and determining the pair based on the parallax, the focal length of the camera, and the inter-camera distance. Since the depth value between the formed subject images is calculated, the depth value can be easily obtained.

  According to the free viewpoint image generation device of the invention according to claim 7, the virtual viewpoint image generation unit includes pixels of the subject representing in three dimensions based on the direct subject image, the indirect subject image, and the depth value. A three-dimensional position or a three-dimensional position of a pixel representing the subject as a three-dimensional figure using a plurality of polygons is determined, and a virtual viewpoint of the subject viewed from a predetermined virtual viewpoint based on the three-dimensional position of the pixel Since an image is generated, a virtual viewpoint image of a subject viewed from an arbitrary virtual viewpoint can be easily obtained.

It is a conceptual diagram explaining the whole structure of the free viewpoint image generation apparatus of one Example of this invention. It is a functional block diagram explaining the principal part of the calculation control function of the image processing computer of FIG. It is a figure which illustrates the picked-up image imaged with the right camera of FIG. 1 and FIG. It is a figure which illustrates the background image previously imaged and memorize | stored by the right camera of FIG. 1 and FIG. FIG. 4 is a diagram illustrating a direct subject image and an indirect subject image separated from the captured image of FIG. 3 by the subject image separation unit of FIG. 2. The pair of direct subject images and the two pairs of indirect subject images separated from the captured images taken by the left and right cameras by the subject image separation unit in FIG. 2 are associated with the pair of left and right cameras and the two pairs of virtual cameras. It is a figure shown. It is a figure which shows the picked-up image of the right camera containing the frame of a right mirror among the processes which determine the position of the virtual camera formed in the mirror image position of a right camera with a right mirror. Of the process of determining the position of the virtual camera formed at the mirror image position of the right camera by the right mirror, based on the length and position of how the right mirror frame with a known shape is reflected in the captured image FIG. 5 is a diagram showing the position of the center point of the right mirror MR calculated using the geometric relationship. Of the process of determining the position of the virtual camera formed at the mirror image position of the right camera by the right mirror, the three-dimensional vector from the right camera position to the position of the center point of the right mirror and the vector symmetric with respect to the plane of the right mirror And a virtual camera located at the symmetric vector tip. Among a pair of direct subject images and two indirect subject images GRMR, GRML, GLML, and GLMR separated by the subject image separation unit in FIG. FIG. 4A is a diagram illustrating a direct subject image, where FIG. 5A illustrates a direct subject image captured by the left camera, and FIG. 5B illustrates a direct subject image captured by the right camera. It is a figure explaining the parallax between a pair of direct subject images similar to FIG. 10 used for the calculation of the depth value by the depth value calculation unit of FIG. It is a figure explaining the calculation method of the depth value based on the parallax, the focal distance of a camera, and the distance between a pair of cameras by the depth value calculation part of FIG. A depth value between a pair of similar direct subject images obtained by the depth value calculation unit shown in FIG. 2 and the direct subject image is generated and stored. The smaller the depth value is, the whiter (brighter) it is. The gray scale image based on the direct subject image of the right camera processed in FIG. It is a figure explaining the production | generation procedure of the virtual viewpoint image produced | generated by the virtual viewpoint image production | generation part of FIG. It is a figure which shows the mutual arrangement position of an experimental installation, and the obtained virtual viewpoint image when it produces | generates when a viewpoint is 60 degree | times according to the production | generation procedure of FIG. It is a figure which shows the mutual arrangement position of an experimental installation, and the obtained virtual viewpoint image when it produces | generates when a viewpoint is 30 degree | times according to the production | generation procedure of FIG. 2 is a flowchart for explaining a main part of an arithmetic control operation of the image processing computer of FIG. 1.

  Hereinafter, an embodiment of the present invention will be described with reference to conceptual drawings. In addition, since each figure is a conceptual diagram, the mechanical structure of a detail, the dimensional ratio of each part, etc. are not necessarily drawn correctly.

  FIG. 1 is a plan view of a configuration of a free viewpoint image generation apparatus 10 having two cameras CR and CL and two mirrors MR and ML, which is one of the simplest optical systems, as an embodiment of the present invention. It is the schematic demonstrated visually. In FIG. 1, a free viewpoint image generation apparatus 10 includes a pair of left and right cameras CR and CL disposed in front of the subject 12 for imaging the subject 12, and the subject 12 disposed behind the subject 12. When a pair of left and right mirrors MR and ML that reflect light toward the pair of left and right cameras CR and CL, respectively, and image signals representing images taken by the cameras CR and CL are input from the cameras CR and CL, respectively. An image processing computer (image processing arithmetic control device) 14 that processes those image signals according to a program stored in advance to generate a free viewpoint image, and an operation signal that specifies the position or angle of the free viewpoint, etc. Output from the input operation device 16 such as a keyboard and the image processing computer 14. And an image display device 18 such as a color liquid crystal monitor for displaying an image.

  The pair of left and right cameras CR and CL are each provided with an optical axis that is spaced apart by a predetermined interval T in the horizontal direction, that is, the x direction, and is substantially parallel to the horizontal direction and the z direction perpendicular to the x direction. The right camera CR is arranged so that the subject 12 and at least a part of the right mirror MR and the left mirror ML are located within the imaging field of view, and the left camera CL is at least the subject 12, at least the right mirror MR and the left mirror ML. A part is arranged so as to be located in the imaging field of view.

  The pair of left and right mirrors MR and ML are plane mirrors that are substantially parallel to the vertical direction, that is, the y direction, and are substantially the same so that the reflection point at which the light from the subject 12 is reflected toward the cameras CR and CL is approximately the center. It is inclined with respect to the z direction at an angle. FIG. 1 shows a pair of virtual cameras VCRR and VCLR at the mirror image positions of a pair of left and right cameras CR and CL on the right mirror MR, and a pair of virtual cameras VCRL and VCLL on the left and right sides of the left mirror MR. It is shown in the mirror image position of a pair of cameras CR and CL.

  The cameras CR and CL are, for example, an image sensor composed of a CCD element or the like arranged at a position where an image is formed by a lens, and an image for A / D converting a signal from the image sensor into a predetermined digital image signal. The image signal IMi (m, n) (where i = R, G, B) indicating the density of each pixel having a predetermined number m × n of pixels is output to the image processing computer 14. To do. For example, in the case of an image having 480 × 640 pixels, the values are in the range of m = 1 to 480 and n = 1 to 640.

  A captured image ER captured by the right camera CR is shown in FIG. The captured image ER captured by the right camera CR includes a background image HR including left and right mirrors MR and ML, a direct subject image GR representing the subject 12, an indirect subject image GRMR through the right mirror MR, Indirect subject image GRML through mirror ML is included. The captured image EL captured by the left camera CL is an image obtained by horizontally inverting FIG. The captured image EL captured by the left camera CL includes a background image HL including left and right mirrors MR and ML, a direct subject image GL representing the subject 12, an indirect subject image GLML through the left mirror ML, Indirect subject image GLMR through mirror MR is included. In FIG. 3, the reference numerals corresponding to the captured image EL captured by the right camera CL are shown in parentheses.

  FIG. 2 is a functional block diagram for explaining a main part of the image processing control function of the image processing computer 14. 2, the image processing computer 14 functions as an image processing calculation control device, and includes a subject image separation unit 30, a camera position calculation unit 32, a depth value calculation unit 34, and a virtual viewpoint image generation unit 36. Yes.

  The subject image separation unit 30 directly captures subject images GR and GL representing the subject 12 directly captured by the cameras CR and CL from the captured images ER and EL from the cameras CR and CL and the cameras CR and CL. Thus, the indirect subject images GRMR and GLMR indirectly imaged through the right mirror MR and the indirect subject images GRML and GLML indirectly imaged through the left mirror ML are separated. That is, the subject image separation unit 30 subtracts the image signal representing the background image HR (HL) shown in FIG. 4 that has been captured and stored in advance from the image signal representing the captured image ER (EL) shown in FIG. Image signals representing the direct subject image GR (GL), the indirect subject image GRMR (GLMR), and GRML (GLML) shown in FIG. 5 are calculated (extracted). Next, the subject image separation unit 30 outputs each pixel of the image signal representing the direct subject image GR (GL), the image signal representing the indirect subject image GRMR (GLMR), and the image signal representing the indirect subject image GRML (GLML). Each set is labeled, and among them, the image having the maximum number of pixels constituting the image (the number of pixels) or the image having the maximum height or area is directly used as an image signal representing the subject image GR (GL). Of the image signal representing the remaining indirect subject image GRMR (GLMR) and the image signal representing the indirect subject image GRML (GLML), the one located on the right side is determined as the indirect subject image GRMR (GLMR) and The position is determined as an image signal representing the indirect subject image GRML (GLML).

  In FIG. 6, the direct camera image GR, the indirect camera image GRMR, and the indirect camera image GRML obtained by the image separation processing of the camera image separation unit 30 are formed by the right camera CR and the right mirror MR. Are associated with the virtual camera VCRL related to the right camera CR formed by the left mirror ML. Similarly, the left camera in which the direct subject image GL, the indirect subject image GLMR, and the indirect subject image GLML obtained by the image separation process of the subject image separation unit 30 are formed by the left camera CL and the right mirror MR. A virtual camera VCLR related to CL and a virtual camera VCLL related to the left camera CL formed by the left mirror ML are shown respectively.

  Returning to FIG. 2, the camera position calculation unit 32 is positioned at the mirror image position of the pair of left and right cameras CR and CL in the right mirror MR based on the positions of the cameras CR and CL and the positions of the right mirror MR and left mirror ML. And the positions of the virtual cameras VCRL and VCLL positioned at the mirror image positions of the pair of left and right cameras CR and CL in the left mirror ML, respectively. That is, first, when the shapes of the right mirror MR and the left mirror ML are known, the camera position calculation unit 32 determines how the frames of the right mirror MR and the left mirror ML are in the captured images ER and EL. Based on the length and position of what is captured, geometrical relationships are used to determine the internal parameters of camera CR and CL (focal length f and distortion factor) and external parameters (xyz of cameras CR and CL). A position PCR (x, y, z) and PCL (x, y, z) in the three-dimensional space of the coordinate system, and directions of the optical axes of the cameras CR and CL are calculated.

  Next, the camera position calculation unit 32 obtains center points AR and AL that are intersections of the diagonal lines of the right mirror MR and the left mirror ML, respectively, and calculates the position PCR (xx) of the calculated cameras CR and CL in the xyz three-dimensional space. , Y, z) and PCL (x, y, z) to calculate the three-dimensional vectors VRRA, VRLA, VRLA and VLLA from the center points AR and AL of the right mirror MR and left mirror ML, respectively, Symmetric vectors VRRB, VRLB, VRLB, and VRLB are calculated on the planes of the left mirror ML and the right mirror MR of the original vectors VRRA, VRLA, VRLA, and VLLA, respectively, and the tip positions of the symmetrical vectors VRRB, VRLB, VRLB, and VRLB are calculated. Are determined as the positions of the virtual cameras VCRR, VCRL, VCLR, and VCLL. The positions of these virtual cameras VCRR, VCRL, VCLR, and VCLL are three-dimensional, similar to the positions PCR (x, y, z) and PCL (x, y, z) of the cameras CR and CL in the xyz three-dimensional space. The positions PVCRR (x, y, z), PVCRL (x, y, z), PVCLR (x, y, z), and PVCLL (x, y, z) in space are indicated.

  7 to 9 illustrate the process of determining the position of the virtual camera VCRR as a representative. FIG. 7 shows a captured image ER of the right camera CR including the frame of the right mirror MR, and FIG. The center point AR of the right mirror MR calculated using the geometric relationship based on the length and position of how the frame of the right mirror MR whose shape is known is reflected in the captured image ER. FIG. 9 shows a three-dimensional vector VRRA from the position of the right camera CR to the position of the center point AR, a symmetric vector VRRB that is symmetric in the plane of the right mirror MR, and a virtual camera located at the tip of the symmetric vector VRRB. VCRR.

  The depth value calculation unit 34 is a pair of the positions of the cameras CR and CL and the positions of the virtual cameras VCRR, VCLR, VCRL, and VCLL, and the direct subject images GR and GL and the indirect subject images GRMR, GRML, GLML, and GLMR. Depth value Z is calculated for each pair of images based on the locally correlated portions between the images forming the image, that is, the relative positions of the highly correlated portions of density and color. That is, the depth value calculation unit 34 correlates relatively in size and shape from among the direct subject images GR and GL and the indirect subject images GRMR, GRML, GLML, and GLMR, which are images of the subject 12 acquired so far. Select two similar images with high. Thus, in this embodiment, the direct subject images GR and GL directly captured by the cameras CR and CL, the indirect subject images GRMR and GLMR captured by the virtual cameras VCRR and VCLR, and the virtual cameras VCRL and VCLL are used. Three pairs of images of the captured indirect subject images GRML and GLML are selected.

The depth value Z is similarly calculated for each of the three selected pairs of images. In the following, a case where the direct subject image GL shown in FIG. 10A and the direct subject image GR shown in FIG. Following the selection of the paired images, the depth value calculation unit 34 uses the same height in the image in units of one pixel (one pixel) in one of the direct subject image GL and the direct subject image GR. Search for one having a high correlation coefficient in lightness or chromaticity in the other position (pixel matching) from the other image, or in units of a group of pixels (for example, 3 × 3 9 pixels) in one image The image having the highest correlation coefficient at the same height position (y position) in the image is searched from the other image, the one having the highest correlation coefficient is determined, and the parallax d is calculated. For example, a pair of pixels having the highest correlation with each other has a position Xl in the n direction (x direction) in the image plane of the direct subject image GL and a position in the n direction in the image plane of the direct subject image GR is Xr. 11 is calculated, and the parallax d and the focal lengths f _R of the cameras CR and CL obtained in advance from the geometric relationship shown in FIG. 12 are calculated. And a depth value Z (= (f × T) / d) is calculated based on f _L (f _R = f _L = f) and a distance T in the x direction between the cameras CR and CL.

  FIG. 11 illustrates the parallax d between the direct subject image GL and the direct subject image GR. In FIG. 11, as described above, the pair of pixels having the highest correlation with each other has the position in the n direction (x direction) in the image plane of the direct subject image GL as Xl, and in the image plane of the direct subject image GR. When the position in the n direction at Xr is Xr, the direct subject image GL and the direct subject image GR are overlapped so that the upper left reference point is the same, and the direct direction of the subject image GL in the image plane (x direction) ) Is defined as a point P in the image plane of the direct subject image GR, and between the point P in the image plane of the direct subject image GR and the position Xr in the n direction in the image plane of the direct subject image GR. The distance that is the relative position is indicated as parallax d.

FIG. 12 shows an example in which the cameras CR and CL are modeled as pinhole cameras, and an image at the same height position Y of the other direct subject image GR is searched with reference to one pixel in one direct subject image GL. Is shown. In FIG. 12, for example, one pixel IM (m _Y , n _Xl ) in the direct subject image GL and the pixel IM (m _Y , n _Xr ) in the direct subject image GR show the maximum correlation, and one pixel IM The x-direction position in the image plane of the direct subject image GL of (m _Y , n _Xl ) is Xl, and the x-direction position in the image plane of the direct subject image GR of the other pixel IM (m _Y , n _Xr ) is Xr and The geometric relationship is shown. Here, a straight line LR connecting the pin hole PHR of the camera CR and the pixel IM (m _Y , n _Xr ), a straight line LL connecting the pin hole PHL of the camera CL and the pixel IM (m _Y , n _Xl ), and a pin A straight line T connecting the hole PHR and the pinhole PHL forms a triangle having a vertex Q, a base T, and a height Z. If this triangle is rotated 180 degrees so that its apex Q coincides with the pinhole PHL to form a triangle indicated by a one-dot chain line, a triangle having a base T and a height Z, a base (X1-Xr), and a height f are provided. Since it has a similar relationship with the triangle, it is clear that the relationship T / Z = f / (X1-Xr) is established and the depth value Z is calculated as described above.

  The depth value Z calculated for each pixel in this way is reflected on each of the planar image, that is, the direct subject image GL that is a two-dimensional image, and the direct subject image GR, thereby obtaining a grayscale image as a three-dimensional depth image. A GD is generated and saved. FIG. 13 illustrates a pair of direct subject images GL shown in the upper part, and a gray scale image GD in which the depth value Z obtained from the direct subject image GR is directly reflected in the subject image GR. This gray scale image GD shows that the closer to black (the depth value Z is larger), the closer to white (the depth value Z is small).

  Of the indirect subject images GRMR, GRML, GLML, and GLMR, the depth values of the pair of indirect subject images GRMR and GLMR and the pair of indirect subject images GRML and GLML that are highly correlated with each other in size and shape are the same as described above. Each Z is determined, and a gray scale image GD as a three-dimensional depth image reflecting the depth value Z is generated and stored. The geometric relationship in this case is as follows: the virtual camera VCRR position PVCRR (x, y, z) calculated by the camera position calculation unit 32, the virtual camera VCRL position PVCRL (x, y, z), and the virtual camera VCLR. The position PVCLR (x, y, z) and the position PVCLL (x, y, z) of the virtual camera VCLL are used as a reference.

  Returning to FIG. 2, the virtual viewpoint image generation unit 36 is based on the direct subject images GR and GL and the indirect subject images GRMR, GRML, GLML, and GLMR and the depth value Z calculated by the depth value calculation unit 34. A virtual viewpoint image GK of the subject 12 viewed from a desired virtual viewpoint K input by an operator is generated using a known rendering (display) algorithm known as, for example, image-based rendering or model-based rendering. Image-based rendering directly generates a three-dimensional image at a desired virtual viewpoint position based on a plurality of pieces of three-dimensional image information. Model-based rendering uses a large number of polygons (polygons) from a plurality of pieces of three-dimensional image information. ) And the like are respectively generated, and an image obtained by viewing the plurality of three-dimensional models from a desired virtual viewpoint position is drawn.

  FIG. 14 is a diagram illustrating generation of a free viewpoint image by the image-based rendering executed in the virtual viewpoint image generation unit 36. In FIG. 14, for each grayscale image GD as a plurality of types of three-dimensional depth images, processing is performed using 3D warping processing, so that the pixel position at a predetermined three-dimensional spatial position HS is determined using the depth value Z. Each is calculated. The pixel group whose three-dimensional position is calculated reflects the subject 12 which is a three-dimensional object. Therefore, the pixel group around the vertical line passing through the center of the subject 12 and parallel to the y-axis is between the left camera CL and the right camera CR. An image viewed from a free viewpoint camera CF installed at a virtual viewpoint angle θ, which is a rotation angle, is calculated based on the pixel group whose three-dimensional position is calculated. The image viewed from the free viewpoint camera CF installed at the arbitrary virtual viewpoint angle θ toward the three-dimensional space position HS is the pixel group from which the three-dimensional position is calculated, and the position of the free viewpoint camera CF and its arbitrary position. By converting (projecting) the position of the pixel group based on the coordinate system (x′−y′−z ′) of the virtual viewpoint angle θ, the virtual viewpoint image GK of the subject 12 viewed from an arbitrary virtual viewpoint angle θ. Is generated.

FIG. 15 shows a virtual viewpoint image GK ₆₀ with the geometrical arrangement of the subject 12, the left camera CL and the right camera CR, the left mirror ML and the right mirror MR at the upper stage when the virtual viewpoint angle θ is 60 degrees. It is a figure shown in the lower stage. FIG. 16 shows the virtual viewpoint image GK with the geometrical arrangement of the subject 12, the left camera CL and the right camera CR, the left mirror ML and the right mirror MR in the upper stage when the virtual viewpoint angle θ is 30 degrees. it is a diagram showing a ₃₀ to lower. The unit of the numbers indicating the dimensions in FIGS. 15 and 16 is mm.

  FIG. 17 is a flowchart for explaining a main part of an image processing control operation by the image processing computer 14. This image processing control operation is started automatically in response to simultaneous imaging by the right camera CR and left camera CL or in response to a manual start operation after the simultaneous imaging. In step S1 of FIG. 17 (hereinafter, step is omitted), captured images ER and EL captured by the right camera CR and the left camera CL are read into a predetermined storage area. Next, in S2 corresponding to the subject image separation unit 30, the direct subject images GR and GL representing the subject 12 directly captured by the cameras CR and CL and the cameras CR and CL from the respective captured images ER and EL. The indirect subject images GRMR and GLMR that are indirectly imaged through the right mirror MR and the indirect subject images GLML and GRML that are indirectly imaged through the left mirror ML are the above-described captured images ER and EL and the background images HR and HL, It is separated by calculating the difference of.

  Subsequently, in S3 corresponding to the camera position calculation unit 32, the mirror image positions of the pair of left and right cameras CR and CL in the right mirror MR are determined based on the positions of the cameras CR and CL and the positions of the right mirror MR and the left mirror ML. The positions of the virtual cameras VCRR and VCLR located and the positions of the virtual cameras VCRL and VCLL located at the mirror image positions of the pair of left and right cameras CR and CL in the left mirror ML are calculated, respectively.

Next, in S4 corresponding to the depth value calculation unit 34, the positions of the cameras CR and CL and the positions of the virtual cameras VCRR, VCLR, VCRL, and VCLL, the direct subject images GR and GL, and the indirect subject images GRMR, GRML, and GLML Depth value Z is calculated for each pair of images based on the local interrelationship between the paired images of GLMR, that is, the relative position of the portion with high density and color correlation. Is done. That is, similar images having a relatively high correlation in size and shape are obtained from the direct subject images GR and GL and the indirect subject images GRMR, GRML, GLML, and GLMR, which are images of the subject 12 acquired so far. A pair is selected, and for each of the selected pair of images, one having a high correlation coefficient in lightness or chromaticity at the same height position in the image in units of one pixel (one pixel) in the other image. Search (pixel matching) from one of the images, or shade or color at the same height position (y position) in the image in units of one group of pixels (for example, 3 × 3 9 pixels) in one image The one with the highest correlation coefficient is searched from the other image, the one with the highest correlation coefficient is determined, and the parallax d as shown in FIG. 11 is calculated. Then, from the geometric relationship shown in FIG. 12, the parallax d and the focal lengths f _R and f _L (f _R = f _L = f) of the cameras CR and CL and the cameras CR and CL determined in advance. The depth value Z (= (f × T) / d) is calculated based on the distance T in the x direction. The depth value Z calculated for each pixel in this way is reflected on each of the planar image, that is, the direct subject image GL that is a two-dimensional image, and the direct subject image GR, thereby obtaining a grayscale image as a three-dimensional depth image. A GD is generated and saved.

  Next, in S5 corresponding to the virtual viewpoint image generation unit 36, based on the direct subject image GR, GL and the indirect subject image GRMR, GRML, GLML, GLMR and the depth value Z calculated by the depth value calculation unit 34, A virtual viewpoint image GK of the subject 12 viewed from a desired virtual viewpoint K input by an operator is generated using a known rendering (display) algorithm known as, for example, image-based rendering or model-based rendering. In S6, the virtual viewpoint image GK is displayed on the image display device 18.

  As described above, according to the free viewpoint image generation device 10 of the present embodiment, a pair of direct subjects captured directly by the image processing computer (image processing calculation control device) 14 by the pair of left camera CL and right camera CR. An image GL and a direct subject image GR, and two pairs of indirect subject images GLML, GRML and an indirect subject image GLMR captured indirectly by the left camera CL and right camera CR via a pair of left mirror ML and right mirror MR , GRMR, and the depth value Z obtained from these direct subject images GL, GR and indirect subject images GLML, GRML, GLMR, GRMR, a virtual viewpoint image GK is generated. According to such a configuration, the subject 12 is obtained because the direct subject image GL, GR and the indirect subject image GLML, GRML, GLMR, GRMR are instantaneously obtained by at least one imaging of the left camera CL and the right camera CR. It is not limited to an immovable object, and a pair of left camera CL and right camera CR need only be installed in front of left mirror ML and right mirror MR. 10 is obtained.

  In addition, according to the free viewpoint image generation device 10 of the present embodiment, the image processing computer (image processing arithmetic control device) 14 detects the left camera CL and the right from within the captured images EL and ER of the left camera CL and the right camera CR. Indirect subject images GLML, GRML, GLMR, and GRMR captured indirectly by the camera CR and indirectly captured by the left camera CL and the right camera CR via the left mirror ML and the right mirror MR. A subject image separation unit 30 to be separated, and a camera position calculation unit 32 that calculates the positions of the virtual cameras VCLL, VCRL, VCLR, and VCRR based on the positions of the left camera CL and the right camera CR and the positions of the left mirror ML and the right mirror MR. And a pair of direct subject images GL, GR and two pairs of indirect subject images GLML, GRML, Depth value calculation units that respectively calculate depth values Z based on the relative positions of locally correlated portions between subject images that form a highly correlated pair among a plurality of subject images including LMR and GRMR 34 and the depth value Z calculated by the depth value calculation unit based on the paired subject images GL and GR, GLML and GRML, GLMR and GRMR, and the depth value Z, from the angle θ of the virtual viewpoint A virtual viewpoint image generation unit 36 that generates an image of the viewed subject 12. Accordingly, the image processing computer 14 mutually selects a plurality of subject images including the pair of direct subject images GL and GR and the two indirect subject images GLML, GRML, GLMR, and GRMR separated by the subject image separation unit 30. From the virtual viewpoint angle θ based on the subject image forming a highly correlated pair and the depth value Z calculated by the depth value calculating unit 34 based on the relative positions of the locally interrelated portions between them. Since the virtual viewpoint image GK of the viewed subject 12 is generated, the direct subject image GL, GR and the indirect subject image GLML, GRML, GLMR, GRMR can be obtained instantaneously, so the subject 12 is not limited to an immovable object. Moreover, since a pair of left camera CL and right camera CR need only be installed in front of the left mirror ML and right mirror MR, the installation is easy and compact. In addition, an inexpensive free viewpoint image generation apparatus 10 can be obtained.

  Further, according to the free viewpoint image generating apparatus 10 of the present embodiment, the subject image separation unit 30 is imaged in the captured images EL and ER of the left camera CL and the right camera CR, and the left camera CL and the right camera CR. By calculating the difference image from the background images HL and HR that do not include the subject 12, subject images GL, GR, GLML, GRML, GLMR, GRMR of only the subject 12 are generated, and among the plurality of subject images Are extracted as direct subject images GL and GR, and relatively small ones are extracted as indirect subject images GLML, GRML, GLMR and GRMR, so that the direct subject images GL and GR are indirect. The subject images GLML, GRML, GLMR, and GRMR are easily separated.

  Further, according to the free viewpoint image generation apparatus 10 of the present embodiment, the camera position calculation unit 32 determines the left camera based on the positions of the left camera CL and the right camera CR and the positions of the left mirror ML and the right mirror MR. Since the positions of the virtual cameras VCLL, VCRL, VCLR, VCRR at the mirror image positions with respect to the left mirror ML and the right mirror MR of the CL and right camera CR are calculated, the virtual cameras VCLL, VCRL, VCLR, VCRR The position is easily calculated.

  Further, according to the free viewpoint image generation device 10 of the present embodiment, the camera position calculation unit 32 obtains the center points AL and AR, which are the intersections of the diagonal lines of the left mirror ML and the right mirror MR, respectively, and calculates the calculated values. A three-dimensional vector VLLA from the positions PCL (x, y, z) and PCR (x, y, z) in the xyz three-dimensional space of the cameras CL and CR to the center points AL and AR of the left mirror ML and the right mirror MR, respectively. , VRLA, VRLA, VRRA are calculated respectively, and symmetric vectors VLLB, VRLB, VRLB, VRRB that are symmetric with respect to the left mirror ML and the right mirror MR of the three-dimensional vectors VRLA, VRLA, VRRA, VRRA are calculated, Whether the tip positions of the symmetric vectors VLLB, VRLB, VRLB, VRRB are calculated as the positions of the virtual cameras VCLL, VCRL, VCLR, VCRR The positions of these virtual cameras VCLL, VCRL, VCLR, and VCRR are the positions PVCLL (x, y, z), PVCRL (x, y, z), PVCLR (x, y, z) in the three-dimensional space, And PVCRR (x, y, z) are easily calculated.

Further, according to the free viewpoint image generation apparatus 10 of the present embodiment, the depth value calculation unit 34 includes the pair of direct subject images GL and GR separated by the subject image separation unit 30 and the two pairs of indirect subject images GLML and GRML. Among a plurality of subject images including GLMR and GRMR, one or a plurality of pixels constituting the locality of one subject image and the locality of the other subject image are formed between subject images that form a highly correlated pair. The parallax d, which is the relative position between a pair of one pixel or a plurality of pixels having the highest correlation with one pixel or a plurality of pixels, is calculated, and the parallax d and the left camera CL and the right camera are calculated. Since the depth value Z between the paired subject images is calculated based on the focal length f (= f _L = f _R ) of CR and the inter-camera distance T, the depth value Z can be easily obtained. But it can.

  Further, according to the free viewpoint image generation apparatus 10 of the present embodiment, the virtual viewpoint image generation unit 36 includes the direct subject images GL and GR and the two pairs of indirect subject images GLML, GRML, GLMR, GRMR, and among them. Based on the depth value Z obtained between each pair of similar images, the three-dimensional position of the pixel representing the subject 12 in three dimensions, or the pixel representing the three-dimensional figure using the plurality of polygons for the subject 12 Since the three-dimensional position is determined and the virtual viewpoint image GK of the subject 12 viewed from the predetermined virtual viewpoint angle θ is generated based on the three-dimensional positions of the pixels, the virtual viewpoint image GK is viewed from an arbitrary virtual viewpoint angle θ. The virtual viewpoint image GK of the subject 12 can be easily obtained.

  As mentioned above, although one Example of this invention was described in detail with reference to drawings, this invention is implemented also in another aspect.

  For example, in the free viewpoint image generating apparatus 10 of the above-described embodiment, a pair of right camera CR and left camera CL and a pair of right mirror MR and left mirror ML are used. However, in order to increase the accuracy of the free viewpoint image. Three or more cameras and mirrors may be used. In addition, when a free viewpoint image of the entire circumference of the subject 12 is not necessary, the present invention is applied even if the subject 12 includes at least one camera and at least one mirror. In the case of one camera and one mirror, one direct subject image directly captured by one camera and one indirectly captured by one camera through one mirror Based on the indirect subject image and the depth value Z obtained from the direct subject image and the indirect subject image, in the angle range between the one camera and the virtual camera at the mirror image position of the one mirror, An image of the virtual viewpoint of the subject 12 can be generated.

  The subject image separation unit 30 subtracts the image signals representing the background images HR and HL shown in FIG. 4 that have been captured and stored in advance from the image signals representing the captured images ER and EL shown in FIG. The image signals representing the direct subject images GR and GL and the indirect subject images GRMR and GLMR, GRML and GLML are calculated (extracted). For example, based on characteristics such as a color common to the pixels representing the subject 12, By extracting pixels having the characteristics, the image signals representing the direct subject images GR and GL and the indirect subject images GRMR and GLMR, GRML and GLML may be separated.

  In the camera position calculation unit 32 of the above-described embodiment, first, when the shapes of the right mirror MR and the left mirror ML are known, the frames of the right mirror MR and the left mirror ML are captured images ER and EL. Based on the length and position of how the image appears in the camera, the geometrical relationship is used to determine the camera CR and CL internal parameters (focal length f and distortion factor) and the external parameters (cameras CR and CL. The position PCR (x, y, z) and PCL (x, y, z) in the three-dimensional space of the xyz coordinate system and the directions of the optical axes of the cameras CR and CL) are calculated. When the positions and orientations of the cameras CR and CL are known in advance from their set positions, for example, when such information is input and stored in advance, the camera position calculation unit 32 Without calculating the position and orientation of the Luo cameras CR and CL, it may be configured to calculate the position of the virtual camera VCRL and VCLL virtual camera VCRR and VCLR.

  In addition, the depth value calculation unit 34 of the above-described embodiment includes the positions of the cameras CR and CL and the positions of the virtual cameras VCRR, VCLR, VCRL, and VCLL, the direct subject image GR, GL, and the indirect subject image GRMR, GRML, Depth value Z was calculated based on the relative position (parallax d) of GLML and GLMR, which are locally related to each other, that is, where the density and color are highly correlated, but the focal length can be changed. For example, the depth value Z may be calculated based on the focal length of each local part in the subject 12 obtained from a simple camera.

  The virtual viewpoint image generation unit 36 is input by an operator using image rendering processing or model-based rendering processing based on the direct subject images GR and GL and the indirect subject images GRMR, GRML, GLML, and GLMR and the depth value Z. Although the image GK of the subject 12 viewed from the desired virtual viewpoint K is generated, other processes such as 3D mesh warping, 3D mesh morphing, and MFFD morphing may be used.

  In addition, what was mentioned above is an illustration to the last and can be suitably changed as needed. In addition, although not illustrated one by one, the present invention can be variously modified without departing from the spirit of the present invention.

10: Free viewpoint image generation device 12: Subject 14: Image processing computer (image processing calculation control device)
30: Subject image separation unit 32: Camera position calculation unit (virtual camera position calculation unit)
34: Depth value calculator 36: Virtual viewpoint image generator CR: Right camera CL: Left camera MR: Right mirror ML: Left mirror

Claims (7)

A free viewpoint image generating device for converting and outputting an image of a subject viewed from a desired virtual viewpoint from imaging data of the subject,
At least one camera for imaging the subject;
At least one mirror located within the imaging field of view of the camera and reflecting light from the subject toward the camera;
At least one direct subject image directly captured by the camera, at least one indirect subject image indirectly captured by the camera via the mirror, and a depth obtained from the direct subject image and the indirect subject image A free viewpoint image generation device, comprising: an image processing calculation control device that generates an image of the virtual viewpoint based on a value. The image processing arithmetic control device
The direct subject image representing the subject directly captured by the camera and the indirect subject image representing the subject indirectly captured by the camera via the mirror are separated from the captured image of the camera. A subject image separation unit;
A virtual camera position calculation unit that calculates the position of the virtual camera based on the position of the camera and the position of the mirror;
A plurality of subject images including the at least one direct subject image and at least one indirect subject image separated by the subject image separation unit are locally related to each other between subject images that form a highly correlated pair. A depth value calculation unit for calculating each of the depth values based on the relative position of the part to perform,
A pair of subject images for which the depth value has been calculated by the depth value calculation unit, and a virtual viewpoint image generation unit for generating a virtual viewpoint image of the subject viewed from the virtual viewpoint based on the depth value; The free viewpoint image generation device according to claim 1, further comprising:   The subject image separation unit calculates a difference image between an image including the subject captured by the camera and a background image not including the subject captured by the camera, thereby obtaining a subject image of only the subject. 3. The free viewpoint image generating apparatus according to claim 2, wherein a relatively large one of the subject images is extracted as the direct subject image, and a relatively small one is extracted as the indirect subject image.   The freedom of claim 2 or 3, wherein the virtual camera position calculation unit calculates the position of the virtual camera at a mirror image position of the camera with respect to the mirror based on the position of the camera and the position of the mirror. Viewpoint image generation device.   The virtual camera position calculation unit calculates a three-dimensional vector from the camera position to the center of the mirror, calculates a symmetric vector symmetric with respect to the mirror surface of the three-dimensional vector and the mirror, The free viewpoint image generation apparatus according to claim 4, wherein the tip position is calculated as the position of the virtual camera.   The depth value calculation unit includes a plurality of subject images including the at least one direct subject image and the at least one indirect subject image separated by the subject image separation unit. A pair of one or more pixels having the highest correlation between one pixel or a plurality of pixels constituting the local area of one subject image and one pixel or a plurality of pixels constituting the local area of the other subject image The parallax that is a relative position between a plurality of pixels is calculated, and the depth value between the paired subject images is calculated based on the parallax, the focal length of the camera, and the inter-camera distance. The free viewpoint image generation device according to any one of 2 to 5.   The virtual viewpoint image generation unit uses a three-dimensional position of a pixel representing the subject in three dimensions based on the direct subject image, the indirect subject image, and the depth value, or uses a plurality of polygons for the subject. The virtual viewpoint image of the subject viewed from a predetermined virtual viewpoint is generated based on the three-dimensional position of the pixel represented by the three-dimensional figure, and based on the three-dimensional position of the pixel. Free viewpoint image generator.