JP2015125494A - Image generation method, image generation device, and image generation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image generation method capable of generating an image, seen from a desired viewpoint position, that has the quality sufficient to give high sense of presence.SOLUTION: Provided is an image generation method executed by an image generation device equipped with a plurality of image-capturing means for capturing the image of a subject from positions differing from each other, a depth map image-capturing means for capturing the image of a depth map relative to the subject, and a viewpoint position setting means for setting a viewpoint position relative to the subject, the image generation method including: a selection step of selecting two or more of the image-capturing means in which a positional relation between the subject and the viewpoint position and a positional relation between the subject and the image-capturing means are close; an image generation step of generating, on the basis of the plurality of images and the depth map captured by the plurality of image-capturing means selected in the selection step, an image of the subject seen from the viewpoint position set by the viewpoint position setting means; and an image addition step of making an image obtained by adding up the plurality of generated images an image at the viewpoint position.

Description

  The present invention relates to an image generation method, an image generation apparatus, and an image generation program for generating an image at a desired viewpoint position from an existing image.

  2. Description of the Related Art Conventionally, a technique for displaying an image corresponding to a change in viewpoint on a two-dimensional display is known (see, for example, Patent Document 1). FIG. 6 (cited from FIG. 1 of Patent Document 1) is a conceptual diagram showing an image display image according to the prior art. In FIG. 6, when the user 1 changes the viewpoint (corresponding to three-axis movement), the depth position of the conversation partner 2 in the video (the conversation partner 2 and the background wall 3) is displayed in two dimensions according to the change of the viewpoint. By displaying on the display 4, the appearance of the conversation partner 2 and the background wall 3 can be reproduced as if they were real.

  When the user 1 faces the two-dimensional display 4 at the position P1, the conversation partner 2 is displayed as if viewed from the front, and when the user 1 is viewed from the left position P2, When the dialog partner 2 is displayed as if viewed from the left side, reflecting the positional relationship (depth) between the dialog partner 2 and the wall 3 in the background, and the user 1 is viewing from the right position P3 Reflecting the positional relationship (depth) between the conversation partner 2 and the wall 3 behind it, the conversation partner 2 is displayed as if viewed from the right side.

  In order to realize this conventional technique, two techniques of detecting the viewpoint position / attitude of the user 1 with respect to the display 4 and generating / displaying the video of the conversation partner 2 in accordance with the viewpoint position of the user 1 are important. Proposed to separate a person and background from a captured 2D video, generate a 3D video with depth by multilayering, and project and display it on the display surface according to the user's viewpoint position. .

  However, in the video display method described in Patent Literature 1, although the positional relationship and size between the user and the background can be expressed correctly, if the background has a three-dimensional structure, the viewpoint of the user 1 is Since the background does not change even if it changes, there is a problem that the background state cannot be expressed correctly. Such a problem can be solved by generating a background image corresponding to the line-of-sight direction. As a technique for solving such a problem, a virtual viewpoint image generation method for generating a video from a viewpoint at a desired position from a plurality of camera images is known (for example, see Patent Document 2).

  However, in the virtual viewpoint image generation method described in Patent Document 2, since the depth position is estimated by matching a plurality of images and the image is calculated, the depth cannot be acquired depending on the texture state of the image, There is a problem that the error is too large. In the virtual viewpoint image generation method described in Patent Document 2, the image quality is improved in consideration of the likelihood of the depth position, but an image from a specific viewpoint with a quality that gives a high sense of reality is created. There is a problem that it is difficult to do.

Japanese Patent No. 5237234 Japanese Patent No. 4052331

  By the way, in the virtual viewpoint image generation method described in Patent Document 1, since the depth position is estimated by matching a plurality of images and the image is calculated, the depth cannot be acquired depending on the texture state of the image, There is a problem that the error is too large. Further, in the virtual viewpoint image generation method described in Patent Document 2, the image quality is improved in consideration of the likelihood of the depth position, but the image from a specific viewpoint with a quality that gives a high sense of reality. There is a problem that it is difficult to create.

  The present invention has been made in view of such circumstances, and an image generation method and image capable of generating an image viewed from a desired viewpoint position with respect to a subject having a wide depth range with a quality that provides a high sense of presence. An object is to provide a generation device and an image generation program.

  The present invention includes: a plurality of image pickup means for picking up a subject from different positions; a depth map image pickup means for picking up a depth map for the subject; and a viewpoint position setting means for setting a viewpoint position for the subject. In the image generation method performed by the apparatus, a selection step of selecting a plurality of imaging units having a close positional relationship between the subject and the viewpoint position and a positional relationship between the subject and the imaging unit, and An image generation step of generating a plurality of images of the subject viewed from the viewpoint position set by the viewpoint position setting unit based on the plurality of images captured by the plurality of selected imaging units and the depth map. And an image adding step in which an image obtained by adding the plurality of generated images is an image at the viewpoint position; Characterized in that it has.

  The present invention is characterized in that the plurality of images generated in the image generation step are images on a virtual screen set in a plurality of different depth ranges.

  According to the present invention, in the image generation step, the luminance value of the target pixel of the image to be generated is a value obtained by weighted averaging from the luminance values of the target pixel of the plurality of images captured by the plurality of selected imaging units. It is characterized by doing.

  In the present invention, when the luminance value of the target pixel of the image to be generated in the image generation step is obtained by a weighted average, the value of the depth map captured by the depth map imaging unit corresponding to the imaging unit is the depth range. Is not included, the weighted average is calculated by setting the luminance value of the target pixel of the image captured by the imaging unit to zero.

  The present invention is characterized in that, in the image generation step, a weighting coefficient for the weighted average is a value corresponding to the proximity of the positional relationship.

  The present invention is characterized in that, in the image generation step, a weighting coefficient for the weighted average is set to a value inversely proportional to the proximity of the positional relationship.

  The present invention includes a plurality of imaging means for imaging a subject from different positions, a depth map imaging means for imaging a depth map for the subject, a viewpoint position setting means for setting a viewpoint position for the subject, the subject, A selection unit that selects a plurality of the imaging units that are close in positional relationship to the viewpoint position and a positional relationship between the subject and the imaging unit; and a plurality of imaging units that are captured by the plurality of imaging units selected in the selection step Based on the image and the depth map, the image generation means for generating a plurality of images of the subject viewed from the viewpoint position set by the viewpoint position setting means, and the plurality of generated images are added. Image adding means for setting an image as the image at the viewpoint position is provided.

  The present invention is an image generation program for causing a computer to execute the image generation method.

  According to the present invention, an effect that an image viewed from a desired viewpoint position with respect to a subject in a wide depth range can be generated with a quality that gives a high sense of presence.

It is a block diagram which shows the structure of one Embodiment of this invention. It is explanatory drawing which shows the outline of the image generation process which the image generation part 5 shown in FIG. 1 performs. It is a flowchart which shows the processing operation of the image generation part 5 shown in FIG. It is a figure which shows the relationship between the weighted average of two images, and an outline position. It is explanatory drawing in case there exists a viewpoint position behind camera position. It is a conceptual diagram which shows the image of the video display by a prior art.

  Hereinafter, an image generation apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. In this figure, reference numeral 1 denotes a subject for image generation. Reference numeral 2 denotes a virtual screen that displays an image to be generated. Reference numerals 3-1 and 3-2 denote a plurality of cameras that capture an image of the subject 1. This camera has not only a 2D video (texture) shooting function but also a depth video (depth map) shooting function. Although only two cameras are illustrated in FIG. 1, three or more cameras may be provided as necessary.

  In order to improve the quality of an image to be generated, it is desirable to have a large number of cameras. It is desirable that the plurality of cameras be arranged in a plane parallel to the screen 2. Reference numeral 4 denotes a viewpoint position setting unit that detects the viewpoint position of a person who views the generated image and sets the viewpoint position of the image to be generated. Reference numeral 5 denotes an image generation unit that generates an image of the subject 1 viewed from the viewpoint position set by the viewpoint position setting unit 4. Reference numeral 6 denotes a display device that displays a generated image generated by the image generation unit 5.

  Next, the principle of image generation of the image generation unit 5 shown in FIG. 1 will be described with reference to FIG. FIG. 2 is an explanatory diagram showing an outline of image generation processing performed by the image generation unit 5 shown in FIG. In FIG. 2, a plurality of depth ranges parallel to the depth direction are provided (in the example illustrated in FIG. 2, a virtual screen position is assumed for each of the depth range 1, the depth range 2, and the depth range 3). The depth range is determined based on the value of the depth map. FIG. 2 shows a cross section perpendicular to the screen 2, and shows an outline of a method of calculating the luminance of a pixel on the screen among the pixels in the depth range 2. A method for generating an image when viewed from the viewpoint position shown in FIG. 2 will be described. After calculating an image on the virtual screen 2 for each depth range, an image obtained by adding all the images in the depth range is used as an image at the viewpoint position.

  A method for calculating an image in each depth range will be described in detail below. The luminance of the target pixel on the virtual screen 2 is calculated from an image captured from a camera position in the vicinity of the light beam from which the light emitted from the target pixel reaches the viewpoint position. In the example illustrated in FIG. 2, the setting is made from images captured by two cameras, a first camera position (for example, the position of the camera 3-1) and a second camera position (for example, the position of the camera 3-2). An image of the viewpoint position is generated.

For the luminance L of the target pixel, the luminance (pixel value) of the pixel at the position of the target pixel on the virtual screen 2 of the image captured from the first camera position and the second camera position is set to L1 and L2. If the distances from the position and the second camera position to the light beam are d 1 and d 2, they are obtained by the equation (1). Here, the values of d1 and d2 are calculated by calculating the coordinates X1 and X2 of the first and second camera positions and the coordinate X0 of the viewpoint position as d1 = X0−X1 and d2 = X2−X0. . However, when the depth map value of the pixel of the image captured by the camera is outside this depth range, the pixel brightness (pixel value) is calculated as 0.

  In the case of a color image, the same calculation may be performed for each primary color. Further, in order to reduce the calculation amount, only the green color having the highest resolution among the three primary colors may be calculated, and the pixel values of one camera image may be used for the other primary colors. Further, when an image is provided with a luminance signal and a color difference signal as in YPbPr, the same calculation may be performed only for the luminance signal, and information of one camera may be used for the color difference signal. The distance between the camera and the light beam is the length of the perpendicular line drawn from the camera position to the light beam, but when the camera is arranged in a lattice shape, the distance between the intersection of the surface forming the lattice and the light beam and the camera position is used. You may approximate.

Although the number of nearby cameras is two, it may be more than two. In this case, assuming that the number of nearby cameras is n, the luminance L of the target pixel is obtained by equation (2).

  If the viewpoint position is the same as the camera position, equations (1) and (2) become unstable. In this case, the luminance value of the camera image arranged at the viewpoint position is used as the luminance L of the target pixel as it is. Use it. However, when the depth map value of the pixel of the image captured by the camera is outside the depth range, the pixel brightness (pixel value) is calculated as 0.

  Next, the processing operation of the image generation unit 5 shown in FIG. 1 will be described with reference to FIG. FIG. 3 is a flowchart showing the processing operation of the image generation unit 5 shown in FIG. The processing operation shown in FIG. 3 is a processing operation for generating one still image, but it is also possible to deal with a moving image by continuously repeating the processing operation shown in FIG.

  Since the image generation processing is performed for each depth range, first, the image generation unit 5 selects a depth range to be calculated (step S1). Subsequently, the image generation unit 5 selects one target pixel in the image on the screen to be generated (step S2). Then, the image generation unit 5 calculates a light ray (straight line) connecting the selected target pixel and the viewpoint position set by the viewpoint position setting unit 4 (step S3).

  Next, the image generation unit 5 selects a camera located in the vicinity of the obtained light beam (step S4). Here, the selection of the cameras located in the vicinity is performed by (1) calculating the distance from the light ray for each camera position and selecting a predetermined number in order of closeness. It is possible to apply techniques such as selecting the number, (3) arranging the cameras in a grid, and selecting the camera position corresponding to the vertex of the polygon formed by the unit cells of the grid.

  Next, the image generation unit 5 performs a weighted average of the luminance of the target pixel on the screen of the selected camera image with a predetermined weight using the distance from the light ray as a parameter, and obtains the luminance value of the target pixel (step). S5). Then, the image generation unit 5 determines whether or not an uncalculated pixel remains (step S6). If it remains, the process returns to step S2 and repeats the calculation of another target pixel. move on.

  Next, the image generation unit 5 determines whether or not the depth range for viewing calculation remains (step S7), and if it remains, returns to step S1 and calculates the uncalculated depth range. If all the depth ranges have been calculated, the images generated in each depth range are added and combined to form a generated image (step S8). An image created by this processing operation is an image captured from a plurality of camera positions, and is a weighted average of images that are shifted due to a difference in camera positions. The width of the image shift between adjacent cameras is 3 arc min. When the depth width of the depth range is selected so as to be a small value, the contour position continuously changes depending on the weighting ratio as shown in FIG. Will be generated. FIG. 4 is a diagram illustrating the relationship between the weighted average of two images and the contour position.

  In the above description, the case where the camera position and the viewpoint are aligned on the same straight line has been described, but they may be different. FIG. 5 is an explanatory diagram when the viewpoint position is behind the camera position. In such a case, image generation may be performed according to the processing operation shown in FIG. According to the processing operation shown in FIG. 3, the weighted average image changes depending on the pixel. For the first target pixel (pixel 1) shown in FIG. 5, the images at the first and second camera positions, for the second target pixel (pixel 2), the images at the second and third camera positions, the third The target pixel (pixel 3) is an image at the third and fourth camera positions, but an image can be generated by a similar processing operation even if the camera position changes back and forth.

  Assume that the images captured from each camera position used in the present embodiment are obtained by projecting the images captured at the respective camera positions onto the screen surface. Such an image can be easily obtained by capturing an image from a predetermined position with a general camera and performing keystone correction corresponding to a deviation of the optical axis of the camera from the normal line of the screen. Alternatively, the image may be taken with the optical axis of the camera parallel to the screen normal, and a portion corresponding to the display area of the screen may be cut out. In addition, the camera optical axis is parallel to the screen normal, and a shiftable lens is used to capture and capture the offset between the center of the screen display area and the optical axis, and the portion corresponding to the display area is cut out. May be.

  As described above, in generating an image viewed from a desired viewpoint position, when generating an image of a specific viewpoint from images of a plurality of cameras, depth information that causes image deterioration is not used. It is possible to generate an image viewed from a desired viewpoint position with a quality that is given.

  In the above description, all the cameras have the depth map photographing function. However, some cameras may be provided, and the equipment may be simplified by calculating by coordinate conversion from the positional relationship of the cameras.

  Next, a result of displaying an image at a desired viewpoint position generated by the image generation apparatus according to the above-described embodiment on the display device 6 will be described with reference to FIG. The viewpoint position of the user (the person who views the image) is detected, the viewpoint position is set by the viewpoint position setting unit 4, and an image corresponding to the viewpoint position is displayed on the display device 6 by the image generation unit 5. Since the image displayed on the display device 6 is an image corresponding to the viewpoint position, it has a high sense of orientation and a highly realistic display.

  Further, by making the display image displayed on the display device 6 the actual size of the subject 1, it was possible to obtain a higher sense of realism. Also, when the image is generated according to the position of the left and right eyes of the user and the images are presented to each of the left and right eyes using 3D glasses, the image can be viewed with a high sense of presence as if there is a real thing. I was able to. Furthermore, when a naked eye 3D display device is used as a display device and an image corresponding to each viewpoint position is generated and displayed, it is as if there is a real thing without wearing special equipment such as 3D glasses. I was able to see images with such a high sense of reality.

  As described above, in the generation of an image at a desired viewpoint position, an image is generated by weighted average of images captured at positions near a straight line (light ray) connecting the viewpoint position and the position on the screen at a predetermined ratio. I did it. As the predetermined ratio, an amount that is inversely proportional to the distance between the straight line (light ray) and the imaging position is used as a weight. In addition, as a predetermined ratio, the weight for an image having the closest distance between the straight line (light ray) and the imaging position is increased. Further, by limiting the depth range and calculating, it is possible to generate a high-quality image in which a double image is not conspicuous.

  With this configuration, in the generation of an image at a desired viewpoint position, it is possible to generate a high-quality image by using the depth information to suppress a shift between target images that are weighted averages. Thereby, it is possible to realize generation and display of a highly realistic expression image.

  You may make it implement | achieve the viewpoint position setting part 4 and the image generation part 5 in embodiment mentioned above with a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Therefore, additions, omissions, substitutions, and other modifications of the components may be made without departing from the technical idea and scope of the present invention.

  The present invention can be applied to applications where it is indispensable to generate an image at a desired viewpoint position from an existing image.

  DESCRIPTION OF SYMBOLS 1 ... Subject, 2 ... Screen, 3-1, 3-2 ... Camera, 4 ... Viewpoint position setting part, 5 ... Image generation part, 6 ... Display apparatus

Claims (8)

An image generated by an image generation apparatus comprising: a plurality of imaging means for imaging a subject from different positions; a depth map imaging means for imaging a depth map for the subject; and a viewpoint position setting means for setting a viewpoint position for the subject. A generation method,
A selection step of selecting a plurality of the imaging means having a positional relationship between the subject and the viewpoint position and a positional relationship between the subject and the imaging means;
A plurality of images of the subject viewed from the viewpoint position set by the viewpoint position setting unit are generated based on the plurality of images captured by the plurality of imaging units selected in the selection step and the depth map. An image generation step,
An image addition method comprising: an image addition step in which an image obtained by adding the plurality of generated images is an image at the viewpoint position.   The image generation method according to claim 1, wherein the plurality of images generated in the image generation step are images on a virtual screen set in a plurality of different depth ranges.   In the image generation step, a value obtained by weighted averaging from luminance values of target pixels of a plurality of images captured by the selected plurality of imaging means is set as a luminance value of the target pixels of the image to be generated. The image generation method according to claim 1 or 2.   When the luminance value of the target pixel of the image to be generated in the image generation step is obtained by a weighted average, the depth map value captured by the depth map imaging unit corresponding to the imaging unit is not included in the depth range. 4. The image generation method according to claim 3, wherein the weighted average is calculated by setting the luminance value of the target pixel of the image captured by the imaging unit to zero.   5. The image generation method according to claim 4, wherein, in the image generation step, a weighting coefficient for the weighted average is set to a value corresponding to the proximity of the positional relationship.   6. The image generation method according to claim 5, wherein, in the image generation step, a weighting coefficient for the weighted average is set to a value inversely proportional to the proximity of the positional relationship. A plurality of imaging means for imaging the subject from different positions;
A depth map imaging means for imaging a depth map for the subject;
Viewpoint position setting means for setting a viewpoint position with respect to the subject;
A selection means for selecting a plurality of the imaging means having a close positional relationship between the subject and the viewpoint position and a close positional relationship between the subject and the imaging means;
A plurality of images of the subject viewed from the viewpoint position set by the viewpoint position setting unit are generated based on the plurality of images captured by the plurality of imaging units selected in the selection step and the depth map. Image generating means for
An image generation apparatus comprising: an image addition unit configured to add an image obtained by adding the plurality of generated images to the image at the viewpoint position.   An image generation program for causing a computer to execute the image generation method according to claim 1.

Images