JP2014072809A - Image generation apparatus, image generation method, and program for the image generation apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image generation apparatus and the like for generating a more natural stereoscopic image.SOLUTION: A first image 30a and a second image 30b are scanned in a scanning direction relating to an installation direction in which a plurality of imaging means (20) are installed and corresponding feature points in a first image 30a and a second image 30b are determined (S6), depth information in the feature points is calculated from a distance between the corresponding feature points (S7), and occlusion noise is reduced from the depth information (S20). On the basis of the processed depth information, depth information in pixels between the feature points in the scanning direction is complemented (S22) and in a direction different from the scanning direction, relevancy of the complemented depth information is increased (S23). Averaging in the direction different from the scanning direction is performed (S24) on the depth information of which the relevancy is increased in the direction different from the scanning direction, and a multi-viewpoint image with different viewpoints is generated from the depth information averaged in the direction different from the scanning direction and at least one of the first image and the second image (S9).

Description

  The present invention relates to an image generation apparatus, an image generation method, and a program for an image generation apparatus that generate an image for a stereoscopic image.

  Conventionally, in order to enable a realistic autostereoscopic view, using two images (stereo images) taken with two cameras, an intermediate position when taken from a position between the two cameras An image is generated and multi-viewed. For example, Patent Document 1 discloses that a first image acquisition unit including a first imaging lens unit that forms an image of a subject and a first imaging element, a second imaging lens unit, and an image formed thereby. A second image acquisition unit having a first lens array unit in which a plurality of lenses that receive the captured image are arranged in an array and a second imaging element, and the second image acquisition unit includes a first image acquisition unit A stereoscopic video imaging device is disclosed that is disposed away from the subject in the horizontal direction when viewed from the subject.

JP 2010-78768 A

  By the way, in the prior art, processing for grouping optical flows is performed. However, when there is occlusion and it is intended to simply obtain information on the depth from a stereo image, noise simply due to occlusion is included only by grouping. Therefore, it was difficult to generate a natural stereoscopic image.

  Therefore, the present invention has been made in view of the above-described problems and the like, and an example of the problem is to provide an image generation apparatus that generates a more natural stereoscopic image.

  In order to solve the above-described problem, the invention according to claim 1 is an image that receives input of a first image captured from a first position and a second image captured from a second position by a plurality of imaging means. An input means, a feature point extracting means for extracting feature points of the first image and the second image from the first image and the second image, and a scanning direction related to an installation direction in which the plurality of imaging means are installed And a distance between corresponding point search means for scanning the first image and the second image to obtain corresponding feature points in the scanning direction in the first image and the second image. Further, depth information calculation means for calculating depth information at the feature point, and occlusion noise caused by occlusion that exists in one of the first image and the second image but does not exist in the other image. Occlusion noise reducing means for reducing from the depth information, and scanning direction complementing means for complementing depth information in pixels between the feature points in the scanning direction based on the depth information processed by the occlusion noise reducing means, Relevance increasing means for increasing the relevance of the complemented depth information in a direction different from the scanning direction, and depth information whose relevance is increased in a direction different from the scanning direction. A first averaging means for averaging in a direction different from the first direction, depth information averaged in a direction different from the scanning direction, and at least one of the first image and the second image. And a multi-viewpoint image generation means for generating a viewpoint image.

  According to a second aspect of the present invention, in the image generating apparatus according to the first aspect, the occlusion noise reducing unit is configured to add the depth information at a feature point in the vicinity of the predetermined feature point in the scanning direction. Based on this, it is determined whether or not to delete the depth information of the predetermined feature point, and when it is determined to delete the depth information, the depth information of the predetermined feature point is deleted.

  According to a third aspect of the present invention, in the image generating apparatus according to the second aspect, the occlusion noise reducing unit includes depth information of the predetermined feature point and the predetermined feature point in the scanning direction. The depth information of the predetermined feature point is deleted when the difference from the depth information at the feature point in the vicinity is a predetermined value or more.

  According to a fourth aspect of the present invention, in the image generating apparatus according to any one of the first to third aspects, the relevance increasing unit is arranged in the vicinity of a predetermined pixel in a direction different from the scanning direction. The depth information in the predetermined pixel is changed on the basis of the depth information in the pixel.

  According to a fifth aspect of the present invention, in the image generating apparatus according to the fourth aspect, the relevance increasing means is provided for two pixels in the vicinity of the predetermined pixel in a direction different from the scanning direction. When the difference between the average of the depth information and the depth information of the predetermined pixel is equal to or greater than a predetermined value, the depth information of the predetermined pixel is used as the average depth information.

  According to a sixth aspect of the present invention, in the image generating apparatus according to any one of the first to fifth aspects, a second average that performs averaging in the scanning direction with respect to the complemented depth information. It is characterized by further comprising a conversion means.

  The invention according to claim 7 is the image generating apparatus according to any one of claims 1 to 6, wherein the first image and the second image are moving images, and the direction is different from the scanning direction. Further, it is characterized by further comprising inter-frame averaging means for averaging the depth information averaged between the frames of the moving image.

  The invention according to claim 8 is the image generation apparatus according to any one of claims 1 to 7, wherein the multi-viewpoint image generation unit is configured to perform the third position with respect to the first position and the second position. The depth information averaged in a direction different from the scanning direction in the first image and the depth information averaged in a direction different from the scanning direction in the second image, according to the positional relationship. Depth information averaged in a direction different from the scanning direction in the three images is obtained, and depth information in the first image, the second image, and the third image, the first image, and the second image, An image is generated.

  The invention according to claim 9 is an image generation method in which the image generation apparatus generates an image, and the first image captured from the first position and the second position are captured by a plurality of imaging means. An image input step for receiving an input with the second image; a feature point extracting step for extracting feature points of the first image and the second image from the first image and the second image; and the plurality of imaging means. A corresponding point search step of scanning the first image and the second image in a scanning direction related to the set installation direction to obtain a corresponding feature point in the scanning direction in the first image and the second image; Depth information calculation step for calculating depth information at the feature point from the distance between the corresponding feature points, and one of the first image and the second image, but not in the other image The occlusion noise reduction step for reducing occlusion noise caused by the occlusion from the depth information and the depth information processed in the occlusion noise reduction step are used to complement the depth information in the pixels between the feature points in the scanning direction. A scanning direction complementing step, a relevance increasing step for increasing the relevance of the complemented depth information in a direction different from the scanning direction, and a depth information whose relevance is increased in a direction different from the scanning direction. On the other hand, a first averaging step for averaging in a direction different from the scanning direction, depth information averaged in a direction different from the scanning direction, and at least one of the first image and the second image And a multi-viewpoint image generation step for generating multi-viewpoint images having different viewpoints. Characterized in that it.

  According to a tenth aspect of the present invention, there is provided an image input unit that receives input of a first image captured from a first position and a second image captured from a second position by a plurality of imaging units. Feature point extraction means for extracting feature points of the first image and the second image from the first image and the second image, and the first image in a scanning direction related to an installation direction in which the plurality of imaging means are installed. And corresponding point search means for scanning the first image and the second image to obtain corresponding feature points in the scanning direction, and the depth at the feature points based on the distance between the corresponding feature points. Depth information calculating means for calculating information, occlusion noise caused by occlusion that exists in one of the first image and the second image but does not exist in the other image, Occlusion noise reducing means for reducing from information, scanning direction complementing means for complementing depth information in pixels between the feature points in the scanning direction based on depth information processed by the occlusion noise reducing means, and the scanning direction Relevance increasing means for increasing the relevance of the complemented depth information in different directions, and for depth information whose relevance is increased in a direction different from the scanning direction, an average in a direction different from the scanning direction A multi-viewpoint image having different viewpoints is generated from first averaging means for performing averaging, depth information averaged in a direction different from the scanning direction, and at least one of the first image and the second image. It functions as a multi-viewpoint image generation means.

  According to the present invention, since the occlusion noise caused by occlusion is reduced from the depth information at the feature points of the captured image, the relevance of the depth information is increased and averaged in a direction different from the scanning direction. Therefore, accurate depth information that is less uncomfortable for human beings is calculated, and a more natural stereoscopic image can be generated based on the depth information.

1 is a block diagram illustrating a schematic configuration example of an image generation system according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating an example of a schematic configuration of image generation in FIG. 1. FIG. 2 is a schematic diagram illustrating an example of a positional relationship between the position of the imaging device in FIG. 1 and the position of a virtual imaging device. 3 is a flowchart illustrating an operation example of the image generation apparatus in FIG. 1. It is a schematic diagram which shows an example of the image imaged with the imaging device of FIG. It is a schematic diagram which shows an example of the scanning direction with respect to the imaged image. It is a schematic diagram which shows an example of selection of a feature point. It is a schematic diagram which shows an example of a corresponding feature point. It is a graph which shows an example of the similarity degree at the time of searching for the corresponding feature point. It is a schematic diagram which shows an example of the depth map before performing a noise process. It is a schematic diagram which shows an example of the produced | generated depth map. It is a schematic diagram which shows an example of multiview video production | generation. It is a schematic diagram which shows an example of multiview video production | generation. It is a schematic diagram which shows an example of a stereoscopic view. It is a flowchart which shows an example of the subroutine of construction of a depth map. It is a schematic diagram which shows an example of the x direction low pass filter process in the line data of a depth map. It is a schematic diagram which shows an example of the x direction average process in the line data of a depth map. It is a schematic diagram which shows an example of the x direction FILL process in the line data of a depth map. It is a schematic diagram which shows the 1st modification of selection of a feature point. It is a schematic diagram which shows the 2nd modification of selection of a feature point.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiment described below is an embodiment when the present invention is applied to an image generation system.

[1. Overview of image generation system configuration and functions]
(1.1 Configuration and Function of Image Generation System 1)

  First, the configuration and outline function of an image generation system according to an embodiment of the present invention will be described with reference to FIG.

  FIG. 1 is a block diagram illustrating a schematic configuration example of an image generation system 1 according to the present embodiment.

  As shown in FIG. 1, the image generation system 1 captures an image of the subject 5 at a predetermined position and an image generation device 10 that generates an image when the image is captured from a virtual position from an image of the subject 5 captured at the predetermined position. Imaging device (an example of imaging means) 20.

  The image generation device 10 separates the received image data into left and right moving image images, and an image input unit 10a that receives input of image data of a plurality of moving image images captured by the two left and right imaging devices 20 (20a and 20b). LR separation means 10b, image reduction / filter means 10c for reducing the left and right images and applying a Gaussian filter in one frame of the moving image, and grayscale conversion means for obtaining a grayscale image from color image data of RGB signals 10d, the feature points of the left and right images are obtained, and feature point selection means 10e for selecting the feature points by predetermined processing from the obtained feature points, and pattern matching for obtaining the corresponding feature points from the left and right images of the gray scale Based on the means 10f and the corresponding feature points, the depth information is associated with the image data. Depth map construction means 10g constituting the map, depth map time averaging means 10h for averaging the depth map in the temporally previous frame, and a depth map storage for storing the depth map in the temporally previous frame Means 10i, a time-averaged depth map, an image composition means 10j that composes a three-dimensional display image from one frame left and right images, and a display unit 13 that displays the synthesized image.

  Here, the depth map is a map showing depth information for each pixel of the image.

  The imaging device 20 includes a left-eye imaging device 20a and a right-eye imaging device 20b, which are examples of imaging means, and constitutes a stereo camera. For example, the imaging device 20a and the imaging device 20b are installed at a predetermined distance from the subject 5 in the horizontal direction. The position where the imaging device 20a is installed is an example of the first position, and the position where the imaging device 20b is installed is an example of the second position. The imaging devices 20a and 20b are a still camera that captures still images or a video camera that captures moving images, and have the same performance with respect to resolution, color gradation, and the like. The outputs of the imaging devices 20a and 20b are digital outputs, but an analog output imaging device has an A / D converter and outputs a digital signal.

  The imaging device 20a captures the subject 5 from the left side and outputs an image for the left eye (an example of a first image captured at the first position). The imaging device 20b captures the subject 5 from the right side and outputs an image for the right eye (an example of a second image captured at the second position).

  The imaging device 20 a and the imaging device 20 b are connected to the image input unit 10 a of the image generation device 10.

(1.2 Configuration and Function of Image Generation Device 10)
Next, the configuration and function of the image generation apparatus 10 will be described with reference to FIG.
FIG. 2 is a block diagram illustrating an example of a schematic configuration of the image generation apparatus 10.

  As illustrated in FIG. 2, the image generation device 10 is, for example, a personal computer or a server device, and includes a communication unit 11, a storage unit 12, a display unit 13, an operation unit 14, and an input / output interface unit 15. The system control unit 16 is provided. The system control unit 16 and the input / output interface unit 15 are connected via a system bus 17.

  The communication unit 11 transmits and receives data and signals to and from the outside of the image generation apparatus 10. For example, it is used for network connection.

  The storage unit 12 includes, for example, a hard disk drive and stores an operating system, a control program, an image processing program, and the like. In the case of a moving image, the depth map one frame before is stored.

  The display unit 13 is configured by, for example, a liquid crystal display element or an EL (Electro Luminescence) element. The display unit 13 displays the image of the subject 5 captured by the imaging device 20 and the generated image. The display unit 13 includes a screen such as a lenticular lens or a parallax barrier.

  The operation unit 14 is configured by, for example, a keyboard and a mouse.

  The input / output interface unit 15 is a part directly connected to the output ends of the imaging device 20a and the imaging device 20b, and is an interface that realizes the function of the image input means 10a. This is an interface between the imaging device 20 (20a, 20b), the communication unit 11, the storage unit 12, and the system control unit 16.

  The system control unit 16 includes, for example, a CPU 16a, a ROM 16b, and a RAM 16c. In the system control unit 16, the CPU 16 a reads out and executes various programs stored in the ROM 16 b, the RAM 16 c, and the storage unit 12. For example, the system control unit 16 executes an image processing program and realizes functions such as a feature point selection unit 10e, a pattern matching unit 10f, a depth map construction unit 10g, a depth map time averaging unit 10h, an image synthesis unit 10j, and the like. To do. The system control unit 16 may read and execute various programs from the recording medium.

(1.3 Position of virtual imaging device)
Next, the position (virtual position) of the virtual imaging device will be described with reference to FIG.
FIG. 3 is a schematic diagram illustrating an example of a positional relationship between the positions of the imaging devices 20a and 20b and the position of the virtual imaging device.

  As shown in FIG. 3, the imaging device 20a and the imaging device 20b are installed at a distance L in the x direction (for example, the horizontal direction). For example, three virtual imaging devices 25 are set between the imaging device 20a and the imaging device 20b. The imaging device 25a, the imaging device 25b, and the imaging device 25c are set at equal intervals. Assuming that the first position 0 of the installation position of the imaging device 20a is the second position L of the installation position of the imaging device 20b, as an example of the third position, in the case of the imaging device 25a, the position L / 4, the imaging device 25b. In this case, the position L / 2 is set, and in the case of the imaging device 25c, the position 3L / 4 is set. Here, an image taken by the virtual imaging device 25 is taken as an example of the third image. If there are eight viewpoints, six virtual imaging devices 25 are installed.

[2. Operation of image generation system]
(2.1 Operation of image generation system)
Next, the operation of the image generation system according to the embodiment of the present invention will be described with reference to FIGS.
FIG. 4 is a flowchart illustrating an operation example of the image generation apparatus. FIG. 5 is a schematic diagram illustrating an example of an image captured by the imaging device. FIG. 6 is a schematic diagram illustrating an example of a scan direction with respect to a captured image. FIG. 7 is a schematic diagram illustrating an example of selection of feature points. FIG. 8 is a schematic diagram illustrating an example of corresponding feature points. FIG. 9 is a graph showing an example of similarity when searching for a corresponding feature point. FIG. 10 is a schematic diagram illustrating an example of a depth map before performing noise processing. FIG. 11 is a schematic diagram illustrating an example of the generated depth map. FIG. 12 is a schematic diagram illustrating an example of multi-view video generation. FIG. 13 is a schematic diagram illustrating an example of multi-view video generation.

  First, as an operation of the image generation system, the system control unit 16 of the image generation apparatus 10, as illustrated in FIG. 3, uses the positions of virtual imaging devices 25 a, 25 b, and 25 c (for example, L / 4, L / 2, 3L / 4) are set in advance.

  Next, as illustrated in FIG. 4, the image generation apparatus 10 receives input of left and right images (step S <b> 1). Specifically, as illustrated in FIG. 5, the system control unit 16 of the image generation apparatus 10 receives an image 30a (an example of a first image) obtained by imaging the subject 5 from the left side, and captures the image from the right side. The received image 30b (an example of the second image) is received from the imaging device 20b. The image 30a is an image captured from the position of the imaging device 20a (an example of the first position), and the image 30b is an image captured from the position of the imaging device 20b (an example of the second position). The image data is RGB signal data.

  As described above, the image generation device 10 functions as an example of an image input unit that receives input of the first image captured from the first position and the second image captured from the second position by the plurality of imaging units.

  The image 30a and the image 30b are side-by-side images of HDMI (High-Definition Multimedia Interface), and the image 30a and the image 30b are multiplied by 0.50 in the horizontal direction (x direction). The left and right images may be images created by computer graphics.

  Next, the image generation device 10 separates the image into the image L and the image R (step S2). Specifically, the system control unit 16 separates a side-by-side image into an image L (image 30a) and an image R (image 30b) as an example of the LR separation unit 10b. Note that this process may be omitted if it is already separated.

  Next, the image generation device 10 reduces the image (step S3). As an example of the image reduction / filter unit 10c, the system control unit 16 reduces the left and right high-resolution images 30a and 30b to, for example, 1/3 vertical and 1/3 horizontal in order to speed up the following processing. To do. Then, the system control unit 16 applies a Gaussian filter to the reduced image to generate a blurred image.

  Next, the image generation device 10 performs gray scale conversion (step S4). As an example of the gray scale conversion unit 10d, the system control unit 16 converts each image 30a, 30b that has been reduced and applied a Gaussian filter into a luminance value (brightness) and converted into a gray scale.

  Next, the image generation device 10 selects feature points (step S5). Specifically, as an example of the feature point selection unit 10e, the system control unit 16 scans images 30a and 30b converted to gray scale in the x direction as shown in FIG. The brightness difference between adjacent pixels is obtained. This difference in brightness is set as an eigenvalue. For example, eigenvalues in image data (line data) for one line scanned in the x direction are as shown in FIG. Then, as shown in FIG. 7, the system control unit 16 selects, as the feature point pixels 40 a and 41 a, the coordinates of pixels whose eigenvalues exceed the set threshold (for example, the eigenvalue is 10 or more). Extract. The x direction in the scan direction is an example of a direction connecting the first position and the second position. In addition, the eigenvalue may be obtained from a Hessian matrix, edge strength, or the like.

  As described above, the image generation device 10 functions as an example of a feature point extraction unit that extracts feature points of the first image and the second image from the first image and the second image.

  Next, the image generation device 10 searches for a corresponding feature point (step S6). As an example of the pattern matching unit 10f, the system control unit 16 uses, as shown in FIG. 8, for example, the pattern of the positions of the feature point pixels 40a and 41a obtained using the reduced grayscale image 30a on the left. Are retrieved from the reduced grayscale image 30b on the right by pattern matching and used as corresponding points. The pattern matching may be an existing pattern matching. For example, the system control unit 16 uses the pattern of the feature point position of the image 30a as a template, SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), Pattern matching is performed by a normalized cross correlation method.

As shown in FIG. 9, the system control unit 16 obtains the depth information of the feature point from the distance between the most similar pattern position (corresponding point) obtained by pattern matching and the feature point position (deviation of the x coordinate). calculate. The system control unit 16 defines the calculated depth information as a depth value S, and stores this depth information for each pixel (particularly, for each pixel that is a feature point). For example, when the left image 30a is a reference image, the right image 30b is a comparison image, the feature point position is Pref, and the matching point (corresponding point) position in the comparison image (image 30b) is Pcmp,
Depth value S is
S = Pref−Pcmp
It becomes. This depth value ranges from minus to plus. Positive if there is a corresponding point on the left side of the right image 30a, negative if it is on the right side.

  As shown in FIG. 10, a depth map prototype (a depth map before noise processing) is obtained by applying the depth value in each pixel. Note that, in the original depth map, depth values are not determined for pixels that are not feature points. Further, the original depth map includes occlusion noise caused by occlusion that exists in one of the images 30a and 30b but does not exist in the other image. For example, due to occlusion, there is a feature point that exists in the image 30a and does not exist in the image 30b. Therefore, the correspondence between the feature points becomes inaccurate, resulting in an incorrect depth value. There is also noise in which depth values are assigned to feature points that do not have corresponding points.

  As described above, the image generation apparatus 10 scans the first image and the second image in the scanning direction (for example, the x direction) related to the installation direction in which the plurality of imaging units are installed, and the first image and the first image are scanned. It functions as an example of corresponding point search means for obtaining corresponding feature points in the scanning direction in two images. Further, the image generation device 10 functions as an example of a depth information calculation unit that calculates depth information (for example, depth value) at a feature point from a distance between corresponding feature points.

  Next, the image generation device 10 constructs a depth map (step S7). Since there are noises such as occlusion noise and pixels whose depth values are not found in the original depth map, it is necessary to construct a natural depth map by removing noise and the like from the original depth map. Therefore, as an example of the depth map construction unit 10g, the image generation device 10 applies LPF (low-pass filter) in the horizontal direction (x direction) to the original depth map (horizontal low-pass filter processing). , Processing to apply horizontal average (horizontal averaging processing), processing to apply horizontal FILL (horizontal complement processing), processing to apply LPF in the vertical direction (y direction) (vertical low-pass filter processing), A process of applying an average in the vertical direction (vertical direction averaging process) is sequentially performed. By the process of constructing the depth map, a depth map for one frame in the moving image is generated as shown in FIG. The process for constructing the depth map will be described in detail in the subroutine for constructing the depth map. Note that the image generation apparatus 10 may construct a depth map for the left image 30a and the right image 30b, or a depth map for the left image 30a and the right image 30b.

Next, the image generation device 10 obtains a time average of the depth map (step S8). As an example of the depth map time averaging means 10h, the system control unit 16 reads the depth map (PAST_DEPTH) time-averaged one frame before from the storage unit 12, and the depth map one frame before and the constructed depth map Time averaging with the map (CURRENT_DEPTH) is performed to obtain a time averaged depth map (USE_DEPTH).
USE_DEPTH = α × PAST_DEPTH + (1-α) × CURRENT_DEPTH
Here, the blend rate α between the depth map one frame before and the constructed depth map is 0.0 to 1.0.

  Then, as an example of the depth map storage unit 10i, the system control unit 16 uses the storage unit 12 to calculate the time averaged depth map (PAST_DEPTH) one frame before, and obtain the time averaged depth map (USE_DEPTH). Used to blend the depth map of the next frame.

  In step 7, the constructed depth map may change greatly between the previous frame and the current frame when applied to a moving image. This is a factor that results in an unnatural stereoscopic image for the observer (viewer). In order to avoid this, a natural depth map is expressed by buffering the amount of change in depth by time averaging processing. Note that the time averaging process reduces quantization errors, noise due to compression, and errors due to flickering of fluorescent lamps.

  In this way, the image generating apparatus 10 uses the inter-frame averaging means for averaging the depth information averaged in the direction different from the scanning direction when the first image and the second image are moving images. Functions as an example.

  Next, the image generation device 10 synthesizes the depth map and the original image (step S9). As an example of the image composition unit 10j, the system control unit 16 reduces the image in step S3 and is a depth map constructed for the reduced image. Therefore, the system control unit 16 simply stretches the depth map to the resolution of the display image. Expanding.

  Then, as shown in FIG. 12, in the case of the depth map of the left image 30a, the system control unit 16 synthesizes the enlarged depth map and the image 30a to generate a multi-viewpoint image.

  Before generating the multi-viewpoint image, the image generation apparatus 10 enlarges the image 30a as necessary if the high-resolution image 30a is different from the size of the display screen. Generate a viewpoint image. For example, in the case of a side-by-side image, the width is only half, so it is doubled in the horizontal direction as necessary.

  As shown in FIG. 12, the system control unit 16 of the image generation apparatus 10 shifts the pixels of the line data of the image 30 a according to the depth value and the position of the viewpoint in the depth map. Is generated. As the depth value in the depth map increases, the pixel of the left image 30a is shifted to the left according to the viewpoint. A multi-viewpoint image can be generated by changing the degree of deviation. The image generation apparatus 10 may generate a multi-viewpoint image from the depth map of the right image 30b and the right image 30b, or the depth map of the left and right images 30a and 30b and the left and right images 30a and 30b. A multi-viewpoint image may be generated from the above.

The shift amount shift in the scanning direction of the pixel is obtained by the following calculation formula.
shift = ((Sight-1) / 8) x Coef x Depth (x, y)

  Here, Sight is a viewpoint number and varies from 1 to 8. Coef is a coefficient, for example, 1.0. Depth (x, y) is a depth value of xy coordinates, and a value of −10 to 10 is entered.

  By performing this calculation, the image generation apparatus 10 obtains the necessary shift amount of the pixel in the xy coordinates for each viewpoint, and generates an image for each viewpoint. In addition, since there is a vacant space where pixels do not exist just by shifting, the image generation apparatus 10 compares the depth values at both ends of the vacant space as shown in FIG. 13, and vacant according to the value of the pixel with the lower depth value. Fill. Further, in this multi-viewpoint image generation method, the shift amount can be easily obtained as in the above calculation formula, the memory is small, and the processing is accelerated.

  As described above, the image generation apparatus 10 generates a multi-viewpoint image having different viewpoints from the depth information averaged in a direction different from the scanning direction and at least one of the first image and the second image. It functions as an example of generating means.

  Next, the image generation device 10 displays the synthesized image (step S10). As illustrated in FIG. 14, the system control unit 16 displays a multi-viewpoint image, which is an example of a combined image, on the display unit 13. For example, in order to perform format conversion to a naked-eye type three-dimensional image, the image generation apparatus 10 assigns each of RGB data of pixels of the multi-viewpoint image according to each viewpoint (an example of the third position), and displays the display unit. 13 is displayed. By arranging the multi-viewpoint images in units of pixels, the images are multiplexed and displayed on the screen of the display unit 13 to perform stereoscopic viewing.

  As described above, the image generation apparatus 10 performs the depth information averaged in a direction different from the scanning direction in the first image and the scanning in the second image according to the positional relationship of the third position with respect to the first position and the second position. The depth information averaged in the direction different from the scanning direction in the third image is obtained from the depth information averaged in the direction different from the direction, and the depth information in the first image, the second image, and the third image is obtained. And, it functions as an example of a multi-viewpoint image generation unit that generates a stereoscopic image from the first image and the second image.

(2.2 Subroutine for depth map construction)
Next, a depth map construction subroutine will be described with reference to FIGS.
FIG. 15 is a flowchart illustrating an example of a depth map construction subroutine. FIG. 16 is a schematic diagram illustrating an example of an x-direction low-pass filter process on the depth map line data. FIG. 17 is a schematic diagram illustrating an example of an x-direction average process in the depth map line data. FIG. 18 is a schematic diagram illustrating an example of an x-direction FILL process for line data of a depth map.

  As shown in FIG. 15, the image generating apparatus 10 applies LPF in the horizontal direction (x direction) to the original depth map (step S20). Specifically, the system control unit 16 of the image generation apparatus 10 removes noise from the depth information obtained by pattern matching by applying a horizontal low-pass filter to the original depth map having depth information at feature points. To do. This process is performed for each line.

As shown in FIG. 15, there are intermittent depth values at feature point pixels. The image generation apparatus 10 sequentially refers to the depth value from the pixel at the head feature point in the scanning direction x. For example, assuming that the depth value depth of the pixel of the feature point to be processed (an example of the predetermined feature point) is the depth value pre_depth (predetermined value of the pixel of the previous feature point in the scanning direction x. From the depth value pre_pre_depth (an example of depth information at the feature point near the predetermined feature point) of the pixel of the previous feature point from the following: The absolute value of the difference is obtained as in the equation.
D1 = abs (pre_pre_depth−depth)
D2 = abs (pre_depth−depth)

  As shown in FIG. 16, when the values D1 and D2 are equal to or smaller than a predetermined threshold θ1 (D1 <= θ and D2 <= θ1), the image generating apparatus 10 leaves the depth value of the pixel at the feature point, Otherwise, the pixel at that feature point is deleted. The image generation apparatus 10 performs this process on the original depth map corresponding to each line data. Note that the depth values of the pixels of the first two feature points in the scanning direction x are not processed. Also, whether the depth value is left or deleted, the depth value pre_pre_depth is replaced with the depth value pre_depth, the depth value pre_depth is replaced with the depth value depth, and the process proceeds to the next feature point to be processed. The threshold value θ1 may be different for D1 and D2.

  As described above, the image generation apparatus 10 calculates the digital value from the depth value of the pixel of the feature point to be processed, the depth value of the pixel of the previous feature point, and the depth value of the pixel of the previous feature point. Occlusion noise reduction processing (lateral low-pass filter) is performed by performing processing such as a low-pass filter. In particular, spike noise is removed. In addition, the recessed portion is removed as compared with the depth value of the surrounding feature point like an undershoot. Note that the occlusion noise reduction processing may be performed based on the depth value of the pixel at the feature point to be processed, or the depth value of the pixel at the previous or next feature point. In addition, the occlusion noise reduction processing is performed with the depth value of the pixel of the feature point to be processed, the depth value of the pixel of the previous feature point, the depth value of the pixel of the previous feature point, and the second previous value. You may carry out by the depth value of the pixel of a feature point. Further, the occlusion noise reduction process may be a process of reducing the depth value without deleting the feature point pixels as in a low pass filter of a normal digital filter.

  As described above, the image generation apparatus 10 is an example of an occlusion noise reduction unit that reduces occlusion noise caused by occlusion that exists in one of the first image and the second image but does not exist in the other image from the depth information. Function. Further, the image generation apparatus 10 determines whether or not to delete the depth information of the predetermined feature point based on the depth information of the feature point in the vicinity of the predetermined feature point in the scanning direction, and determines to delete the information. In this case, it functions as an example of an occlusion noise reduction unit that deletes the depth information of a predetermined feature point. In addition, in the scanning direction, the image generation apparatus 10 determines that a difference (for example, D1 and D2) between depth information of a predetermined feature point and depth information of a feature point near the predetermined feature point is a predetermined value (for example, , Θ1) or more, it functions as an example of an occlusion noise reduction unit that deletes depth information of a predetermined feature point.

Next, the image generation system 1 applies the horizontal direction (x direction) average (step S21). The system control unit 16 averages the depth values of the feature point pixels in the scanning direction from the state in which the noise due to the occlusion by the horizontal low-pass filter is deleted. For example, as shown in FIG. 17, the system control unit 16 performs averaging according to the following equation using pixels of an odd number of feature points in order to make the central depth value in the average dominant.
Average depth value = (d1 + d2 + d2 + d3) / 4

  Here, the depth value d2 is the depth value of the pixel of the feature point to be processed, the depth value d1 is the depth value of the pixel of the previous feature point in the scanning direction x, and the depth value d3 is 1 in the scanning direction x. This is the depth value of the pixel at the next feature point.

  As described above, the image generation apparatus 10 functions as an example of a second averaging unit that performs averaging in the scanning direction with respect to the complemented depth information.

  In addition, although the horizontal direction average process may be omitted, a more natural depth map is generated by the effect of smoothing the depth value and the effect of removing noise more by this process. It should be noted that it is preferable to execute with a power of 2 pixels in order to facilitate calculation by hardware.

  Next, the image generating apparatus 10 applies the x direction FILL (step S22). As shown in FIG. 18, the depth values of the feature point pixels intermittently exist in the storage unit 12 that stores the depth map prototypes averaged in step S <b> 21. Since it is necessary to obtain a depth value in pixels that do not correspond to feature points, it is necessary to fill in the depth values in pixels between feature points. Specifically, as shown in FIG. 18, the system control unit 16 has a depth value on both sides or one side of a pixel for which a depth value is not set. Fill in the part where the value is not set and complement the depth value.

  As described above, the image generation device 10 functions as an example of a scanning direction complementing unit that complements depth information in pixels between feature points in the scanning direction based on the depth information processed by the occlusion noise reduction unit.

Next, the image generating apparatus 10 applies the vertical direction (y direction) LPF (step S23). In step S22, when the depth value is filled, a depth map having a low relation in the y direction is generated. In order to increase the relationship in the vertical direction (an example of a direction different from the scanning direction), vertical LPF processing is performed. Specifically, the system control unit 16 determines the depth value C_DEPTH of the pixel to be processed (an example of the predetermined pixel) and the depth value PRE_DEPTH (predetermined to the pixel one pixel above the processing target pixel (opposite to the y direction). An example of depth information in a pixel in the vicinity of a pixel), and a depth value NXT_DEPTH (an example of depth information in a pixel in the vicinity of a predetermined pixel) that is one pixel below the processing target pixel (y direction),
A_DEPTH = (PRE_DEPTH + NXT_DEPTH) / 2
D = abs (C_DEPTH-A_DEPTH)
Is calculated.

  When D> θ2, the system control unit 16 substitutes the depth value A_DEPTH for the depth value of the pixel to be processed. The system control unit 16 performs this process for all pixels.

  In this way, the image generation apparatus 10 performs processing like a low pass filter of a digital filter from the depth value of the pixel to be processed and the depth value of a pixel adjacent in the y direction, thereby relevance in the y direction. The relevance increasing process (vertical low-pass filter) for converting to a high depth value is performed. Note that the relevance increasing process may use the depth value of a pixel in the vicinity of the pixel to be processed, and is not limited to the y direction or adjacent pixels, and may be a pixel in an oblique direction or a neighboring pixel.

  As described above, the image generation device 10 functions as an example of a relevance increasing unit that increases the relevance of the complemented depth information in a direction (for example, the y direction) different from the scanning direction. Further, the image generation device 10 functions as an example of a relevance increasing unit that changes the depth information at a predetermined pixel based on the depth information at a pixel in the vicinity of the predetermined pixel in a direction different from the scanning direction. In addition, the image generation apparatus 10 may calculate the difference between the average depth information (for example, A_DEPTH) of two pixels in the vicinity of the predetermined pixel and the depth information of the predetermined pixel (for example, in a direction different from the scanning direction). When D) is equal to or greater than a predetermined value (for example, θ2), it functions as an example of a relevance increasing unit that uses the depth information of a predetermined pixel as average depth information.

  Next, the image generation device 10 applies the average in the vertical direction (y direction) (step S24). Specifically, the system control unit 16 averages the depth information subjected to the vertical low-pass filter. That is, the system control unit 16 averages lines using a plurality of vertical lines, and a depth map is constructed. For example, when it is desired to obtain an average value of a line (y axis value is y), the system control unit 16 sums seven depth values from y−3 to y + 3 in the y direction in each pixel of the line. Is calculated. Then, the system control unit 16 adds the depth value of y to this sum again and divides by 8 to obtain the average value of the line y. The system control unit 16 performs this vertical direction averaging process on all the lines, and constructs a depth map.

  Thus, the image generation apparatus 10 functions as an example of a first averaging unit that performs averaging in a direction different from the scanning direction with respect to depth information whose relevance is increased in a direction different from the scanning direction. .

  As described above, according to the present embodiment, the depth information in the feature point of the captured image is reduced in the depth information in the direction (y direction) different from the scanning direction (x direction) after reducing the occlusion noise caused by the occlusion. Since the relevance is increased and averaged, accurate depth information that is less uncomfortable for humans is calculated, and a more natural stereoscopic image can be generated based on this depth information.

  In addition, a stereo image can be converted into a multi-view stereoscopic video in real time. It also enables high-resolution stereo multi-viewpoint conversion in real time and with high accuracy.

  In step S3, since the depth map is constructed after the image is reduced, the construction of the depth map can be simplified. Also, by reducing the image, a depth map suitable for 3D can be generated at high speed.

  Whether or not to delete the depth information of the predetermined feature point based on the depth information of the feature point in the vicinity of the predetermined feature point in the scanning direction (the feature point on the line data) as the horizontal low-pass filter If the depth information of a predetermined feature point is deleted when it is determined to be deleted, the inaccurate depth information is deleted by occlusion, and the influence of occlusion on the horizontal direction averaging process and the horizontal direction complement process is deleted. Since it does not reach, a more natural stereoscopic image can be generated.

  As a horizontal low-pass filter, when the difference between the depth information of a predetermined feature point and the depth information of a feature point in the vicinity of the predetermined feature point is equal to or greater than a predetermined value in the scanning direction, When deleting the depth information, the difference in the depth information is obtained and deleted when the difference is large, so the occlusion can be deleted with high accuracy. Note that the protruding depth value (when the depth value is extremely large or extremely small compared to the depth value of the feature point in the vicinity in the x direction) is often occlusion. Further, when the differences D1 and D2 are obtained in two stages and the determination is performed on the differences D1 and D2, the occlusion can be deleted with higher accuracy.

  When changing depth information in a predetermined pixel based on depth information in a pixel near the predetermined pixel in a direction different from the scanning direction (for example, the y direction), the depth value of the pixel in the vicinity in the y direction is changed. Increased relevance and more natural depth maps.

  In addition, when the difference between the average of the depth information of two pixels in the vicinity of the predetermined pixel and the depth information of the predetermined pixel is equal to or greater than a predetermined value in a direction different from the scanning direction, the depth of the predetermined pixel When the information is the average depth information, if the difference is large, the average is used instead as the depth value, so that a smoother and more natural depth map can be constructed.

  Further, when averaging in the scanning direction (x direction) is performed on the complemented depth information, noise can be further reduced and a more natural depth map can be constructed.

  In addition, the first image (image 30a) and the second image (image 30b) are moving images, and depth information averaged in a direction different from the scanning direction (for example, the y direction) is averaged between frames of the moving images. In this case, a natural stereoscopic image can be generated even with a moving image.

  The depth information averaged in a direction (for example, y direction) different from the scanning direction in the first image (image 30a) according to the positional relationship of the third position with respect to the first position and the second position, and the second image From the depth information averaged in a direction different from the scanning direction in (image 30b), depth information averaged in a direction different from the scanning direction in the third image (intermediate viewpoint image) is obtained, and the first information is obtained. When generating a stereoscopic image from the image, the depth information in the second image and the third image, the first image, and the second image, the stereoscopic image can be reproduced finely according to the visual angle.

  Next, a modified example of the feature point selection in step S5 will be described with reference to FIGS.

  FIG. 19 is a schematic diagram illustrating a first modification of feature point selection. FIG. 20 is a schematic diagram illustrating a second modification of feature point selection.

  As shown in FIG. 19, when selecting a feature point in step S5, the image generation device 10 performs selection by winning in the x direction among eigenvalues in each pixel, and satisfies the condition with the number of wins. It may be selected as a point. For example, as an example of the winning selection, there is a winning selection in which a pixel having a higher eigenvalue than that of the neighbor in the x direction is left. In this case, feature points are selected stably with little dependence on the picture characteristics of the image. FIG. 19 shows the feature points selected by winning once.

  In addition, as shown in FIG. 20, with respect to the feature point selected in step S5, a pixel located in a region between the feature point pixels 42a and 43a having a high eigenvalue is newly added to the feature point. You may add as pixels 45a and 46a. Pattern matching using a high eigenvalue as the center is not optimal, but it is effective to perform pattern matching using a strong pattern around the eigenvalue. Therefore, if a feature point is selected so as to include a part of the pattern, a good search in pattern matching can be performed. Thus, the accuracy and accuracy of the depth map can be increased by adding feature points. The feature point to be added may be a pixel adjacent to a feature point having a high eigenvalue, or may be one point or three points. Further, a feature point may be added at the center of the feature point pixels 42a and 43a.

  The imaging device 20a and the imaging device 20b may be installed in a direction other than the horizontal direction (x direction). For example, when installed in the vertical direction (y direction), the scan direction is preferably the y direction.

  Further, the present invention is not limited to a lenticular lens type three-dimensional display, and may be applied to a parallax barrier type three-dimensional display. Alternatively, it may be applied to a three-dimensional display that requires a plurality of viewpoints.

  Furthermore, the present invention is not limited to the above embodiments. Each of the embodiments described above is an exemplification, and any configuration that has substantially the same configuration as the technical idea described in the claims of the present invention and has the same operational effects can be used. It is included in the technical scope of the present invention.

1: Image generation system 10: Image generation device 20 (20a, 20b): Imaging device (imaging means)
30 (30a, 30b): image (first image, second image)
40a, 40b, 41a, 41b, 42a, 43a, 45a, 46a: feature point pixels (feature points)

Claims (10)

Image input means for receiving inputs of a first image picked up from a first position and a second image picked up from a second position by a plurality of image pickup means;
Feature point extraction means for extracting feature points of the first image and the second image from the first image and the second image;
The first image and the second image are scanned in a scanning direction related to an installation direction in which the plurality of imaging units are installed, and corresponding feature points in the scanning direction are detected in the first image and the second image. A corresponding point search means to be obtained;
Depth information calculating means for calculating depth information at the feature points from the distance between the corresponding feature points;
Occlusion noise reduction means for reducing occlusion noise caused by occlusion that is present in one of the first image and the second image but not in the other image, from the depth information;
Based on the depth information processed by the occlusion noise reducing means, scanning direction complementing means for complementing depth information in pixels between the feature points in the scanning direction;
Relevance increasing means for increasing relevance of the complemented depth information in a direction different from the scanning direction;
First averaging means for performing averaging in a direction different from the scanning direction with respect to depth information whose relevance is increased in a direction different from the scanning direction;
Multi-viewpoint image generation means for generating a multi-viewpoint image having a different viewpoint from depth information averaged in a direction different from the scanning direction and at least one of the first image and the second image;
An image generation apparatus comprising: The image generation apparatus according to claim 1,
The occlusion noise reducing means determines whether or not to delete the depth information of the predetermined feature point based on the depth information at the feature point in the vicinity of the predetermined feature point in the scanning direction, An image generating apparatus characterized by deleting depth information of the predetermined feature point when it is determined to be deleted. The image generation apparatus according to claim 2,
The occlusion noise reducing means is configured to detect the predetermined information when a difference between the depth information of the predetermined feature point and the depth information at a feature point in the vicinity of the predetermined feature point is a predetermined value or more in the scanning direction. An image generation apparatus characterized by deleting depth information of feature points. The image generation apparatus according to any one of claims 1 to 3,
The image generating apparatus according to claim 1, wherein the relevance increasing unit changes the depth information in the predetermined pixel based on the depth information in a pixel in the vicinity of the predetermined pixel in a direction different from the scanning direction. The image generation apparatus according to claim 4,
In the direction different from the scanning direction, the relevance increasing unit is configured such that a difference between an average of the depth information in two pixels near the predetermined pixel and the depth information of the predetermined pixel is equal to or larger than a predetermined value. In this case, the depth information of the predetermined pixel is the average depth information. In the image generation device according to any one of claims 1 to 5,
An image generating apparatus, further comprising second averaging means for averaging the complemented depth information in the scanning direction. In the image generation device according to any one of claims 1 to 6,
The first image and the second image are moving images,
An image generating apparatus, further comprising an inter-frame averaging means for averaging depth information averaged in a direction different from the scanning direction between frames of the moving image. In the image generation device according to any one of claims 1 to 7,
The multi-viewpoint image generation means, in accordance with a positional relationship of the third position with respect to the first position and the second position, depth information averaged in a direction different from the scanning direction in the first image; Depth information averaged in a direction different from the scanning direction in the third image is obtained from depth information averaged in a direction different from the scanning direction in the image, the first image, the second image, and An image generating apparatus that generates a stereoscopic image from depth information in a third image, the first image, and the second image. An image generation method in which an image generation device generates an image,
An image input step for receiving input of a first image captured from a first position and a second image captured from a second position by a plurality of imaging means;
A feature point extracting step of extracting feature points of the first image and the second image from the first image and the second image;
The first image and the second image are scanned in a scanning direction related to an installation direction in which the plurality of imaging units are installed, and corresponding feature points in the scanning direction are detected in the first image and the second image. A corresponding point search step to be obtained;
Depth information calculation step for calculating depth information at the feature points from the distance between the corresponding feature points;
An occlusion noise reduction step for reducing occlusion noise caused by occlusion that is present in one of the first image and the second image but not in the other image, from the depth information;
Based on the depth information processed in the occlusion noise reduction step, a scanning direction complementing step for complementing depth information in pixels between the feature points in the scanning direction;
A relevance increasing step for increasing relevance of the complemented depth information in a direction different from the scanning direction;
A first averaging step for performing averaging in a direction different from the scanning direction for depth information whose relevance is increased in a direction different from the scanning direction;
A multi-viewpoint image generation step of generating a multi-viewpoint image having a different viewpoint from depth information averaged in a direction different from the scanning direction and at least one of the first image and the second image;
An image generation method characterized by comprising: Computer
Image input means for receiving inputs of a first image picked up from a first position and a second image picked up from a second position by a plurality of image pickup means;
Feature point extraction means for extracting feature points of the first image and the second image from the first image and the second image;
The first image and the second image are scanned in a scanning direction related to an installation direction in which the plurality of imaging units are installed, and corresponding feature points in the scanning direction are detected in the first image and the second image. Corresponding point search means to be obtained,
Depth information calculating means for calculating depth information at the feature points from the distance between the corresponding feature points;
Occlusion noise reducing means for reducing, from the depth information, occlusion noise caused by occlusion that is present in one of the first image and the second image but not in the other image;
Scanning direction complementing means for complementing depth information in pixels between the feature points in the scanning direction based on depth information processed by the occlusion noise reducing means;
Relevance increasing means for increasing relevance of the complemented depth information in a direction different from the scanning direction;
First averaging means for performing averaging in a direction different from the scanning direction with respect to depth information whose relevance is increased in a direction different from the scanning direction; and
An image generation apparatus that functions as a multi-viewpoint image generation unit that generates a multi-viewpoint image having a different viewpoint from depth information averaged in a direction different from the scanning direction and at least one of the first image and the second image. Program.