JP2013223008A - Image processing device and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem of picture quality degradation in the vicinity of an object boundary in free viewpoint image synthesis technology that since a foreground color and a background color coexist in the vicinity of the object boundary, distance information contains a large estimation error and exact separation between foreground and background is difficult, resulting in picture quality degradation.SOLUTION: An image is divided into three layers: a main layer, a boundary foreground layer and a boundary background layer. Distance information based three-dimensional model generation and rendering processing are performed on the main layer, the boundary foreground layer and the boundary background layer in that order to generate a free viewpoint image. The boundary foreground layer and the boundary background layer are calculated from an object boundary, and the three-dimensional model generation and rendering processing are performed in sub-pixel units in an attempt to improve the picture quality of free viewpoint image synthesis.

Description

  The present invention relates to a free viewpoint image synthesis technique using image data and distance information captured from a plurality of viewpoints. In particular, the present invention relates to a free-viewpoint image synthesis technique for multi-viewpoint image data captured by a multi-view imaging device.

  In recent years, 3D content has been actively used mainly in the movie industry. The development of multi-view imaging technology and multi-view display technology is progressing in search of a higher sense of reality.

  In 2-viewpoint display, a glasses-type 3D display is the mainstream. By generating image data for the right eye and image data for the left eye, and switching the images that can be seen by the respective eyes by controlling the glasses, the observer can view a stereoscopic image. In multi-viewpoint display, a 3D display without glasses using a lenticular lens and a parallax barrier system has been developed, and is mainly used for digital signage.

  In the imaging apparatus, a multi-lens imaging apparatus such as a stereo camera for two-viewpoint imaging and a Plenoptic camera or a camera array system for multi-viewpoint imaging of three or more viewpoints has been developed. In addition, research on a field called “computational photography” in which a multi-viewpoint image can be captured without modifying the existing camera configuration by contriving with an image pickup apparatus has been actively conducted.

  When a multi-viewpoint image captured by a multi-view imaging device is displayed on a multi-view display device, it is necessary to adjust the difference in the number of viewpoints between the imaging device and the display device. For example, when a three-viewpoint image captured by a three-lens camera is displayed on a nine-viewpoint glassesless 3D display, images for six viewpoints that are not captured must be complementarily generated. Also, when displaying an image captured by a stereo camera on a glasses-type 3D display, both have two viewpoints, but since the parallax optimal for viewing differs depending on the display, the image is reconstructed from a viewpoint different from the captured image. May be output.

  In order to realize the use case as described above, a free viewpoint image synthesis technique has been developed as a technique for generating image data other than the captured viewpoint.

  As a related technology, standardization work of MPEG-3DV (3D Video Coding) is in progress. MPEG-3DV is a method for encoding depth information together with multi-viewpoint image data. Assuming output from multi-viewpoint image data to a display device with various number of viewpoints such as existing 2D display, spectacle-type 3D display, and 3D display without glasses, control of the number of viewpoints using free viewpoint image composition technology I do. As a technique for interactively viewing a multi-viewpoint video, a free-viewpoint image synthesis technique has been developed (Patent Document 1).

JP 2006-012161 A

  One of the problems in the free viewpoint image synthesis technique is image quality degradation near the object boundary. This is because the foreground color and the background color are mixed in the vicinity of the object boundary, the estimation error of the distance information becomes large, and it is difficult to accurately separate the foreground and the background. How to perform image synthesis in the vicinity of the object boundary is the key to improving image quality in the free viewpoint image synthesis technology.

  An image processing apparatus according to the present invention synthesizes the multi-viewpoint image data with a specifying unit that specifies a boundary region of an object in an image of multi-viewpoint image data obtained by imaging from a plurality of viewpoints. Synthesizing means for generating image data, wherein the synthesizing means is configured such that a rendering unit when generating pixel values of the boundary region is more than a rendering unit when generating pixel values of regions other than the boundary region. It is small.

  According to the present invention, free viewpoint image composition using multi-viewpoint image data can be performed with high image quality.

It is the figure which showed an example of the imaging device by a multi-view system provided with the several imaging part. It is a block diagram which shows the internal structure of an imaging device. It is a figure which shows the internal structure of an imaging part. It is a functional part lock figure which shows the internal structure of an image process part. It is a flowchart which shows the flow of a distance information estimation process. FIG. 6 is a diagram for explaining the progress of the distance information estimation process. (A) And (c) is a figure which shows the example of a viewpoint image, (b) is a figure which shows the state which filtered the viewpoint image and divided | segmented into the small area | region, (d) is a viewpoint image of another imaging part. The figure which shows the state which overlapped the small area | region in a viewpoint image, (e) is a figure which shows the state by which the shift | offset | difference which has arisen in (d) was eliminated. It is a figure which shows an example of a histogram, (a) has shown the histogram with a high peak, (b) has shown the histogram of a low peak, respectively. It is a figure explaining adjustment of the amount of initial parallax. It is a flowchart which shows the flow of an image separation process. It is a figure explaining a mode that each pixel in a viewpoint image is classified into three, a foreground pixel, a background pixel, and a normal pixel. 6 is a flowchart illustrating a flow of free viewpoint image generation processing according to the first embodiment. It is a figure explaining a mode that a viewpoint image is isolate | separated into three layers. It is a figure explaining the mode of rendering of the main layer. It is a figure which shows an example of the rendering result of the main layer. It is a figure explaining the mode of rendering of a boundary foreground layer. It is a figure explaining the mode of rendering of a boundary background layer. 10 is a flowchart illustrating a flow of image separation processing according to the second embodiment. It is a figure explaining the mode of the alpha estimation process which concerns on Example 2. FIG.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings.

  FIG. 1 is a diagram illustrating an example of a multi-eye imaging device including a plurality of imaging units according to the present embodiment.

  The casing of the imaging apparatus 100 includes nine imaging units 101 to 109 that acquire color image data and an imaging button 110. The nine imaging units 101 to 109 all have the same focal length and are equally arranged on the square lattice.

  When the user presses the imaging button 110, the imaging units 101 to 109 receive light information of the subject with a sensor (imaging device), the received signal is A / D converted, and a plurality of color images (digital data) are simultaneously displayed. To be acquired.

  With such a multi-eye imaging device, it is possible to obtain a color image group (multi-view image data) obtained by imaging the same subject from a plurality of viewpoint positions.

  Although the number of imaging units is nine here, the number of imaging units is not limited to nine. The present invention is applicable as long as the imaging apparatus has a plurality of imaging units. In addition, although an example in which nine imaging units are equally arranged on a square lattice has been described here, the arrangement of the imaging units is arbitrary. For example, they may be arranged radially or linearly, or may be arranged at random.

  FIG. 2 is a block diagram illustrating an internal configuration of the imaging apparatus 100.

  A central processing unit (CPU) 201 generally controls each unit described below.

  The RAM 202 functions as a main memory, work area, and the like for the CPU 201.

  The ROM 203 stores a control program executed by the CPU 201 and the like.

  The bus 204 is a transfer path for various data. For example, digital data acquired by the imaging units 101 to 109 is sent to a predetermined processing unit via the bus 204.

  The operation unit 205 corresponds to buttons, mode dials, and the like, and user instructions are input via these buttons.

  A display unit 206 displays captured images and characters. In general, a liquid crystal display is widely used as the display unit 206. Further, a touch screen function may be provided, and in that case, a user instruction using the touch screen can be handled as an input of the operation unit 205.

  A display control unit 207 performs display control of images and characters displayed on the display unit 206.

  The imaging unit control unit 208 controls the imaging system based on an instruction from the CPU 201 such as focusing, opening / closing a shutter, and adjusting an aperture.

  The digital signal processing unit 209 performs various processes such as white balance processing, gamma processing, and noise reduction processing on the digital data received via the bus 204.

  The encoder unit 210 performs processing for converting digital data into a predetermined file format. The external memory control unit 211 is an interface for connecting to a PC or other media (for example, hard disk, memory card, CF card, SD card, USB memory).

  The image processing unit 212 calculates distance information from the multi-view image data acquired by the imaging units 101 to 109 or the multi-view image data output from the digital signal processing unit 209, and generates free-viewpoint combined image data. . Details of the image processing unit 212 will be described later.

  Although there are other components of the image pickup apparatus than the above, they are not the main point of the present invention, and thus the description thereof is omitted.

  FIG. 3 is a diagram illustrating an internal configuration of the imaging units 101 to 109.

  The imaging units 101 to 109 include lenses 301 to 303, a diaphragm 304, a shutter 305, an optical low-pass filter 306, an iR cut filter 307, a color filter 308, a sensor 309, and an A / D conversion unit 310. The lenses 301 to 303 are a zoom lens 301, a focus lens 302, and a shake correction lens 303, respectively. The sensor 309 is a sensor such as a CMOS or CCD.

  When the light amount of the subject is detected by the sensor 309, the detected light amount is converted into a digital value by the A / D conversion unit 310, and is output to the bus 204 as digital data.

  In this embodiment, the configuration and processing of each unit will be described on the assumption that all images captured by the imaging units 101 to 109 are color images. However, a part of the images captured by the imaging units 101 to 109 is described. Or you may change all into a monochrome image. In that case, the color filter 308 is omitted.

  FIG. 4 is a functional block diagram showing the internal configuration of the image processing unit 212.

  The image processing unit 212 includes a distance information estimation unit 401, a separation information generation unit 402, and a free viewpoint image generation unit 403. In this embodiment, the image processing unit 212 is described as one component in the imaging apparatus, but the function of the image processing unit 212 may be realized by an external device such as a PC. That is, the image processing unit 212 in the present embodiment can be realized as one function of the imaging device or as an independent image processing device.

  Hereinafter, each component of the image processing unit 212 will be described.

  Color multi-viewpoint image data acquired by the imaging units 101 to 109 or color multi-viewpoint image data output from the digital signal processing unit 209 (9 viewpoints in this embodiment) is input to the image processing unit 212. Then, it is sent to the distance information estimation unit 401 first.

  The distance information estimation unit 401 estimates information indicating the distance from the imaging unit to the subject (hereinafter referred to as “distance information”) for each viewpoint image in the input multi-viewpoint image data. Details of the distance information estimation will be described later. Instead of providing the distance information estimation unit 401, equivalent distance information may be input from the outside.

  The separation information generating unit 402 divides each viewpoint image constituting the multi-viewpoint image data into three layers (a boundary foreground layer that is a foreground portion of a subject boundary, a boundary background layer that is a background portion, and a main layer that is not a subject boundary). Generate information (separation information) that is the basis for separating the data. Specifically, each pixel in each viewpoint image is classified into three types: foreground pixels / background pixels adjacent to the boundary of the subject (hereinafter referred to as “object boundary”) and normal pixels other than the foreground pixels / background pixels. And generating information that can identify which type each pixel corresponds to. Details of the separation information generation will be described later.

  The free viewpoint image generation unit 403 generates image data (free viewpoint image data) at an arbitrary viewpoint position by rendering each three-dimensional model of the main layer, the boundary foreground layer, and the boundary background layer. Details of the free viewpoint image generation will be described later.

(Distance information estimation process)
A distance information estimation method in the distance information estimation unit 401 will be described. FIG. 5 is a flowchart illustrating the flow of the distance information estimation process according to the present embodiment. In the following description, it is assumed that the input multi-viewpoint image data is image data of nine viewpoints captured by the imaging apparatus 100 having the nine imaging units 101 to 109 shown in FIG.

  In step 501, the distance information estimation unit 401 applies an edge holding type smoothing filter to one viewpoint image (target viewpoint image) in the input nine viewpoint image data.

  In step 502, the distance information estimation unit 401 divides the target viewpoint image into regions of a predetermined size (hereinafter referred to as “small regions”). Specifically, adjacent pixels (pixel groups) having a color difference equal to or less than a threshold are sequentially integrated, and finally the target viewpoint image is displayed in a small area (for example, an area of 100 to 1600 pixels) having a predetermined number of pixels. Split. The threshold value is set to a value suitable for determining that the compared colors are the same color, for example, “6” when RGB is quantized with 8 bits (256 colors). . Initially, adjacent pixels are compared and if the color difference is less than or equal to the threshold value, both pixels are integrated. Then, an average color is obtained for each of the integrated pixel groups, compared with the average color of adjacent pixel groups, and the pixel groups having a color difference equal to or less than the threshold value are integrated. Such processing is repeated until the size (number of pixels) of the pixel group reaches a small area composed of the above-mentioned fixed number of pixels.

  In step 503, the distance information estimation unit 401 determines whether or not the division into small areas has been completed for all nine viewpoint images included in the nine viewpoint image data. If the division into small areas has been completed, the process proceeds to step 504. On the other hand, if the area division has not been completed, the process returns to step 501 to perform the process of applying a smoothing filter and the process of dividing into small areas using the next viewpoint image as the target viewpoint image.

  In step 504, the distance information estimation unit 401 refers to surrounding viewpoint images (here, viewpoint images positioned in the upper, lower, left, and right directions) for all viewpoint images, and calculates an initial amount of parallax for each divided small region. calculate. For example, when calculating the initial parallax amount of the viewpoint image related to the central imaging unit 105, the viewpoint images of the imaging units 102, 104, 106, and 108 are referred to. In the case of the viewpoint image related to the image capturing unit at the end, for example, the viewpoint image of the image capturing unit 107 is referred to each viewpoint image of the image capturing unit 104 or 108, and in the case of the viewpoint image of the image capturing unit 108, Each viewpoint image is referred to, and the initial amount of parallax is calculated. The initial amount of parallax is calculated as follows.

  First, each small region of the viewpoint image for which the initial amount of parallax is obtained is compared with the corresponding small region in the referenced viewpoint image (reference viewpoint image). Here, the corresponding small area is a small area in the reference viewpoint image shifted by the amount of parallax with respect to the position of each small area of the viewpoint image for which the initial amount of parallax is obtained.

  Next, the color difference between each pixel of the viewpoint image for obtaining the initial parallax amount and the corresponding pixel in the reference viewpoint image shifted by the parallax amount is calculated for all the pixels in the small region, and a histogram is created.

  Then, the amount of parallax is changed to create a histogram for each.

  In the histogram thus obtained, the parallax amount having a high peak is the initial parallax amount to be obtained. Note that the corresponding region in the viewpoint image to be referenced is set by adjusting the amount of parallax in the vertical direction and the horizontal direction. This is because the parallax amount of one pixel in the vertical direction and the parallax amount of one pixel in the horizontal direction do not indicate the same distance.

  The processing so far will be described using a specific example.

  FIG. 6A is a diagram illustrating an example of a viewpoint image of the imaging unit 105, in which an object 601 is shown. FIG. 6B is a diagram illustrating a state in which the viewpoint image of the imaging unit 105 is subjected to an edge holding filter and is divided into small regions. Here, it is assumed that one of the small areas is a small area 602 and the center coordinates of the small area 602 are 603. FIG. 6C is a diagram illustrating an example of a viewpoint image of the imaging unit 104. In the case of the imaging unit 104, the same object is captured from the right side of the imaging unit 105, so the object 604 in the viewpoint image of the imaging unit 104 is shown on the left side of the object 601 in the viewpoint image of the imaging unit 105.

  Now, the corresponding viewpoint image for the small area 602 is compared using the target viewpoint image as the viewpoint image of the imaging unit 105 and the reference viewpoint image as the viewpoint image of the imaging unit 104. FIG. 6D shows a state in which the small area 602 in the viewpoint image of the imaging unit 105 is superimposed on the viewpoint image of the imaging unit 104, and there is a shift in the corresponding area. Then, the pixel value of the small area 602 in the viewpoint image of the imaging unit 105 (with the edge holding filter) is compared with the pixel value in the viewpoint image of the imaging unit 104 (with the edge holding filter). To create a histogram. Specifically, the color difference of each pixel in the corresponding small area is acquired, and the horizontal axis represents the color difference and the vertical axis represents the number of matched pixels. In this way, the parallax amount is changed (for example, the small area is moved one pixel at a time), and a histogram for each parallax amount is sequentially created. FIG. 7 shows an example of a histogram. A histogram distribution having a high peak as shown in FIG. 7A has a high degree of reliability of the parallax amount, and a histogram distribution having a low peak as shown in FIG. 7B. Determines that the reliability of the amount of parallax is low. Here, the parallax amount of the histogram having a high peak is set as the initial parallax amount. 6E shows a state in which the shift generated in FIG. 6D is eliminated, and the small area 602 in the viewpoint image of the imaging unit 105 is not shifted to the corresponding area in the viewpoint image of the imaging unit 104. overlapping. The parallax amount indicated by the arrow 605 in FIG. 6E corresponds to the initial parallax amount to be obtained. Here, the small region is moved pixel by pixel to generate a histogram, but the amount of movement may be arbitrarily set, such as moving by 0.5 pixel.

  Returning to the flowchart of FIG.

  In step 505, the distance information estimation unit 401 adjusts the initial parallax amount repeatedly using the color difference between the small regions, the difference in the initial parallax amount, and the like. Specifically, based on the idea that neighboring small areas with close color differences have similar amounts of parallax, and neighboring small areas with close differences in expected amount of parallax are likely to have similar amounts of parallax. Adjust the amount of parallax.

  FIG. 8 is a diagram illustrating the adjustment of the initial parallax amount. FIG. 8A is a diagram illustrating a result of calculating the initial parallax amount for each small region in FIG. 6B (state before adjustment), and FIG. 8B is a diagram after adjustment is performed. It is a figure which shows the state of. In FIG. 8A, the parallax amounts of the three small areas in the object area 800 (the area inside the bold line) are represented by hatched lines 801, 802, and 803, respectively. Here, diagonal lines 801/803 are diagonal lines from the upper left to the lower right, and diagonal lines 802 are diagonal lines from the upper right to the lower left, which indicate that the parallax amounts of the two are different. In this case, for the background area (the area outside the thick line), the diagonal line from the upper right to the lower left is the correct amount of parallax, and for the object area, the diagonal line from the upper left to the lower right is the correct amount of parallax. In FIG. 8A, the correct amount of parallax can be calculated as the amount of parallax of the object region for the amounts of parallax 801 and 803, but the amount of parallax of the background region has been calculated for the amount of parallax 802. It can be seen that the correct amount of parallax is not achieved. In the adjustment of the amount of parallax, an error that occurs when the amount of parallax is estimated in units of such small areas is corrected using the relationship with the surrounding small areas. For example, in the case of FIG. 8A, the parallax amount 802 that has been the amount of parallax in the background region is adjusted using the parallax amount 801 and the parallax amount 803 of the adjacent small region, and as a result, FIG. As shown in (b), the correct amount of parallax 804 is directed from the upper left to the lower right.

  In step 506, the distance information estimation unit 401 performs a process of converting the parallax amount obtained by adjusting the initial parallax amount into a distance to obtain distance information. The distance information is calculated by (camera interval × focal length) / (parallax amount × 1 pixel length). Since the length of one pixel is different in the vertical and horizontal directions, the vertical and horizontal parallax amounts are the same. Necessary transformations are performed to indicate the distance.

  Further, the converted distance information is quantized to, for example, 8 bits (256 gradations). The distance information quantized to 8 bits is stored as 8-bit gray scale (256 gradations) image data. In the gray scale image of distance information, the color of the object is expressed as a color closer to white (value: 255) as the distance from the camera is closer, and a color closer to black (value: 0) as the distance from the camera is farther away. Is done. For example, the object area 800 in FIG. 8 is expressed in white, and the background area is expressed in black. Of course, the distance information may be quantized with other bit numbers such as 10 bits and 12 bits, or may be stored as a binary file without being quantized.

  In this way, distance information corresponding to each pixel of each viewpoint image is calculated. In this embodiment, the distance is calculated by dividing the image into small areas each having a predetermined number of pixels, but other estimation methods may be used as long as the distance is obtained based on the parallax between multi-viewpoint images. I do not care.

  The distance information and multi-viewpoint image data corresponding to each viewpoint image obtained by the above processing are sent to the subsequent separation information generation unit 402 and free viewpoint image generation unit 403. Note that distance information and multi-viewpoint image data corresponding to each viewpoint image may be sent only to the separation information generation unit 402, and these data may be sent from the separation information generation unit 402 to the free viewpoint image generation unit 403.

(Separation information generation process)
Next, processing for separating each viewpoint image into three layers in the separation information generation unit 402 will be described. FIG. 9 is a flowchart illustrating the flow of image separation processing according to the present embodiment.

  In step 901, the separation information generation unit 402 acquires multi-viewpoint image data and distance information obtained by distance information estimation processing.

  In step 902, the separation information generation unit 402 extracts an object boundary in the viewpoint image. In this embodiment, a portion where the difference between the distance information of the target pixel and the distance information of the neighboring pixels (hereinafter referred to as “distance information difference”) is equal to or greater than a threshold is specified as the boundary of the object. Specifically, it is as follows.

  First, scanning is performed in the vertical direction, and the difference in distance information is compared with a threshold value to identify pixels that are equal to or greater than the threshold value. Next, scanning is performed in the horizontal direction, and similarly, a difference in distance information is compared with a threshold value to identify pixels that are equal to or greater than the threshold value. Then, the union of the pixels specified in the vertical direction and the horizontal direction is taken and specified as the object boundary. The threshold is set to a value such as “10” when the distance information is quantized (0 to 255) with 8 bits.

  Here, the object boundary is obtained based on the distance information, but other methods such as dividing the image into object boundaries may be used. However, it is desirable that the object boundary obtained by image segmentation and the object boundary obtained from the distance information match as much as possible. When the object boundary is obtained by dividing the image, it is preferable to correct the distance information in accordance with the obtained object boundary.

  In step 903, the separation information generation unit 402 classifies each pixel in the viewpoint image into three types of foreground pixels, background pixels, and normal pixels. Specifically, referring to the distance information acquired in step 901, for the pixels adjacent to the object boundary specified in step 902, the pixel with the nearer distance is the foreground pixel and the pixel with the farther distance is the pixel. Each background pixel is determined. FIG. 10 is a diagram for explaining how each pixel in the viewpoint image is classified into three, a foreground pixel, a background pixel, and a normal pixel. With respect to adjacent pixels straddling the object boundary 1001, the closer pixel is classified as the foreground pixel 1002, the farther pixel is classified as the background pixel 1003, and the remaining pixels are classified as normal pixels 1004.

  In step 904, the separation information generation unit 402 determines whether pixel classification has been completed for all viewpoint images included in the input multi-viewpoint image data. If there is an unprocessed viewpoint image, the process returns to step 902, and the processing of step 902 and step 903 is performed on the next viewpoint image. On the other hand, if pixel classification has been completed for all viewpoint images, the process proceeds to step 905.

  In step 905, the separation information generation unit 402 sends the separation information that can specify the foreground pixel, the background pixel, and the normal pixel to the free viewpoint image generation unit 403. If the foreground pixel and the background pixel are known, the remaining pixels are determined to be normal pixels. Therefore, the separation information may be information that can identify the foreground pixel and the background pixel. That is, as the separation information, for example, a flag such as “0” for a pixel determined to be a foreground pixel and “1” for a pixel determined to be a background pixel may be added separately. In the free viewpoint image generation process described later, using such separation information, a predetermined viewpoint image is divided into three layers (ie, a boundary foreground layer composed of foreground pixels, a boundary background layer composed of background pixels, The main layer is composed of pixels.

(Free viewpoint image generation processing)
Next, free viewpoint image generation processing in the free viewpoint image generation unit 403 will be described. FIG. 11 is a flowchart illustrating the flow of the free viewpoint image generation processing according to the present embodiment.

  In step 1101, the free viewpoint image generation unit 403 acquires position information of an arbitrary viewpoint (hereinafter referred to as “free viewpoint”) in the output free viewpoint image. The position information of the free viewpoint is given by the following coordinates, for example. In this embodiment, it is assumed that coordinate information indicating the position of the free viewpoint when the position of the imaging unit 105 is used as a reference coordinate position (0.0, 0.0) is given. In this case, the imaging unit 101 is (1.0, 1.0), the imaging unit 102 is (0.0, 1.0), the imaging unit 103 is (−1.0, 1.0), and the imaging unit 104 is Represented by coordinates of (1.0, 0.0). Similarly, the imaging unit 106 is (−1.0, 0.0), the imaging unit 107 is (1.0, −1.0), the imaging unit 108 is (0.0, −1.0), and the imaging unit. 109 is represented by coordinates of (−1.0, −1.0). Here, for example, when the user wants to synthesize an image having an intermediate position between the four imaging units 101, 102, 104, and 105 as a free viewpoint, the user can input coordinates (0.5, 0.5). It will be good. As a matter of course, the method of defining coordinates is not limited to the above, and the position of the imaging unit other than the imaging unit 105 may be used as a reference coordinate position. Also, the method of inputting the position information of the free viewpoint is not limited to the method of directly inputting the coordinates described above. For example, a UI screen (not shown) indicating the arrangement of the imaging unit is displayed on the display unit 206, and touch operation is performed. For example, a desired free viewpoint may be designated.

  Although not described as an acquisition target in this step, distance information and multi-viewpoint image data corresponding to each viewpoint image are also acquired from the distance information estimation unit 401 or the separation information generation unit 402 as described above.

  In step 1102, the free viewpoint image generation unit 403 sets a plurality of viewpoint images (hereinafter referred to as “reference image group”) to be referred to when generating free viewpoint image data at the designated free viewpoint position. In this embodiment, viewpoint images captured by four imaging units close to the designated free viewpoint position are set as a reference image group. As described above, the reference image group when the coordinates (0.5, 0.5) are designated as the position of the free viewpoint is configured by four viewpoint images captured by the imaging units 101, 102, 104, and 105. Will be. Of course, the number of viewpoint images constituting the reference image group is not limited to four, and may be three around the designated free viewpoint. Furthermore, it is sufficient if it includes the position of the specified free viewpoint. For example, the images are captured by four imaging units (for example, the imaging units 101, 103, 107, and 109) that are not closest to the specified free start point position. The viewpoint image may be set as the reference image group.

  In step 1103, the free viewpoint image generation unit 403 generates a three-dimensional model by separating each viewpoint image included in the set reference image group into three layers: a main layer, a boundary foreground layer, and a boundary background layer. I do. Of the three layers, the region composed of the boundary foreground layer and the boundary background layer can be rephrased as the boundary region of the object in the image. As will be described later, when generating the pixel value of the boundary region, a rendering unit smaller than the rendering unit used when generating the pixel value of the region (main layer) other than the boundary region is used. FIG. 12 is a diagram for explaining how the viewpoint image is separated into the three layers. This will be described in detail below.

  First, generation of a three-dimensional model of the main layer will be described.

  In the case of the main layer, a three-dimensional model is generated by building a quadrilateral mesh by connecting four pixels that are not on the object boundary to each other. In FIG. 12, for example, a quadrilateral mesh 1204 is constructed by connecting four pixels (two normal pixels 1004 and 1201 and two foreground pixels 1202 and 1203) that are not on the object boundary 1001. By repeating such a process, all quadrilateral meshes that become the three-dimensional model of the main layer are constructed. The size of the quadrilateral mesh at this time is 1 pixel × 1 pixel at the minimum. In this embodiment, all the main layers are constructed by a quadrilateral mesh having a size of 1 pixel × 1 pixel, but a larger quadrilateral mesh may be used. Alternatively, a shape other than the quadrilateral, for example, a triangular mesh may be constructed.

  The X coordinate and Y coordinate of the quadrilateral mesh constructed in the above manner correspond to global coordinates calculated from the camera parameters of the imaging apparatus 100, and the Z coordinate is the pixel coordinate obtained from the distance information. This corresponds to the distance to the subject. Then, the color information of each pixel is texture-mapped to a quadrilateral mesh to generate a three-dimensional model of the main layer.

  Next, generation of a three-dimensional model of the boundary foreground layer and the boundary background layer will be described.

  In the boundary foreground layer and boundary background layer in contact with the object boundary, a more detailed three-dimensional model is generated than in the case of the main layer. Specifically, a four-dimensional mesh is constructed by interconnecting four pixels on the object boundary, and a three-dimensional model is generated by subdividing the constructed quadrilateral mesh into sub-pixels. . In FIG. 12, for example, a quadrilateral mesh constructed by four pixels (background pixels 1003, 1205, 1206 and foreground pixel 1002) all hanging on the object boundary is subdivided into sub-pixels to obtain 0.5 pixels. A quadrilateral mesh with a size of 0.5 pixels is constructed. By repeating such a process, all quadrilateral meshes that are three-dimensional models of the boundary foreground layer and the boundary background layer are constructed. Each quadrilateral mesh in units of subpixels is classified into a boundary foreground layer and a boundary background layer based on the characteristics of the corresponding pixel. For example, a quadrilateral mesh 1207 in units of sub-pixels corresponding to the foreground pixel 1002 that is the foreground part of the object boundary is a boundary foreground layer. In addition, quadrilateral meshes 1208, 1209, and 1210 in units of sub-pixels corresponding to background pixels 1003, 1205, and 1206, which are background portions of the object boundary, are boundary background layers.

  The X-coordinate and Y-coordinate of the quadrilateral mesh constructed as described above correspond to the global coordinates calculated by interpolating from the X-coordinate and Y-coordinate of the neighboring pixels, and the Z coordinate is the distance. This corresponds to the distance to the subject in the corresponding pixel obtained from the information. Then, the color information of the corresponding pixel is texture-mapped to the quadrilateral mesh, and a three-dimensional model of the boundary foreground layer and the boundary background layer is generated. For example, the quadrilateral mesh 1207 in sub-pixel units uses the color of the foreground pixel 1002, similarly, 1208 is the color of the background pixel 1003, 1209 is the color of the background pixel 1205, and 1210 is the color of the background pixel 1206, respectively. A three-dimensional model of the layer or boundary background layer is generated.

  Returning to the flowchart of FIG.

  In step 1104, the free viewpoint image generation unit 403 performs main layer rendering. FIG. 13 is a diagram for explaining how the main layer is rendered. The horizontal axis represents the X coordinate and the vertical axis represents the Z coordinate. In FIG. 13, line segments 1301 and 1302 indicate the quadrilateral meshes of the main layer when the three-dimensional model is generated from the reference viewpoint indicated by the white triangle 1303. Here, it is assumed that an object boundary (not shown) exists between the foreground pixel 1304 and the background pixel 1305. As the main layer, a quadrilateral mesh 1301 in which the normal pixel 1306 and the foreground pixel 1304 are connected, and a quadrilateral mesh 1302 in which the normal pixel 1307 and the background pixel 1305 are connected are generated as a three-dimensional model. An image obtained by rendering such quadrilateral meshes 1301 and 1302 at the position of the free viewpoint specified in step 1101 (in FIG. 13, a black inverted triangle 1308) becomes the rendered image of the main layer. In the rendering process, the pixel portion where no color exists remains as a hole. Then, the rendering process as described above is performed for all the reference image groups to obtain the main layer rendered image group. In FIG. 13, arrows 1309/1310 indicate from which position the quadrilateral mesh 1302 can be seen at the viewpoint 1303 / viewpoint 1308. At the viewpoint 1308 on the left side of the viewpoint 1303, the quadrilateral mesh 1302 is located on the right side of the viewpoint 1303. Similarly, arrows 1311/1312 indicate from which position the quadrilateral mesh 1301 can be seen from the viewpoint 1303 / viewpoint 1308.

  FIG. 14 is a diagram illustrating an example of a main layer rendering result. Here, for simplification of explanation, the two viewpoint images shown in FIG. 6 (the viewpoint image of the imaging unit 105 and the viewpoint image of the imaging unit 104) are used as a reference image group, and the imaging unit 105 and the imaging unit 104 A rendering result when the intermediate viewpoint is an arbitrary viewpoint position is shown. In this case, an intermediate viewpoint image between the viewpoint image of the imaging unit 105 and the viewpoint image of the imaging unit 104 is generated. 14A shows the rendering result of the main layer in the viewpoint image of the imaging unit 105, and FIG. 14B shows the rendering result of the main layer in the viewpoint image of the imaging unit 104. In this case, the object 1401 is an object at the intermediate viewpoint, and on the left side of the object 601 of the imaging unit 105 (see FIG. 6A), the object 604 of the imaging unit 104 (see FIG. 6C). ) Located on the right side. 14A shows that the occlusion area 1402 remains on the right side of the object 1401 and the occlusion area 1403 remains on the left side of the object 1401 in FIG. 14B.

  Returning to the flowchart of FIG.

  In step 1105, the free viewpoint image generation unit 403 integrates the obtained main layer rendering image group to obtain main layer integrated image data. In the case of the present embodiment, rendering images (four) of the main layer generated from the four viewpoint images as reference images are integrated. The integration process is performed for each pixel, and the color after integration uses a weighted average of each rendering image, specifically, a weighted average based on the distance between the designated free viewpoint position and the reference image. For example, when the designated free start point position is a position equidistant from the four imaging units corresponding to each reference image, the weights are equal to 0.25. On the other hand, when the designated free start position is a position close to any one of the imaging units, the weight becomes larger as the distance becomes shorter. Of course, the method of obtaining the average color is not limited to this. A portion having a hole in each rendering image is not used for the integrated color calculation. That is, the color after integration is calculated by a weighted average from a rendering image without a hole. A portion where a hole is formed in all the rendered images remains as a hole. In the case of integration of the rendering images of the main layer shown in FIGS. 14A and 14B, the hole portion (occlusion region 1402) in FIG. 14A is shown in FIG. The hole portion (occlusion region 1403) in (b) of FIG. 14 is filled in with (a) of FIG. That is, since the intermediate viewpoint image is generated with the two reference images, the weight of the color calculation is 0.5, and the color of each pixel of the integrated image is not shown in FIG. The average color of 14 (a) and (b) is obtained. Then, when one has a hole and the other has no hole, the color of the pixel having no hole is adopted. Although the case where two rendering images are integrated has been described for simplification of description, the concept of the integration processing of four rendering images is the same. Note that the portion where the hole is not filled by the integration processing here is filled by integration processing of rendering results of the boundary foreground layer and boundary background layer, which will be described later.

  In this way, the main layer integrated image data is generated.

  In step 1106, the free viewpoint image generation unit 403 performs rendering of the boundary foreground layer. FIG. 15 is a diagram for explaining how the boundary foreground layer is rendered. As in FIG. 13, the horizontal axis represents the X coordinate and the vertical axis represents the Z coordinate, and an object boundary (not shown) exists between the foreground pixel 1304 and the background pixel 1305. In FIG. 15, a line segment 1501 indicates a quadrilateral mesh of the boundary foreground layer when a three-dimensional model is generated from a reference viewpoint 1303 indicated by a white reverse triangle. The boundary foreground layer 1501 is a quadrilateral mesh in sub-pixel units having distance information and color information of the foreground pixels 1304. An image obtained by rendering such a quadrilateral mesh 1501 in units of sub-pixels at the position of the free viewpoint specified in step 1101 (the black inverted triangle 1308 in FIG. 15) becomes the rendered image of the boundary foreground layer. Note that even in the rendering of the boundary foreground layer, the pixel portion where no color exists remains as a hole. Then, the rendering process as described above is performed for all of the reference image groups to obtain a rendering image group of the boundary foreground layer. In FIG. 15, arrows 1502/1503 indicate from which position the quadrilateral mesh 1501 can be seen at the viewpoint 1303 / viewpoint 1308. At the viewpoint 1308 on the left side of the viewpoint 1303, the quadrilateral mesh 1501 is located on the right side of the viewpoint 1303.

  In step 1107, the free viewpoint image generation unit 403 integrates the rendering image group of the boundary foreground layer to obtain integrated image data of the boundary foreground layer. The rendering images (four) of the boundary foreground layer generated from the four viewpoint images as the reference images are integrated by the integration processing similar to step 1105.

  In step 1108, the free viewpoint image generation unit 403 integrates the integrated image data of the main layer and the integrated image data of the boundary foreground layer to obtain two layers (main layer + boundary foreground layer) integrated image data. The integration process here is also performed for each pixel. At this time, since the main layer and the boundary foreground layer can more stably obtain highly accurate images, the integrated image of the main layer is preferentially used. Accordingly, the color of the boundary foreground layer is complemented only when the integrated image of the main layer has a hole and the integrated image of the boundary foreground layer has no hole. When a hole is formed in both the integrated image of the main layer and the integrated image of the boundary foreground layer, it remains as a hole. With the above processing, two-layer integrated image data is obtained.

  In step 1109, the free viewpoint image generation unit 403 renders the boundary background layer. FIG. 16 is a diagram illustrating the rendering of the boundary background layer. As in FIG. 13, the horizontal axis represents the X coordinate and the vertical axis represents the Z coordinate, and an object boundary (not shown) exists between the foreground pixel 1304 and the background pixel 1305. In FIG. 16, a broken line 1601 indicates a quadrilateral mesh of the boundary background layer when a three-dimensional model is generated from a reference viewpoint 1303 indicated by a white reverse triangle. The boundary background layer 1601 is a quadrilateral mesh in units of sub-pixels having distance information and color information of the background pixel 1305. An image obtained by rendering such a quadrilateral mesh 1601 in units of sub-pixels at the position of the free viewpoint specified in step 1101 (the black inverted triangle 1308 in FIG. 16) becomes the rendered image of the boundary background layer. In the case of rendering the boundary background layer, the pixel portion where no color exists remains as a hole. Then, the rendering process as described above is performed for all the reference image groups to obtain a rendered image group of the boundary background layer. In FIG. 16, arrows 1602/1603 indicate from which position the quadrilateral mesh 1601 can be seen at the viewpoint 1303 / viewpoint 1308. At the viewpoint 1308 on the left side of the viewpoint 1303, the quadrilateral mesh 1601 is located on the right side of the viewpoint 1303.

  In step 1110, the free viewpoint image generation unit 403 integrates the boundary background layer rendering images to obtain integrated image data of the boundary background layer. By the same integration process as in step 1105, the rendering images (four) of the boundary background layer generated from the four viewpoint images as the reference images are integrated.

  In step 1111, the free viewpoint image generation unit 403 integrates the two-layer integrated image data obtained in step 1108 and the integrated image data of the boundary background layer, and combines three layers (main layer + boundary foreground layer + boundary background layer). ) Obtain integrated image data. The integration processing here is also performed for each pixel. At this time, the two-layer integrated image is more stable and accurate in the two-layer (main layer + boundary foreground layer) integrated image and the integrated image of the boundary background layer. Therefore, the two-layer integrated image is preferentially used. Therefore, the color of the boundary background layer is complemented only when there is a hole in the two-layer (main layer + boundary foreground layer) integrated image and there is no hole in the integrated image of the boundary background layer. When a hole is formed in both the two-layer integrated image and the boundary background layer integrated image, the hole remains as a hole. Through the above processing, three-layer integrated image data is obtained.

  In the present embodiment, the rendering process is performed in the order of the main layer, the boundary foreground layer, and the boundary background layer in order to suppress image quality deterioration near the object boundary.

  In step 1112, the free viewpoint image generation unit 403 performs hole filling processing. Specifically, the portion remaining as a hole in the three-layer integrated image data obtained in step 1111 is complemented using surrounding colors. In the present embodiment, the filling process is performed by selecting a pixel having distance information in the back from surrounding pixels of the filling target pixel. Of course, other methods may be used for filling holes.

  In step 1113, the free viewpoint image generation unit 403 outputs the free viewpoint image data that has undergone the hole filling process to the encoder unit 210. In the encoder unit 210, an image is output after being encoded by an arbitrary encoding method (for example, JPEG method).

  According to the present embodiment, it is possible to synthesize the captured images between the viewpoints in the multi-viewpoint image data with high accuracy, and display images such as a display with no sense of incongruity on a display having a different number of viewpoints from the captured images, refocus processing, etc. High image quality of processing can be realized.

  In the first embodiment, the generation of a three-dimensional model in the vicinity of the object boundary is performed in detail, and the image quality of the free viewpoint image synthesis is improved by rendering the three layers separately. Next, an embodiment in which higher image quality is realized by introducing an alpha value (transparency) in the generation of a three-dimensional model near the object boundary will be described as a second embodiment. Note that the description of the portions common to the first embodiment (processing in the distance information estimation unit 401) is simplified or omitted, and here, the processing in the separation information generation unit 402 and the free viewpoint image generation unit 403 which are different points is mainly described. It will be explained in the following.

  FIG. 17 is a flowchart illustrating the flow of image separation processing according to the present embodiment.

  In step 1701, the separation information generation unit 402 acquires multi-viewpoint image data and distance information obtained by distance information estimation processing.

  In step 1702, the separation information generating unit 402 extracts an object boundary in the viewpoint image.

  In step 1703, the separation information generation unit 402 classifies each pixel in the viewpoint image into three types of foreground pixels, background pixels, and normal pixels.

  The steps so far are the same as steps 901 to 903 in the flowchart of FIG. 9 of the first embodiment.

  In step 1704, the separation information generation unit 402 estimates the alpha values of the foreground pixels and background pixels specified in step 1703.

  FIG. 18 is a diagram for explaining the state of alpha estimation processing according to the present embodiment.

  First, the foreground pixels (1804, 1805, 1811, 1817) and the background pixels (1803, 1809, 1810, 1816) are set as undetermined regions for alpha estimation. Then, normal pixels (1806, 1807, 1812, 1813, 1818, 1819) on the side where the foreground pixels are located across the object boundary 1801 are set as the foreground region for alpha estimation. Similarly, normal pixels (1802, 1808, 1814, and 1815) on the side where the background pixel is located across the object boundary 1801 are set as the background region for alpha estimation. Here, an area having a width of 2 pixels from the object boundary 1801 is set as the foreground area / background area, but the pixel width can be arbitrarily set. Then, the alpha values of the foreground pixels and background pixels set as the undetermined areas are estimated based on the information of the normal pixels set as the foreground areas / background areas. A known alpha estimation method (for example, Bayesian estimation) can be applied to the estimation. Since the specific content of the alpha estimation is not a feature of the present invention, the description thereof is omitted. In this way, alpha values are added to the foreground and background pixels.

  In step 1705, the separation information generation unit 402 determines whether pixel classification and alpha value estimation have been completed for all viewpoint images included in the input multi-viewpoint image data. If there is an unprocessed viewpoint image, the process returns to step 1702, and the processing of steps 1702 to 1704 is performed on the next viewpoint image. On the other hand, if pixel classification and alpha value estimation have been completed for all viewpoint images, the process proceeds to step 1706.

  In step 1706, the separation information generation unit 402 sends the foreground pixel, background pixel, and normal pixel separation information obtained in step 1703 and the alpha information obtained in step 1704 to the free viewpoint image composition unit 404.

  Next, a free viewpoint image generation process in this embodiment will be described. The basic process is the same as the free viewpoint image generation process in the first embodiment except that the alpha value is added to the foreground pixel and the background pixel. The description will focus on the points peculiar to the present embodiment.

  The acquisition of the position information of the free viewpoint in step 1101 and the setting of the reference image group in step 1102 are the same as in the first embodiment. However, as described above, in addition to the separation information, alpha information is also acquired from the separation information generation unit 402 (step 1101).

  The three-dimensional model generation of the main layer, the boundary foreground layer, and the boundary background layer in step 1103 is the same as in the first embodiment. However, in this embodiment, when generating a three-dimensional model of the boundary foreground layer and boundary background layer, the alpha value of the corresponding pixel is added to the quadrilateral mesh in units of subpixels.

The rendering processing of each layer and the rendering result integration processing in steps 1104 to 1110 are the same as those in the first embodiment. However, the pixel rendered in the boundary foreground layer in the two-layer integrated image obtained by integrating the main layer and the boundary foreground layer has an alpha value. Pixels rendered in the boundary background layer also have an alpha value.
In the generation of the three-layer integrated image data in step 1111, integration processing is performed for each pixel. If a hole is formed in the two-layer integrated image (integrated image of the main layer and the boundary foreground layer), the color of the boundary background layer is used. Complement. In addition, for a pixel that has no hole but is rendered by the boundary foreground layer and has an alpha value, the colors of the boundary foreground layer and the boundary background layer are obtained by a weighted average according to the alpha value. For example, when the alpha value of the pixels in the boundary foreground layer is large (the transparency is low), a weighted average is performed so that more pixel values in the boundary foreground layer are reflected. Conversely, when the alpha value of the pixels in the boundary foreground layer is small (the transparency is high), a weighted average is performed so that more pixel values in the boundary background layer are reflected. By such processing, it is possible to deal with the ambiguity of the color at the boundary portion, and the image quality can be improved.

  The hole filling process in step 1112 and the free viewpoint image data output process in step 1113 are the same as in the first embodiment.

  According to the present embodiment, it is possible to realize higher image quality by introducing an alpha value (transparency) in the generation of a three-dimensional model near the object boundary.

(Other embodiments)
The object of the present invention can also be achieved by executing the following processing. That is, a storage medium that records a program code of software that realizes the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU, MPU, etc.) of the system or apparatus is stored in the storage medium. This is the process of reading the code. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code and the storage medium storing the program code constitute the present invention.

Claims (13)

A specifying means for specifying a boundary region of an object in an image of multi-viewpoint image data obtained by imaging from a plurality of viewpoints;
Combining the multi-viewpoint image data to generate combined image data,
The image processing apparatus according to claim 1, wherein the synthesizing unit has a rendering unit for generating pixel values of the boundary region smaller than a rendering unit for generating pixel values of regions other than the boundary region. The boundary region is composed of a boundary foreground layer that is a foreground part adjacent to the boundary of the object, and a boundary background layer that is a background part adjacent to the boundary of the object;
The image processing apparatus according to claim 1, wherein the specifying unit separates the boundary foreground layer, the boundary background layer, and a main layer constituting an area other than the boundary area into three layers.   The synthesizing unit renders the image at the viewpoint position by rendering the main layer, the boundary foreground layer, and the boundary background layer in this order with respect to the viewpoint to be referred to generate an image at an arbitrary viewpoint position. The image processing apparatus according to claim 2, wherein the images are combined.   The said synthesis | combination means calculates | requires the color of the said boundary region by performing the weighted average according to the alpha value of the pixel corresponding to the said boundary region in the rendering of the said boundary region. The image processing apparatus according to any one of the above.   The image processing apparatus according to claim 1, wherein the specifying unit generates the separation information by using image distance information.   The image processing apparatus according to claim 1, wherein the specifying unit generates the separation information by using region division of an image. A specifying step for specifying a boundary region of an object in an image of multi-viewpoint image data obtained by imaging from a plurality of viewpoints;
Combining the multi-viewpoint image data to generate composite image data,
In the synthesis step, the rendering unit when generating the pixel value of the boundary area is smaller than the rendering unit when generating the pixel value of the area other than the boundary area. The boundary region is composed of a boundary foreground layer that is a foreground part adjacent to the boundary of the object, and a boundary background layer that is a background part adjacent to the boundary of the object;
The image processing method according to claim 7, wherein in the specifying step, the image processing method is divided into three layers of a main layer that constitutes an area other than the boundary foreground layer, the boundary background layer, and the boundary area.   In the synthesis step, the main layer, the boundary foreground layer, and the boundary background layer are rendered in this order with respect to the viewpoint to be referred to in order to generate an image at an arbitrary viewpoint position, and the image at the viewpoint position is rendered. The image processing method according to claim 8, wherein the images are combined.   10. The composition step according to claim 7, wherein in the rendering of the boundary region, a weighted average corresponding to an alpha value of a pixel corresponding to the boundary region is performed to obtain a color of the boundary region. The image processing method according to claim 1.   The image processing method according to claim 7, wherein in the specifying step, the separation information is generated using image distance information.   The image processing method according to claim 7, wherein in the specifying step, the separation information is generated using image segmentation.   A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 6.

Images