JP4806088B1 - Image conversion apparatus, image conversion apparatus control method, image conversion apparatus control program, and recording medium - Google Patents

Abstract

An information amount of a depth image is reduced by a simple process with little delay.
An image encoding device includes: a dividing unit that acquires a depth image; a depth value distribution generating unit that generates a distribution of appearance numbers of depth values in the depth image acquired by the dividing unit; and a depth value. A pixel classification unit 14 that divides the depth value distribution created by the distribution creation unit 13 into a plurality of sections according to the continuity of the depth values in the distribution, and determines a representative value of each section of the depth value distribution; A depth value conversion unit 16 that converts a depth image by converting the depth value included in each section of the depth value distribution divided by the unit 14 into a representative value. Thereby, since the depth value can be converted into the representative value, the information amount of the depth image can be reduced by a simple process with little delay.
[Selection] Figure 1

Description

  The present invention relates to an image conversion apparatus that converts a depth image indicating a depth value of an image, a control method for the image conversion apparatus, a program, and a recording medium.

  In recent years, by using images from a plurality of viewpoints (multi-viewpoint images), a highly realistic video expression that cannot be obtained only by a single viewpoint image that is an image from one viewpoint direction has been realized. Examples of video expression using a plurality of viewpoint images include stereoscopic image display and arbitrary viewpoint image display.

  Stereoscopic image display uses two images with parallax, and the observer views one image with the right eye and the other image with the left eye, so that each image is a planar image, but the observer In the brain, it gives the feeling of looking at a three-dimensional space. This will be specifically described with reference to FIG. FIG. 9 is an explanatory diagram showing an overview of stereoscopic image display. As shown in FIG. 9, the observer views the image 501 with the left eye and the image 502 with the right eye for the two images 501 and 502 with parallax, so that the image 501, It feels as if the objects 504 and 505 in 502 exist three-dimensionally.

  Arbitrary viewpoint image display is to create and display an image of a subject at an arbitrary viewpoint from a plurality of viewpoint images with different viewpoints and the distance between the camera and the subject in each viewpoint image. This will be specifically described with reference to FIG. FIG. 10 is an explanatory diagram showing an overview of arbitrary viewpoint image display. As shown in FIG. 10, from a plurality of viewpoint images 601v, 602v, and 603v having different viewpoints and depth images 601d, 602d, and 603d that indicate the distance between the camera and the subject in each viewpoint image, Images (in the example shown in FIG. 10, viewpoint images 604v and 605v) are created and displayed. Thereby, it is possible to display an image of a subject from a viewpoint that has not been shot.

  Non-Patent Document 1 describes a method of generating a viewpoint image (arbitrary viewpoint image) at an arbitrary viewpoint. The method described in Non-Patent Document 1 generates an arbitrary viewpoint image using two viewpoint images and depth images corresponding to them. Specifically, (1) the depth image is projected onto the virtual viewpoint, (2) the projected depth image is smoothed, and (3) the pixel value of the actual image is mapped to the smoothed depth image. (4) A method of repairing a pixel at a remaining position by using surrounding pixels. As described above, by using the viewpoint images of the two viewpoints and the depth image thereof, it is possible to generate an image from an arbitrary viewpoint near the viewpoints.

  In addition, using the technique for generating an arbitrary viewpoint image leads to the improvement of the above-described stereoscopic image display. This will be described with reference to FIG. FIG. 11 is a diagram for explaining a principle that leads to improvement of stereoscopic display by an arbitrary viewpoint image generation technique. As shown in FIG. 11, it is assumed that subjects 704 and 705 are photographed by two cameras 701 and 702 that are installed with a distance 706 apart, and viewpoint images 701v and 702v are obtained. If the distance 706 is larger than the distance between the left and right eyes of a human (generally said to be around 65 mm), even if the viewpoint image 701v is viewed with the left eye and the viewpoint image 702v is viewed with the right eye, it is blurred. It becomes a three-dimensional image or an image that cannot be seen as a three-dimensional image.

  Therefore, a viewpoint image 703v in which a subject is viewed at a viewpoint position 703 that is separated from the camera 701 by the same distance 707 as the distance between the left and right eyes of a human is created, and an appropriate stereoscopic image is obtained by using the viewpoint image 701v and the viewpoint image 703v. Can be observed.

  Further, even when the distance between the two cameras 701 and 702 is too small compared to the distance between the left and right eyes of the human, it corresponds to the distance between the left and right eyes of the human from the point of the camera 701 or the camera 702. By generating a viewpoint image at a point, it is possible to observe a stereoscopic image that provides a sufficient stereoscopic effect.

  Furthermore, by using the above-described principle, it is possible to observe a stereoscopic image from an arbitrary viewpoint and adjust the stereoscopic effect when observing the stereoscopic image from an arbitrary viewpoint.

  As described above, if a plurality of viewpoint images and corresponding depth images (depth information) are used, the image display expression function can be improved. However, since a depth image is required, there arises a problem that the amount of data during recording and transmission increases.

  In order to solve this problem, in Patent Document 1, when transmitting depth information, a transmission amount is preferentially assigned to frequency components with high perceptual sensitivity according to temporal frequency and spatial frequency characteristics with respect to changes in visual depth. Discloses a technique for encoding a depth value. In Patent Document 1, when compressing the information amount of depth information, a viewpoint image generated by using the depth information quality (that is, using the depth information) by assigning a code amount by paying attention to the sensitivity of human depth perception. Quality), while reducing the amount of information.

Japanese Patent Laid-Open No. 2001-61164 (published March 6, 2001)

Mori, et al .: Free viewpoint image generation by 3D warping using depth images, IEICE General Conference Information and Systems Proceedings 2, D-11-7, March 5, 2008

  However, since the method of Patent Document 1 analyzes the time frequency characteristics and the spatial frequency characteristics of the depth information in order to encode the depth information, the amount of processing increases compared to the method of encoding the depth information as it is. , Processing time will be delayed. In particular, in order to obtain time-frequency characteristics, it is necessary to analyze depth information over a plurality of frames, and a delay of several frames or more is necessarily generated in the processing.

  Since a delay of several frames or more occurs, it cannot be applied to an application that performs encoding / decoding in real time.

  The present invention has been made in view of the above-described problems, and its object is to reduce the amount of information in the depth image by a simple process with little delay while maintaining the perspective relationship of the subject in the depth image. It is to realize an image conversion apparatus and the like that can perform the above.

  In order to solve the above-described problem, an image conversion apparatus according to the present invention is an image conversion apparatus that converts a depth image by converting a depth value, and the acquisition unit that acquires the depth image and the acquisition unit include A depth value distribution creating unit that creates a distribution of the number of occurrences of depth values in the acquired depth image, and a depth value distribution created by the depth value distribution creating unit is divided into a plurality of sections according to the continuity of the depth values in the distribution. A depth value distribution dividing means for dividing the depth value distribution, a representative value determining means for determining a representative value of each section of the depth value distribution divided by the depth value distribution dividing means, and a depth value distribution divided by the depth value distribution dividing means. Image conversion means for converting the depth image by converting the depth value included in each section into the representative value determined by the representative value determination means. It is set to.

  The image conversion device control method according to the present invention is a method of controlling an image conversion device that converts a depth image by converting a depth value, and includes an acquisition step of acquiring the depth image, and an acquisition step of the depth image. A depth value distribution creating step for creating a distribution of the number of occurrences of depth values in the acquired depth image, and the depth value distribution created in the depth value distribution creating step are divided into a plurality of sections according to the continuity of the depth values in the distribution. The depth value distribution dividing step, the representative value determining step for determining the representative value of each section of the depth value distribution divided by the depth value distribution dividing step, and the depth value distribution divided by the depth value distribution dividing step. The depth image is converted by converting the depth value included in each section into the representative value determined in the representative value determining step. It is characterized in that it comprises an image conversion step that, a.

  According to the above configuration or method, the distribution of the number of occurrences of depth values in the depth image is divided into a plurality of sections according to the continuity of the depth values in the distribution, and the depth values in the divided sections are converted into representative values. To convert the depth image.

  As a result, the depth value can be converted into a representative value while maintaining the shape feature of the distribution of the number of occurrences of the depth value in each depth image, so the information amount of the depth image while maintaining the perspective relationship of the depth image Can be reduced.

  In addition, the depth value can be reduced by converting the depth value without using the depth image before and after the depth image to be converted, so the depth value is converted using the depth image before and after. As in the case, it is possible to prevent the conversion process from being delayed.

  In the image conversion apparatus according to the present invention, when the depth value distribution dividing unit includes a discontinuous section in the depth value distribution created by the depth value distribution creating unit, the depth value distribution is divided by using the discontinuous section as a boundary. For the continuous section of the depth value distribution, the maximum value of the number of occurrences in the continuous section is detected, and when there are a plurality of maximum values, the minimum value of the number of appearances in the section between adjacent maximum values is The depth value distribution is divided with the depth value taking the minimum value as a boundary when the smaller maximum value across the interval is less than a value obtained by multiplying the smaller maximum value by a predetermined ratio of less than 1. Is preferred.

  According to said structure, a depth value distribution is divided | segmented on the boundary of the depth value of the appearance number less than the value which multiplied the predetermined ratio with respect to the maximum value pinched | interposed into the discontinuous section or maximum value.

  As a result, the depth value distribution is divided on the basis of the depth value where the number of appearances is small or the number of appearances is zero, so even if the depth values in each section after the division are converted into representative values, the shape of the depth value distribution is changed. It can be maintained accurately. Therefore, it is possible to convert the depth image while accurately maintaining the perspective relationship of the depth image.

  In the image conversion apparatus according to the present invention, the representative value determining unit corresponds to the maximum value when there is one maximum value of the number of appearances in the section of the depth value distribution divided by the depth value distribution dividing unit. It is preferable that the depth value is determined as a representative value of the section.

  In the divided section, when the maximum number of appearances is one, it can be said that the depth value corresponding to the maximum value represents the depth value of the section.

  Then, according to the above configuration, the representative value determining unit displays the representative value of the section of the depth value distribution divided by the depth value distribution dividing unit, and the depth corresponding to the maximum value when the maximum number of occurrences is one. Decide on a value.

  Thereby, the representative value for converting the depth value of the divided section can be accurately determined.

  In the image conversion apparatus according to the present invention, the representative value determining means has a constant number of occurrences or a plurality of maximum values of the number of appearances in the section of the depth value distribution divided by the depth value distribution dividing means. In this case, it is preferable that the depth value at the center of the section is determined as the representative value.

  In the divided section, when the number of appearances is constant or there are a plurality of maximum values of the number of appearances, it is considered that the number of appearances is not biased. It can be said that the depth value is expressed.

  According to the above configuration, the representative value determining unit has a constant number of appearances or a plurality of maximum values of the number of appearances of the representative values of the section of the depth value distribution divided by the depth value distribution dividing unit. If so, the depth value at the center of the section is determined.

  Thereby, the representative value for converting the depth value of the divided section can be accurately determined.

  In the image conversion apparatus according to the present invention, the acquisition unit acquires a plurality of depth images having different viewpoints, and identifies an occlusion region that is a region where the subject overlaps in the plurality of depth images acquired by the acquisition unit. Occlusion area specifying means for performing the processing, and the image converting means may not convert the depth values of the areas specified by the occlusion area specifying means as occlusion areas.

  The occlusion area is an area where subjects overlap in a plurality of depth images with different viewpoints, and the perspective relationship is greatly different in a narrow area. Therefore, if the depth value is converted into the representative value in this region, there is a possibility that the perspective relationship that is greatly different is lost and the depth value becomes flat.

  So, according to said structure, an image conversion means does not convert each depth value about the area | region which the occlusion area specification means specified as the occlusion area.

  Therefore, for the occlusion area, the depth image can be converted while maintaining the perspective relationship as it is. That is, the depth image can be converted while maintaining the quality of the depth image.

  The image conversion apparatus according to the present invention includes image dividing means for dividing the plurality of depth images obtained by the obtaining means into a plurality of divided images at the same division position, and the depth value distribution creating means includes the image dividing means. The means creates a distribution of the number of appearances of the depth value of each divided image divided by the means, and further determines a divided image of another depth image corresponding to the divided image of a certain depth image among the depth images. Corresponding occlusion area specifying means, and the occlusion area specifying means includes 1 for all sections of the depth value distribution of a certain divided image between the corresponding divided images determined by the corresponding divided image determining means. While calculating the difference between the feature value of the target section and the feature values of all sections of the corresponding depth distribution of the divided images, each section is set as the target section. When the difference is greater than a predetermined value, an area composed of pixels having a depth value belonging to the section of interest, or may be specified as occlusion region.

  Since the occlusion area is an area having different perspective relationships among a plurality of depth images, the depth value distribution of the occlusion area in a certain depth image and the depth value distribution of other depth images corresponding to this area are also different.

  Therefore, in the occlusion area, the feature amounts in the section of the depth value distribution between the corresponding divided images are greatly different. In other words, if it is not an occlusion region, the feature amount in the section of the depth value distribution of the corresponding divided image does not change so much.

  Therefore, whether or not the region is an occlusion region can be identified by looking at the difference between the feature amount of the section of the depth value distribution in a certain divided image and the feature amount of the section of the depth value distribution in the corresponding divided image.

  Therefore, according to the configuration described above, the occlusion area specifying unit sets the attention section of the attention section while setting each section of the depth value distribution of a certain divided image as the attention section one by one. The difference between the feature amount and the feature amount of all the sections of the depth value distribution of the corresponding divided image is calculated, and when all the calculated differences are larger than a predetermined value, the pixel having the depth value belonging to the section of interest is calculated. This area is specified as the occlusion area, so that the occlusion area can be specified accurately.

  In the image conversion apparatus according to the present invention, the representative value determining means determines a representative value of each section of the depth value distribution of each depth image divided by the image dividing means, and each division of each depth image. For all images, all the sections obtained by dividing the area occupied by the pixels having the depth values belonging to one section divided by the depth value distribution dividing means in the divided image by the depth value distribution dividing means. The occlusion area specifying means may calculate at least one of the representative value and the occupied area as the feature amount.

  The representative value and the occupied area of each section of the depth value distribution represent the characteristics of the depth value distribution of the section. Therefore, the occlusion area specifying unit can accurately specify the occlusion area by using the representative value and the occupied area as the feature amount.

  The image conversion apparatus may be realized by a computer. In this case, an image conversion apparatus control program for causing the image conversion apparatus to be realized by a computer by causing the computer to operate as the above-described means, and this A computer-readable recording medium on which is recorded also falls within the scope of the present invention.

  As described above, the image conversion apparatus according to the present invention includes an acquisition unit that acquires a depth image, a depth value distribution generation unit that generates a distribution of the number of appearances of depth values in the depth image acquired by the acquisition unit, and the above Depth value distribution dividing means for dividing the depth value distribution created by the depth value distribution creating means into a plurality of sections according to the shape of the distribution, and representative values of each section of the depth value distribution divided by the depth value distribution dividing means Representative value determining means for determining the depth image, and by converting the depth value included in each section of the depth value distribution divided by the depth value distribution dividing means into the representative value determined by the representative value determining means, And an image conversion means for conversion.

  The image conversion apparatus control method according to the present invention includes an acquisition step of acquiring a depth image, a depth value distribution generation step of generating a distribution of the number of appearances of depth values in the depth image acquired in the acquisition step, and A depth value distribution dividing step for dividing the depth value distribution created in the depth value distribution creating step into a plurality of sections according to the shape of the distribution, and representative values of each section of the depth value distribution divided in the depth value distribution dividing step The depth image is converted by converting the depth value included in each section of the depth value distribution divided in the depth value distribution dividing step into the representative value determined in the representative value determining step. An image conversion step of converting.

  As a result, the depth value can be converted into the representative value while maintaining the shape feature of the distribution of the number of occurrences of the depth value in the depth image, so the information amount of the depth image can be reduced while maintaining the perspective relationship of the depth image. There is an effect that it can be reduced.

  In addition, since the depth image can be converted by reducing the depth information without using the front and back depth images, the conversion processing can be performed as in the case of converting the depth value using the front and back depth images. There is an effect that the delay can be prevented.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 illustrates an embodiment of the present invention and is a block diagram illustrating a main configuration of an image encoding device. It is a flowchart which shows the flow of the process in the said image coding apparatus. It is a flowchart which shows the flow of the matching process in the said image coding apparatus. It is a flowchart which shows the flow of the pixel classification | category process in the said image coding apparatus. It is a figure for demonstrating this Embodiment, It is a figure which shows the viewpoint image for left eyes, the viewpoint image for right eyes, and the depth image for right eyes and the depth image for left eyes corresponding to these, (A) is a diagram showing a left eye viewpoint image, (b) is a diagram showing a left eye depth image corresponding to the left eye viewpoint image, and (c) is a right eye viewpoint image. (D) of the figure which shows the right-eye viewpoint image corresponding to the right-eye viewpoint image. It is a figure for demonstrating this Embodiment, and is a figure which shows depth value distribution in the block area | region shown in FIG. (A), (b) of the same figure is a figure (histogram) which shows depth value distribution in a block area | region. It is a figure which shows the example of depth value distribution used as the object which the pixel classification part in the said image coding apparatus performs a classification process. It is a flowchart which shows the flow of the occlusion area | region identification process in the said image coding apparatus. It is explanatory drawing which shows the outline | summary of a stereo image display. It is explanatory drawing which shows the outline | summary of an arbitrary viewpoint image display. It is a figure for demonstrating the principle which leads to the improvement of a three-dimensional display by the production | generation technique of arbitrary viewpoint images.

  An embodiment of the present invention will be described below with reference to FIGS.

(Configuration of image encoding device)
First, an image encoding device (image conversion device) 10 according to the present embodiment will be described. FIG. 1 is a block diagram illustrating a main configuration of the image encoding device 10. The image encoding device 10 acquires depth images corresponding to a plurality of viewpoint images (for example, right-eye images and left-eye images), performs an encoding process on these images, reduces the amount of information, and transmits the images. It is. The right-eye image is an image assumed to be viewed by the observer with the right eye, and the left-eye image is an image assumed to be viewed by the observer with the left eye. By viewing the right-eye image and the left-eye image at the same time, the observer can see the subject displayed in the image in three dimensions. In this embodiment, the case where there are two depth images will be described. However, the present invention is not limited to this, and there may be three or more depth images.

  Note that the depth image is an image in which each pixel is expressed only by the luminance indicating the depth value, which is the distance between the camera and the subject. Usually, a subject closer to the camera is expressed with higher luminance, and the luminance is decreased as the distance from the camera increases.

  As a method for creating a depth image, for example, a method of creating a depth image by irradiating infrared rays or ultrasonic waves from a camera and measuring the distance between the camera and a subject using the reflection to obtain a depth value, Images are taken with multiple cameras, and the arrangement and context of each area in the viewpoint image are estimated from the information about the camera arrangement and shooting conditions at the time of shooting, and the correspondence between the viewpoint images taken with multiple cameras. A method of creating a depth image by obtaining a depth value can be given.

  As shown in FIG. 1, the image encoding device 10 includes a dividing unit (acquiring unit, image dividing unit) 11, a matching unit (corresponding divided image determining unit) 12, and a depth value distribution creating unit (depth value distribution creating unit) 13. , Pixel classification unit (depth value distribution dividing unit, representative value determining unit, occupied area calculating unit) 14, occlusion region specifying unit (occlusion region specifying unit) 15, depth value converting unit (image converting unit) 16, and encoding unit 17 , And a multiplexing unit 18.

  The dividing unit 11 acquires a right-eye depth image and a left-eye depth image corresponding to the right-eye image and the left-eye image, and each of the blocks (divided images) of a predetermined size (for example, 16 × 16 pixels). ). Then, the respective depth images (right-eye image and left-eye image) divided into a plurality of blocks are transmitted to the matching unit 12 and the depth value distribution creating unit 13. The depth image is composed of a plurality of pieces of pixel information, and the pixel information is composed of coordinate information indicating the position in the image and luminance information. Note that the depth image may be a moving image composed of a plurality of frames, or may be a still image. In the present embodiment, the luminance information is described as being expressed by 8 bits, but the present invention is not limited to this.

  The matching unit 12 performs block matching with the other depth image for each of the right-eye depth image and the left-eye depth image obtained by the division unit 11 and divided into a plurality of blocks, and has the highest degree of matching. The block is determined. Then, the blocks determined to have the highest degree of coincidence are associated with each other and transmitted to the occlusion area specifying unit 15.

  A method for determining the degree of coincidence between blocks will be described more specifically. The matching unit 12 calculates, for each block of the left-eye depth image, a cumulative sum AE of absolute difference values for a block within a predetermined range (for example, ± 16 pixels) centering on the same position block of the right-eye depth image. . Then, the block having the smallest cumulative sum AE of difference absolute values is determined as a block having a high matching degree.

  The depth value distribution creating unit 13 counts the number of appearances of the luminance information (depth value) in each block for the right-eye depth image and the left-eye depth image divided into the plurality of blocks acquired from the dividing unit 11, A depth value distribution indicating the relationship between the depth value and the number of appearances (number of pixels) is generated. Then, the generated depth value distribution in each block is transmitted to the pixel classification unit 14 in association with each block.

  The pixel classification unit 14 uses the depth value distribution in each block of the right-eye depth image and the left-eye depth image acquired from the depth value distribution creation unit 13 to convert the pixels of each block into one or more pixel groups. Classify into: Then, a representative value and an occupied area (feature amount) of each pixel group are calculated. Then, the representative value and the occupied area of each pixel group are transmitted to the occlusion area specifying unit 15 in association with each block. A pixel group classification method, a representative value, and an occupied area calculation method will be described later.

  The occlusion area specifying unit 15 correlates each block between the right-eye depth image and the left-eye depth image acquired from the matching unit 12 and the right-eye depth image and left-eye acquired from the pixel classification unit 14. The region where the occlusion occurs is specified using the representative value and the occupied area of the pixel group in each block of the depth image for use. Occlusion means that a subject that exists at a short distance from the camera hides part or all of a subject that is located farther than this. In the present embodiment, an occlusion area is defined as an area where the displayed range of the subject differs between the right-eye image and the left-eye image due to occlusion. And the occlusion area | region specific | specification part 15 will transmit the position in the depth image of the pixel group in which the occlusion has generate | occur | produced in correlation with each block to the depth value conversion part 16, if the area | region where the occlusion has generate | occur | produced is specified. . A method for identifying the region where the occlusion has occurred will be described later.

  The depth value conversion unit 16 uses the representative value and the occupied area of the pixel group for each block acquired from the pixel classification unit 14 and the position where the occlusion acquired from the occlusion region specifying unit 15 is generated. A process of converting the depth value to reduce the resolution of the block is performed and transmitted to the encoding unit 17. Details of the conversion process will be described later.

  The encoding unit 17 compresses and encodes the depth image whose depth value has been converted by the depth value conversion unit 16 based on a predetermined encoding method. Examples of the encoding method include JPEG (Joint Photographic Experts Group) and JPEG2000 if the image to be encoded is a still image. Also, if the image to be encoded is a moving image, MPEG (Moving Picture Experts Group) -2, MPEG-4AVC (Advanced Video Coding) / H. H.264 or the like can be cited. Then, the compression-encoded depth image is transmitted to the multiplexing unit 18.

  The multiplexing unit 18 multiplexes the encoded data of each depth image encoded by the encoding unit 17 in accordance with a predetermined format, and transmits the multiplexed data to a recording device or an external communication unit (not shown). An example of the predetermined format for multiplexing is an MVC (Multi-view Video Coding) format.

(Processing flow in image encoding device)
Next, the flow of processing in the image encoding device 10 will be described with reference to FIG. FIG. 2 is a flowchart showing the flow of processing of the image encoding device 10.

  As illustrated in FIG. 2, when the image encoding device 10 acquires depth images (a left-eye depth image and a right-eye depth image) (S1, acquisition step), the dividing unit 11 converts these depth images, respectively. Divide into predetermined sizes (S2). Next, the matching unit 12 performs block matching with the right-eye depth image for each block of the left-eye depth image divided by the dividing unit 11, and for each block of the right-eye depth image, the left-eye depth. Perform block matching with the image. Then, a matching pair is determined for each block of the right-eye depth image and the left-eye depth image (S3). Details of the process of determining a matching pair will be described later.

  Then, the depth value distribution creating unit 13 creates a depth value distribution for each block divided by the dividing unit 11 (S4, depth value distribution creating step), and the pixel classifying unit 14 creates each block divided by the dividing unit 11. Then, the pixel groups are classified (depth value distribution division step), representative values in each pixel group are determined (representative value determination step), and the occupied area is calculated (S5). Details of the processing for classifying the pixel groups will be described later.

  Next, the occlusion region specifying unit 15 determines the presence / absence of occlusion using the matching pair determined by the matching unit 12 and the representative value and the occupied area of each pixel group calculated by the pixel classification unit 14 (S6). Details of the occlusion area specifying process will be described later.

  After that, the depth value conversion unit 16 converts the depth value according to the presence or absence of the occlusion specified by the occlusion region specifying unit 15 (S7, image conversion step), and the encoding unit 17 uses the depth value conversion unit 16 to change the depth. The depth image whose value has been converted is encoded (S8), and the multiplexing unit 18 multiplexes the encoded depth images (the left-eye depth image and the right-eye depth image) (S8) and records them. It is transmitted to a device or an external communication device. Details of the depth value conversion processing will be described later. Above, the process in the image coding apparatus 10 is complete | finished.

(Processing in the matching unit)
Next, the flow of processing in the matching unit 12 will be described with reference to FIG. FIG. 3 is a flowchart showing the flow of the matching process.

  First, for each block in the left-eye depth image, the matching unit 12 sets the same position block in the right-eye depth image and a block existing within a predetermined range (± 16 pixels) from the block as reference blocks ( S31). Next, the matching unit 12 calculates an accumulated sum AE of absolute differences between the block of the left-eye depth image and the reference block in the set right-eye depth image (S32). When the calculation of the accumulated absolute difference value AE is completed for all the set reference blocks (YES in S33), a block pair having the smallest accumulated absolute value AE is determined as a matching block ( S34). When matching pairs are determined for all the blocks of the left-eye depth image (YES in S35), the matching process is terminated.

  The same processing is performed for the right-eye depth image.

(Processing in the pixel classification unit)
Next, processing in the pixel classification unit 14 will be described with reference to FIGS. FIG. 4 is a flowchart showing the flow of pixel classification processing. In the following description, processing for one block in the depth image will be described. However, the pixel classification unit 14 performs the following processing on all blocks of the depth image (the left-eye depth image and the right-eye depth image). Is what you do.

  First, the pixel classification unit 14 determines whether or not there is a discontinuous section in the depth value distribution for each block created by the depth value distribution creation unit 13 (S51). If there is a discontinuous section (YES in S1), it is classified into two pixel groups with the discontinuous section as a boundary (S52). A pixel group refers to a group of pixel information (information having coordinates and depth values) having depth values classified into the group.

  On the other hand, when there is no discontinuous section in the depth value distribution (NO in S51), or after classifying the pixel group in step S52, the pixel classification unit 14 creates each block created by the depth value distribution creation unit 13. The maximum value of the number of appearances in the continuous section of each depth value distribution is extracted (S53). If the number of extracted maximum values is one (NO in S54), the continuous section of the depth value distribution is set as one group (S55). That is, pixel information having each depth value in the continuous section of the depth value distribution is classified into one pixel group.

  On the other hand, if there are a plurality of extracted maximum values (YES in S54), the pixel classification unit 14 has a predetermined ratio (for example, 20) with respect to both maximum values in a distribution sandwiched between two adjacent maximum values. It is determined whether there are depth values with the number of appearances less than (%) (S55). Then, if there are depth values with the number of appearances less than a predetermined ratio (for example, 20%) with respect to both local maximum values (YES in S55), two pixels with the depth value with the smallest number of appearances as the boundary among the depth values. Sort into groups (S56). That is, pixel information having depth values in a range including the respective local maximum values with the depth value having the smallest occurrence number as a boundary is classified into two groups. On the other hand, if there is no depth value having an appearance number less than a predetermined ratio with respect to both maximum values (NO in S55), the depth values in the range including the maximum values are set to one group (S57).

  Then, the pixel classification unit 14 determines a representative value for each classified pixel group, and calculates an occupation area (S58). The representative value of the pixel group is determined as follows. When only one depth value having the maximum number of appearances exists in the pixel group, the depth value is determined as a representative value. When there are a plurality of depth values having the maximum number of appearances, the central value of the depth values in the pixel group is determined as a representative value.

  Further, the occupation area of the pixel group is calculated by integrating the depth value distribution in the section of the pixel group. Thus, the process in the pixel classification unit 14 ends.

  Next, an example of processing in the pixel classification unit 14 will be described with reference to FIGS. 5 to 7 are diagrams for explaining an example of processing in the pixel classification unit 14.

  FIG. 5 shows a left-eye viewpoint image and a right-eye viewpoint image, and a right-eye depth image and a left-eye depth image corresponding thereto. 5A shows the left eye viewpoint image 210, and FIG. 5B shows the left eye depth image 211 corresponding to the left eye viewpoint image 210. FIG. 5C shows a right-eye viewpoint image 220, and FIG. 5D shows a right-eye depth image 221 corresponding to the right-eye viewpoint image 220.

  FIGS. 5 (a) and 5 (b) are images of a space including two subjects 201 and 202, respectively. However, since the parallax is generated due to the difference in viewpoint, the subject 202 in the right-eye viewpoint image 220 is shown in FIGS. Although all are displayed without being hidden by the subject 201, a part of the subject 202 is not hidden by the subject 201 in the left eye viewpoint image 210. This is a state where the above-described occlusion occurs.

  Here, attention is focused on the block area 203 a of the left-eye depth image 211 and the block area 203 b of the right-eye depth image 221. The block region 203a and the block region 203b are a pair of blocks (matching pair) that is divided by the dividing unit 11 and determined to have the highest degree of matching in the matching unit 12.

  As shown in FIGS. 5B and 5D, the block area 203a of the left-eye depth image 211 includes the subject 202, but the block area 203b of the right-eye depth image 221 includes the subject. 202 is not included. That is, the depth value in the block area 203b of the right-eye depth image 221 is composed of the subject 201 and the background portion.

  FIG. 6 is a diagram showing the depth value distribution in the block area 203a and the block area 203b shown in FIG. In the depth value distribution shown in FIG. 6, the horizontal axis indicates the depth value, and the vertical axis indicates the number of appearances. FIG. 6A is a diagram (histogram) showing the depth value distribution in the block region 203a, and FIG. 6B is a diagram showing the depth value distribution in the block region 203b.

  When the depth value distribution in the block region 203a of the depth image 211 for the left eye in FIG. 5 is a depth value distribution as shown in FIG. 6A, the pixel classification unit 14 first determines that the depth value distribution is invalid. It is determined whether or not there is a continuous section. In the depth value distribution shown in FIG. 6A, a discontinuous section 603 exists. Therefore, the pixel classification unit 14 classifies the depth value distribution into the pixel group 301a and the pixel group 302 with the discontinuous section 603 as a boundary. Then, it is determined whether or not there are a plurality of maximum values of the number of appearances in the depth value distribution classified into each of the pixel group 301a and the pixel group 302. Since the depth value distribution classified into the pixel group 301a has only one maximum value (maximum value 601), the depth value distribution classified into the pixel group 301a is not further classified into a plurality. Similarly, since the depth value distribution classified into the pixel group 302 has only one maximum value (maximum value 602), the depth value distribution classified into the pixel group 302 is not further classified into a plurality of values. .

  As described above, the depth value distribution shown in FIG. 6A is classified into the pixel group 301 a including the local maximum value 601 and the pixel group 302 including the local maximum value 602.

  Then, the pixel classification unit 14 determines the representative value of the pixel group 301 a as a depth value corresponding to the maximum value 601, and determines the representative value of the pixel group 302 as a depth value corresponding to the maximum value 602. Further, by integrating the sections of the pixel group 301a and the pixel group 302 in the depth value distribution, the occupation area of each pixel group is calculated.

  Similarly, when the depth value distribution in the block region 203b of the right-eye depth image 221 in FIG. 5 is a depth value distribution as shown in FIG. 6B, the pixel classification unit 14 first determines the depth value. It is determined whether or not there is a discontinuous section in the distribution. In the depth value distribution shown in FIG. 6B, a discontinuous section 606 exists. Therefore, the pixel classification unit 14 classifies the depth value distribution into the pixel group 301b and the pixel group 303 with the discontinuous section 606 as a boundary. Then, it is determined whether or not there are a plurality of maximum values of the number of appearances in the depth value distribution classified into each of the pixel group 301b and the pixel group 303. Since the depth value distribution classified into the pixel group 301a has only one maximum value (maximum value 604), the depth value distribution classified into the pixel group 301b is not further classified into a plurality. In addition, since there is no maximum value in the depth value distribution classified into the pixel group 303, the depth value distribution classified into the pixel group 303 is not classified into a plurality of depth values.

  As described above, the depth value distribution shown in FIG. 6A is classified into the pixel group 301 b and the pixel group 303 including the local maximum value 604.

  Then, the pixel classification unit 14 determines the representative value of the pixel group 301b as a depth value corresponding to the local maximum value 604, and sets the representative value of the pixel group 303 as an intermediate value 605 that is an intermediate value of the depth values in the pixel group 303. To decide. Further, by integrating the sections of the pixel group 301b and the pixel group 303 in the depth value distribution, the occupation area of each pixel group is calculated.

  Note that if there is no discontinuous section in the depth value distribution, the pixel classification unit 14 performs classification as follows. This will be described with reference to FIG. FIG. 7 is an example showing a depth value distribution in which there is no discontinuous section. When the depth value distribution as shown in FIG. 7 exists, the pixel classification unit 14 first extracts the local maximum value of the depth value distribution. Here, the maximum value 401 and the maximum value 402 are extracted. Then, since a plurality of local maximum values are extracted, it is determined whether or not there are depth values having the number of appearances whose ratio to the local maximum values is less than 20% in a section between the local maximum values. In the example shown in FIG. 7, there is a minimum value 403. Therefore, the pixel classifying unit 14 classifies the pixel group 305 including the maximum value 401 and the pixel group 304 including the maximum value 402 with the minimum value 403 as a boundary. Then, the representative value and the occupied area of each pixel group are calculated by the method described above.

(Processing in the occlusion area identification part)
Next, processing in the occlusion area specifying unit 15 will be described with reference to FIGS. FIG. 8 is a flowchart showing the flow of processing in the occlusion area specifying unit 15.

  The occlusion area specifying unit 15 first determines whether or not the matching pair block determined by the matching unit 12 has been classified into a plurality of pixel groups by the pixel classifying unit 14 (S61). If there is one pixel group in the block (NO in S61), it is determined that no occlusion has occurred (S64). This is because a single pixel group means that there is only one subject area in the block.

  On the other hand, if the pixel group is classified into a plurality of pixel groups (YES in S61), the representative value of the pixel group in each block and the occupied area are compared to determine whether there is a matching pixel group. Whether or not the pixel groups match is determined, for example, by determining whether or not there is a pixel group that matches a block to be compared for a certain pixel group (a region made up of pixels having a depth value included in the target section). In this case, the difference between the representative value and the occupied area is calculated for all pixel groups included in the comparison target block, and there are pixel groups whose difference values are less than 10% of the original value. It is determined whether or not. If there are pixel groups having a difference value of less than 10%, it is determined that the pixel groups match.

  Then, it is determined whether there is a pixel group that does not match between the blocks of the matching pair (S62). If there is a pixel group that does not match (YES in S62), the region of the pixel group is identified as an occlusion region ( S63). Thus, the process in the occlusion area specifying unit 15 is completed.

  Next, an example of processing in the occlusion area specifying unit 15 will be specifically described with reference to FIGS. As described above, the block area 203a in FIG. 5B and the block area 203b in FIG. 5D are matching pairs, and occlusion occurs. In this case, for these blocks, it is determined whether or not the pixel groups match, and whether or not occlusion has occurred is determined as follows.

  As shown in FIGS. 6A and 6B, the block region 203a is classified into a pixel group 301a and a pixel group 302, and the block region 203b is classified into a pixel group 301b and a pixel group 303. Therefore, both the block area 203a and the block area 203b are classified into a plurality of pixel groups.

  Then, when the representative values and occupied areas of the pixel group 301a, the pixel group 302 and the pixel group 301b, and the pixel group 303 are compared, the pixel group 301a and the pixel group 301b match, and the pixel group 302 and the pixel group 303 It can be seen that does not match. Therefore, the occlusion area specifying unit 15 determines that occlusion has occurred in the area of the pixel group 302 in the block area 203a and the area of the pixel group 303 in the block area 203b. Then, the occlusion area specifying unit 15 specifies the area of the pixel group 302 and the area of the pixel group 303 as an area where occlusion occurs, and transmits the area to the depth value conversion unit 16.

(Processing in the depth value converter)
Next, processing in the depth value conversion unit 16 will be described. The depth value conversion unit 16 converts the depth value as follows using the representative value and the occupied area of the pixel group in each block, and the occlusion generation region.
(1) When there are a plurality of pixel groups in the block and no occlusion occurs, the depth value included in each pixel group is replaced with a representative value of each group.
(2) When there are a plurality of pixel groups in a block and occlusion occurs, the depth value included in the pixel group in which occlusion occurs is maintained as it is, and the depth included in other pixel groups. The value is replaced with a representative value for each group.
(3) When there is one pixel value group in the block, all the depth values in the block are replaced with representative values of the pixel group.

  By performing the conversion process as described above, with respect to an image including a plurality of subjects overlapping in the depth direction, the depth value is made uniform in an area where no occlusion has occurred, whereby the data amount and the redundancy of the data are obtained. Is reduced. Therefore, when the depth image whose depth value has been converted by the depth value conversion unit 16 is compressed and encoded by the encoding unit 17, an effect of improving the compression rate can be achieved.

  Further, when the process (3) is performed, the depth value of the block is single. This means that the spatial resolution of the depth image is reduced. For example, when the block size is 16 × 16 = 256 pixels, by performing the process (3), all 256 pixels can be expressed with the same depth value. Therefore, the depth value of 256 pixels can be expressed by the depth value of one pixel, and the data amount can be reduced to 1/256.

  In the above-described embodiment, the depth value conversion unit 16 performs depth value conversion using a criterion of whether or not the region is an occlusion region. However, the criteria for converting the depth value are not limited to this, and other criteria may be used. For example, the depth value may be converted based on variations in the depth value distribution. That is, the depth value variance σ in the pixel group is obtained. When the variance σ is larger than the predetermined value R, the depth value is not converted (that is, the depth value is maintained as it is), and when the depth value is equal to or smaller than the predetermined value R, Depth values included in the pixel group are converted into representative values.

  Thereby, when the subject has a large change in the depth direction (σ> R), the conversion is not performed, and when the subject has a small change in the depth direction (σ ≦ R), the conversion is performed (the depth value is a representative value). Therefore, the amount of information can be reduced without greatly reducing the reproducibility of the perspective of the subject.

  By such a method, depth values can be converted while maintaining the perspective relationship of a plurality of subjects overlapping in the depth direction, so that the amount of information can be reduced while suppressing distortion of perspective in the image. it can.

  As described above, according to the present embodiment, the depth image information amount is reduced by reducing the information amount of the depth value and encoding the depth image while maintaining the context of the plurality of subjects. be able to. In addition, when the information amount of the depth image is reduced, processing can be performed without using a plurality of image frames in the time direction, so that processing delay can be suppressed and encoding can be performed in real time. .

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope shown in the claims. That is, embodiments obtained by combining technical means appropriately modified within the scope of the claims are also included in the technical scope of the present invention.

  Finally, each block of the image encoding device 10, in particular, a dividing unit 11, a matching unit 12, a depth value distribution creating unit 13, a pixel classifying unit 14, an occlusion area specifying unit 15, a depth value converting unit 16, an encoding unit 17, The multiplexing unit 18 may be realized in hardware by a logic circuit formed on an integrated circuit (IC chip), or may be realized in software using a CPU (central processing unit).

  In the latter case, the image encoding device 10 includes a CPU that executes instructions of a control program that implements each function, a ROM (read only memory) that stores the program, a RAM (random access memory) that expands the program, A storage device (recording medium) such as a memory for storing programs and various data is provided. An object of the present invention is a recording medium in which a program code (execution format program, intermediate code program, source program) of a control program of the image encoding device 10 which is software for realizing the functions described above is recorded so as to be readable by a computer. Can also be achieved by reading the program code recorded on the recording medium and executing it by the computer (or CPU or MPU (microprocessor unit)).

  Examples of the recording medium include tapes such as magnetic tape and cassette tape, magnetic disks such as floppy (registered trademark) disks / hard disks, CD-ROM (compact disc read-only memory) / MO (magneto-optical) / Disks including optical discs such as MD (Mini Disc) / DVD (digital versatile disk) / CD-R (CD Recordable), cards such as IC cards (including memory cards) / optical cards, mask ROM / EPROM (erasable) Uses semiconductor memory such as programmable read-only memory (EEPROM) / EEPROM (electrically erasable and programmable read-only memory) / flash ROM, or logic circuits such as PLD (Programmable logic device) and FPGA (Field Programmable Gate Array) be able to.

  Further, the image encoding device 10 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited as long as it can transmit the program code. For example, the Internet, intranet, extranet, LAN (local area network), ISDN (integrated services digital network), VAN (value-added network), CATV (community antenna television) communication network, virtual private network (virtual private network), A telephone line network, a mobile communication network, a satellite communication network, etc. can be used. The transmission medium constituting the communication network may be any medium that can transmit the program code, and is not limited to a specific configuration or type. For example, infra-red data association (IrDA) or remote control such as IEEE (institute of electrical and electronic engineers) 1394, USB, power line carrier, cable TV line, telephone line, ADSL (asynchronous digital subscriber loop) line, etc. , Bluetooth (registered trademark), IEEE 802.11 wireless, HDR (high data rate), NFC (Near Field Communication), DLNA (Digital Living Network Alliance), mobile phone network, satellite line, terrestrial digital network, etc. Is possible.

  Since the depth value corresponding to the viewpoint image can be compressed by a simple process with little delay, it is suitable for a device that processes an image using the depth value, for example, a device that creates a stereoscopic image or an image at an arbitrary viewpoint.

10 Image encoding device (image conversion device)
11 Dividing unit (acquiring means, image dividing means)
12 Matching part (corresponding divided image determining means)
13 Depth value distribution creation unit (depth value distribution creation means)
14 Pixel classification unit (depth value distribution dividing means, representative value determining means, occupied area calculating means)
15 Occlusion area specifying part (occlusion area specifying means)
16 Depth value conversion unit (image conversion means)
17 Encoding unit 18 Multiplexing unit

Claims (10)

An image conversion device for converting a depth image by converting a depth value,
Obtaining means for obtaining the depth image;
A depth value distribution creating means for creating a distribution of the number of occurrences of depth values in the depth image obtained by the obtaining means;
Depth value distribution dividing means for dividing the depth value distribution created by the depth value distribution creating means into a plurality of sections according to the continuity of the depth values in the distribution;
Representative value determining means for determining a representative value of each section of the depth value distribution divided by the depth value distribution dividing means;
Image conversion means for converting the depth image by converting the depth value included in each section of the depth value distribution divided by the depth value distribution dividing means into the representative value determined by the representative value determining means. An image conversion apparatus characterized by that. The depth value distribution dividing means includes:
When there is a discontinuous section in the depth value distribution created by the depth value distribution creating means, the depth value distribution is divided with the discontinuous section as a boundary,
For the continuous section of the depth value distribution, the maximum value of the number of occurrences in the continuous section is detected, and when there are a plurality of maximum values, the minimum value of the number of appearances in the section between adjacent maximum values is The depth value distribution is divided on the basis of a depth value that takes the minimum value when the smaller maximum value across the interval is less than a value obtained by multiplying the smaller maximum value by a predetermined ratio of less than 1. Item 2. The image conversion apparatus according to Item 1.   In the depth value distribution section divided by the depth value distribution dividing means, when there is one maximum value of the number of appearances, the representative value determining means sets the depth value corresponding to the maximum value as the representative value of the section. The image conversion apparatus according to claim 1, wherein the image conversion apparatus is determined.   The representative value determining means, when the number of occurrences is constant in the section of the depth value distribution divided by the depth value distribution dividing means, or when there are a plurality of maximum values of the number of appearances, the depth value at the center of the section The image conversion apparatus according to claim 1, wherein a representative value is determined. The acquisition unit acquires a plurality of depth images having different viewpoints,
In the plurality of depth images acquired by the acquisition unit, the device includes an occlusion region specifying unit that specifies an occlusion region that is a region where the subject overlaps differently.
5. The image conversion apparatus according to claim 1, wherein the image conversion unit does not convert a depth value for an area specified by the occlusion area specifying unit as an occlusion area. 6. An image dividing unit that divides the plurality of depth images acquired by the acquiring unit into a plurality of divided images at the same division position;
The depth value distribution creating means creates a distribution of the number of appearance of the depth value of each divided image divided by the image dividing means,
further,
Corresponding divided image determining means for determining a divided image of another depth image corresponding to a divided image of a certain depth image among the depth images,
The occlusion area specifying means is:
Between the corresponding divided images determined by the corresponding divided image determination means,
For all the sections of the depth value distribution of a certain divided image, the difference between the feature amount of the section of interest and the feature amount of all sections of the corresponding depth value distribution of the divided image is set as the section of interest one by one. Calculate
6. The image conversion apparatus according to claim 5, wherein when all the calculated differences are larger than a predetermined value, an area composed of pixels having a depth value belonging to the section of interest is identified as an occlusion area. The representative value determining means determines a representative value of each section of the depth value distribution of each depth image divided by the image dividing means,
For each divided image of each depth image, an area occupied by an area formed by pixels having depth values belonging to one section divided by the depth value distribution dividing unit is an area occupied in the divided image by the depth value distribution dividing unit. Occupied area calculation means for calculating all sections divided by
The image conversion apparatus according to claim 6, wherein the occlusion area specifying unit uses at least one of the representative value and the occupied area as the feature amount.   An image conversion apparatus control program for operating the image conversion apparatus according to claim 1, wherein the image conversion apparatus control program causes a computer to function as each of the above means.   A computer-readable recording medium on which the image conversion apparatus control program according to claim 8 is recorded. A method for controlling an image conversion apparatus that converts a depth image by converting a depth value,
An acquisition step of acquiring the depth image;
A depth value distribution creating step for creating a distribution of the number of occurrences of depth values in the depth image obtained in the obtaining step;
A depth value distribution dividing step of dividing the depth value distribution created in the depth value distribution creating step into a plurality of sections according to the continuity of the depth values in the distribution;
A representative value determining step for determining a representative value of each section of the depth value distribution divided in the depth value distribution dividing step;
An image conversion step of converting the depth image by converting the depth value included in each section of the depth value distribution divided in the depth value distribution division step into the representative value determined in the representative value determination step. A control method for an image conversion apparatus.