JP2014099716A - Image encoder and control method of the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To highly precisely generate a predicted image and to generate encoded data with a high compression ratio even if an occlusion section and a non-occlusion section coexist in an encoded block.SOLUTION: A mask image generation section 101 generates mask data that shows to which of an occlusion section and a non-occlusion section each pixel of image data of a target view point belongs by using distance image data. A parallax vector retrieval section 103 refers to the mask data and determines whether a noticed block is a mixed block where the occlusion section and the non-occlusion section coexist or not. When the block is determined to be the mixed block, noticed block image data is separated into the occlusion region and the non-occlusion region by referring to identification data, and obtains a predicted vector corresponding to each region. A parallax compensation prediction section 104 extracts predicted image data from the individual vectors and integrates data to generate the predicted image data for predicting and encoding the noticed block.

Description

  The present invention relates to an image encoding technique.

  2. Description of the Related Art Conventionally, a method is known in which video is captured from a plurality of viewpoints, and video representation such as stereo stereoscopic viewing and free viewpoint video synthesis is performed using the captured multi-viewpoint videos. In order to realize such video expression, multi-viewpoint video storage technology is indispensable. Simply, as with a normal digital camera, a single-viewpoint image compression encoding technique is used as many as the number of viewpoints. However, the data amount in this case is proportional to the number of viewpoints.

  In consideration of such a situation, an encoding method that reduces the amount of data by using the correlation of multi-view video has been studied. A typical example is H.264. H.264 / MPEG-4 AVC multiview video coding (hereinafter referred to as MVC) is known. In MVC, when the target viewpoint video is predicted by parallax compensation prediction using the reference viewpoint video, and the difference between the target viewpoint video and the predicted image is encoded, thereby encoding the target viewpoint video itself. Reduce the amount of data compared to. Here, the parallax compensation prediction is obtained by applying a motion compensation prediction technique known as a compression technique using a correlation between frames along a time axis of a general moving image to videos between different viewpoints.

In addition, 3D Video Coding (hereinafter 3DV) is being formulated. MVC encodes multi-view video, whereas 3DV particularly encodes distance images for each viewpoint in order to perform free-view video synthesis. Free viewpoint video synthesis can be realized by estimating the distance from the multi-view video on the decoder side even when only the multi-view video is encoded. However, there is an advantage that the load on the decoder side can be reduced by acquiring the distance on the encoder side and encoding the distance image. Of course, when the distance image is transmitted simultaneously, the data amount of the distance image increases, but when the multi-view image and the distance image are encoded separately by using the correlation between the multi-view image and the distance image. Compared to this, the data volume can be reduced.

As a technique for reducing the code amount of a multi-view video using a distance image, a method described in Patent Document 1 is known. Patent Document 1 discloses a method for reducing the amount of data in encoding a color image of a target viewpoint on the premise that a color image of a reference viewpoint and a distance image of a reference viewpoint are encoded. Specifically, the color image of the target viewpoint is predicted based on the color image of the reference viewpoint, the distance image of the reference viewpoint, and the installation position and orientation of the camera between the viewpoints. At this time, prediction cannot be obtained for pixels (occlusion) that can be seen from the target viewpoint but not from the reference viewpoint, but for such pixels, a prediction value is obtained by estimation from adjacent pixels. .

Japanese Patent No. 4414379

  Here, the problems of the aforementioned MVC will be described below. In MVC, parallax compensation prediction is applied to a variable block from 16 × 16 pixels to 4 × 4 pixels. The smaller the block size, the smaller the difference code amount between the predicted image and the original image. However, since one disparity vector is assigned to the block, there is a trade-off that the amount of disparity vector code increases. Although the method for determining the block size is not defined in the standard, for example, the block size with the highest coding efficiency is adopted by comparing the data amounts when coding with several block sizes. At this time, although the occlusion part and the non-occlusion part are not explicitly specified, the block size can be expected to be small in a region where the occlusion part and the non-occlusion part are mixed. This is because scenes with different depths are mixed in the occlusion part and the non-occlusion part, so that it is considered that a predicted image cannot be generated well with a single disparity vector. That is, in MVC, in order to generate a prediction image near occlusion with high accuracy, there is a problem that the block size becomes small.

  Next, the problem of the method described in Patent Document 1 will be described with reference to FIG. 3A and 3B are images obtained by photographing the same subject with two cameras arranged side by side. 3A and 3B, it is assumed that the halftone dot portion and the shaded portion are backgrounds, have different texture patterns, and are photographed by placing a round object on the foreground. FIG. 3A is an image taken with a camera located on the left and FIG. 3B on the right. 3C is a distance image showing the distance from the camera of the subject shown in FIG. 3A, where the grid portion indicates the background distance and the horizontal line portion indicates the foreground distance. It is assumed that the area and the background area are each a single distance. Here, it is assumed that FIGS. 3A and 3C are first encoded, the prediction image of FIG. 3B is generated using FIGS. 3A and 3C, and the difference is encoded. First, using the distance of each pixel obtained in FIG. 3C and the positional relationship between the two cameras, the position of the camera that has projected and converted each pixel in FIG. 3A and photographed FIG. Convert to the image seen from. As a result, the image of FIG. 3D is obtained. In FIG. 4D, the white portion 301 is occlusion. In order to obtain high coding efficiency, it is necessary to make an image close to FIG. 3B as a predicted image by filling in the occlusion part. Japanese Patent Application Laid-Open No. H10-228561 discloses a method of filling an occlusion portion from an adjacent pixel. However, as shown in FIG. 3D, when the occlusion part is a part of the texture, it is unlikely that the texture can be sufficiently predicted by just filling in the adjacent image. In other words, it can be understood that the method described in Patent Document 1 has a problem that the accuracy of the predicted image is lowered when the occlusion portion is a texture region.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for efficiently predictively encoding an image at another viewpoint from an image at a reference viewpoint and a distance image.

In order to solve this problem, for example, an image encoding device of the present invention has the following configuration. That is,
An image encoding device that predictively encodes image data of a target viewpoint different from the reference viewpoint based on image data at a reference viewpoint and distance image data of the reference viewpoint,
Using the distance image data, identification data generating means for generating identification data for identifying whether each pixel in the image data of the target viewpoint belongs to an occlusion part or a non-occlusion part;
Input means for inputting block image data in units of blocks of a preset size from the image data of the target viewpoint;
Determination means for determining whether the target block image data input by the input means is a mixed block in which an occlusion part and a non-occlusion part are mixed by referring to the identification data generated by the identification data generation means When,
If the determination unit determines that the block is a non-mixed block, the prediction block image data for the block image data of interest is generated with reference to the image data of the reference viewpoint,
When the determination unit determines that the block is a mixed block, referring to the identification data, the block image data of interest includes an occlusion area including occlusion pixels and a non-occlusion pixel including non-occlusion pixels. The prediction area image data corresponding to each area is generated with reference to the image data of the reference viewpoint, and the obtained prediction area image data is integrated to obtain the corresponding block image data. Predicted image generation means for generating predicted block image data;
And encoding means for encoding a difference between the predicted block image data generated by the predicted image generation means and the block image data of interest.

  According to the present invention, even when an occlusion part and a non-occlusion part coexist in a block that is a coding unit, it is possible to generate encoded data with a higher compression rate than before by generating a predicted image with high accuracy. It becomes possible to do.

The block diagram of 1st Embodiment. The flowchart of 1st Embodiment. FIG. 5 is a diagram illustrating an example of an image processed in the first embodiment. The figure which shows the system configuration example of the system of 1st Embodiment. The figure which shows the example of the area | region division process of 1st Embodiment. The figure which shows the example of the area | region integration of 1st Embodiment. The figure which shows the example of the area | region boundary correction | amendment in 1st Embodiment. The flowchart which shows the detail of the mixed block prediction image generation method in 1st Embodiment. The block block diagram of a computer. The flowchart of 2nd Embodiment. The flowchart of 3rd Embodiment.

  Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Each embodiment described below shows an example in the case where the present invention is specifically implemented, and is one of the specific embodiments having the configurations described in the claims.

[First Embodiment]
In the present embodiment, assuming a system that can realize stereo stereoscopic vision and free viewpoint image synthesis, an image of one other target viewpoint is predicted from a multi-viewpoint image, one of the viewpoint images, and its distance image. An example of performing encoding will be described. Therefore, in the following description, two images and one distance image at the viewpoint of one of the images are used.

  First, an outline of the free viewpoint image synthesis and the distance image will be described. Free viewpoint image composition is to shoot an image from a certain viewpoint using a camera or other imaging device, and synthesize an image that can be seen from another viewpoint that is not actually photographed. Technology. A method using a distance image is known as one of free viewpoint image synthesis methods. A distance image is an image in which distance information of each pixel of a photographed viewpoint is stored. If there is a distance image, the coordinates of the photographed subject in the three-dimensional space can be specified, so it can be estimated where the subject appears in the image when viewed from another viewpoint.

  There are an active method and a passive method for obtaining a distance image. The active method is a method of acquiring a distance by irradiating a subject with a laser or the like. As a representative method, there is a time-of-flight method of estimating the distance from the time until the laser reciprocates. The passive method is a method of acquiring a distance using a photographed image, and a typical example is a stereo matching method using photographed images from two different viewpoints.

  When free viewpoint image composition is performed using an actively acquired distance image, it is necessary to encode an actively acquired distance image in addition to the captured multi-viewpoint image. When the distance image is passively acquired, it is not necessary to encode the distance image because the distance image can be acquired from the multi-viewpoint image. However, for the purpose of reducing the processing load at the time of reproduction or the like, when the distance image is passively acquired and encoded at the time of shooting and the distance image is not re-acquired at the time of reproduction, the distance image needs to be encoded.

A distance image used for free viewpoint image synthesis is often encoded as parallax. The parallax is an amount indicating where a point reflected in one viewpoint is captured in a captured image at the other viewpoint when the same subject (point) is captured from two different viewpoints. In general, it is expressed by how many pixels the points (hereinafter referred to as corresponding points) in which the same subject appears in different images are shifted. More specifically, the two optical cameras are parallel to each other and arranged at an interval of b (mm) on the left and right along the horizontal axis, from the photographing surface (a plane perpendicular to the optical axis and passing through the two cameras). When a point separated by z (mm) is photographed, the parallax d (pix) of the corresponding point in the two photographed images is obtained by the following equation (1).
d = bfW / (zC) (Formula 1)
Here, f is the focal length (mm) of the camera, W is the width (pix) of the captured image, and C is the width (mm) of the imaging element. When the cameras are accurately arranged on the left and right, it can be ensured that the vertical parallax of the corresponding points is 0 (epipolar constraint). Therefore, the parallax described here indicates a horizontal shift between corresponding points.

  The description of the free viewpoint image synthesis and the outline of the distance image is thus completed, and the description shifts to the description of the present embodiment.

  FIG. 4 shows a system assumed in the first embodiment. FIG. 4 shows the configuration of each part and the data flow until two images are taken by the photographing apparatus and the two images and the free viewpoint image composition result are displayed.

  First, a color image is photographed by the photographing units 420 and 421. Photographing is performed with two digital cameras with the optical axes parallel and arranged on the left and right. The left viewpoint image is I1, and the right viewpoint image is I2.

  The distance image acquisition unit 422 acquires a distance image S1 of the image I1. Therefore, in reality, the viewpoint position of the distance image acquisition unit 422 is close to that of the imaging unit 420. In the figure, the distance image is obtained by measuring the distance to the subject by the active method. In addition, it is also possible to obtain | require by the stereo matching method using I1 and I2. In the case of handling a 3D CG image, it is also possible to acquire the distance from the 3D model.

  Next, the predictive encoding unit 423 performs predictive encoding of I2 using the image I1 and the distance image S1. A feature of the present embodiment is a prediction method in the predictive encoding unit 423, which will be described in detail later. As a result of the predictive coding, prediction information for generating a predicted image of I2 and code data of a difference between I2 and the predicted image are obtained. It is desirable to perform lossless encoding for prediction information and lossy encoding for differences.

  The encoding unit 424 encodes the image I1. The encoding of I1 is preferably lossless encoding such as PNG. Note that I1 may be irreversibly encoded by JPEG or the like, but in this case, not the I1 but the degraded image decoded after irreversibly encoding I1 is input to the predictive encoding unit 423. Is desirable.

  The encoding unit 425 encodes the distance image S1. The encoding of the distance image S1 is preferably lossless encoding such as ZIP.

  Next, the transmission unit 426 transmits the code data obtained by these and information for specifying the position and orientation of the photographed camera to the reception unit 427. At this time, it is convenient to combine each piece of encoded data and information related to photographing and transmit as one file.

  When receiving the above encoded data, the receiving unit 427 separates the encoded data and supplies each to the corresponding decoding units 428 to 430. The decoding unit 428 decodes the encoded data of the image I1. The decoding unit 430 decodes the encoded data of the distance image S1. Then, the decoding unit 429 decodes the prediction difference encoded data. Then, the decoding unit 429 generates a prediction image based on the image I1 and the distance image S1 obtained by decoding and information related to shooting, and applies the prediction difference data to the prediction image. The image I2 is decoded (generated). In this embodiment, only the image I2 is irreversibly encoded, so that the output result of the decoding unit 429 is shown as I2 'in order to distinguish it from the image before deterioration.

  The color images I1 and I2 'taken so far can be displayed. Stereo stereoscopic vision can be realized with these two images. Furthermore, by performing the free viewpoint image composition by the free viewpoint image composition unit 431 using the color image I1 and the distance image S1, it is also possible to display a viewpoint image I3 that has not been photographed by the photographing apparatus.

  This is the end of the description of the overall image of the system assumed in this embodiment. Details of the process of the predictive coding unit 423, which is a feature of this embodiment, will be described below.

  Details of processing performed by the predictive encoding unit 423 will be described with reference to the flowchart of FIG. As is clear from FIG. 4, the input to the predictive coding unit 423 is the color images I1 and I2 and the distance image S1 of the color image I1. In the following description, for the sake of explanation, FIGS. 3A and 3B are input as the color images I1 and I2, respectively, and FIG. 3C is input as the distance image S1.

  In S201 of FIG. 2, first, the distance image S1 is projected and converted to estimate the distance image at the viewpoint of the color image I2 (identification data generating means). As an example, FIG. 3E shows a distance image obtained by projecting the distance image of FIG. Here, the white region 302 is an occlusion portion that cannot be seen from the viewpoint of the color image I1. Even in the case of not being an occlusion portion, a hole having no value may occur in a distance image obtained by projection conversion. This may be due to a distance image quantization error or the like. In order to suppress this influence, it is desirable to fill a minute hole of about one pixel with an intermediate value (or average value) of surrounding pixels.

  Next, identification data for identifying whether the pixel belongs to the occlusion part or the non-occlusion part is generated for each pixel from the distance image of FIG. In the present embodiment, as will be described later, it is also data for separating the occlusion part and the non-occlusion part, so that the two-dimensional array of this identification data is expressed in the same way as a normal image, but is distinguished from a normal image. For this reason, it is hereinafter referred to as a “mask image”. FIG. 3F shows an example of this mask image. In FIG. 3F, the white portion is an occlusion portion, and the shaded portion is a non-occlusion portion. The mask image only needs to be able to determine (identify) the occlusion portion and the non-occlusion portion, and therefore may be a binary image of 1 bit per pixel.

  Next, the process proceeds to S202. In S202 to S212, the color image I2 is encoded in units of blocks including a predetermined number of pixels. In the example of the present embodiment, the color image in FIG. 3B will be described as being encoded in units of block image data composed of 8 × 8 pixels.

  In S202, it is determined by referring to the mask image whether the encoding target block in the color image I2 is a block in which the occlusion part and the non-occlusion part are mixed. The determination is performed using the mask image generated in S201.

  First, the operation when it is determined as a non-mixed block in S202 will be described. For example, the block 303 shown in FIG. 3B is determined to be a non-mixed block because the corresponding mask images 305 in FIG. 3F are all non-occlusion portions. The process in the case of a non-mixed block basically performs the same process as the parallax compensation prediction used in MVC as described below.

  First, it progresses to S203 and performs a parallax vector search. In the disparity vector search, a block (predicted block image data) used for prediction of the block 303 in FIG. 3B is searched in the image in FIG. 3A, and information for specifying the position of the block in S204 is a disparity vector. Is encoded as The encoding is preferably lossless encoding such as entropy encoding.

  As a criterion for determining a block to be used for prediction in S203, a block that minimizes the sum of absolute differences of colors of pixels in the encoding target block and the prediction candidate block is selected.

  In S203, when the encoding target block is a block including only the non-occlusion portion, the prediction candidate block limits the search range according to the epipolar constraint as described above. That is, as in the present embodiment, when the camera is arranged horizontally, a block that is shifted only in the horizontal (horizontal) direction with the position of the encoding target block as a reference is set as a search range. In this case, the disparity vector may be encoded with one-dimensional position information indicating the position in the horizontal direction. Further, when all the pixels in the block are composed only of occlusion pixels, the corresponding block is searched from the entire image ignoring the epipolar constraint. In this case, the disparity vector is two-dimensional position information.

  In step S205, predicted image generation is performed. In the case of a non-mixed block, the block specified by the disparity vector searched in S203 may be used as the predicted image.

  Next, in S212, the difference value of each pixel between the prediction image of the block generated in S205 and the encoding target block is encoded. For the difference encoding, discrete cosine transform (hereinafter referred to as DCT) is performed, the transform coefficient is quantized, and entropy encoding is performed. The differential signal may be encoded by other lossy encoding such as JPEG2000 or lossless encoding such as ZIP. Then, in S213, it is determined whether or not the processing has been completed for all the blocks. If not, the position of the block image data of interest is shifted, and the processing after step S202 is performed.

  Above, description of the process when it determines with a non-mixed block by S202 is finished. Next, a process when it is determined in S202 that the target block image data is a mixed block will be described.

  The basic idea of encoding when it is determined to be a mixed block is that an occlusion area composed of pixels determined as an occlusion part in the block and a non-occlusion area composed of pixels determined as a non-occlusion part The prediction accuracy is improved by assigning prediction blocks to each of them and integrating them. That is, two disparity vectors are assigned to one block.

  Hereinafter, description will be given using the block 304 in FIG. 3B as an example of the mixed block. The block 304 corresponding to the mask image 306 in FIG. 3F is determined to be a mixed block in S202 because the hatched portion and the white portion, that is, the occlusion portion and the non-occlusion portion are mixed.

  In S206, the block of interest determined as the mixed block is divided into an occlusion part and a non-occlusion part. An example of division will be described with reference to FIG. Reference numerals 304 and 306 in FIG. 5 are the same blocks as 304 in FIG. 3B and 306 in FIG. In S206, the encoding target block is divided (separated) into a non-occlusion region 501 composed of non-occlusion pixels and an occlusion region 502 composed of occlusion pixels according to the mask image. The white areas 503 and 504 in FIG. 5 are pixels having no value, and it is sufficient to store a value that the pixel cannot take such as a negative value in terms of mounting.

  Next, in S207, a corresponding block for predicting the occlusion area is searched from the color image I1. In the present embodiment, the search is performed in FIG. As a feature when searching for a corresponding block, pixels having no value in the block 502 are ignored when evaluating the similarity of the block, thereby searching the predicted area image data of each of the occlusion area and the non-occlusion area. In this embodiment, the similarity between blocks is evaluated by the sum of absolute differences regarding pixels that are occluded in the block. In the occlusion area, it is considered that the same subject is not shown in the image of the reference viewpoint, so not only the search on the epipolar constraint on the assumption that the same object is shown between different cameras, but the entire image is searched. Is desirable. In this case, the disparity vector encodes two-dimensional data. Note that only the epipolar constraint may be searched for the purpose of reducing the processing load. In this case, there is a high possibility that the prediction accuracy of the occlusion area is lowered, but there is also an advantage that data for expressing the disparity vector can be suppressed to one dimension.

  Here, the effect of the process of S207 will be supplemented. In the image of FIG. 3A, there should be no object similar to the object existing in the occlusion area. However, there are many scenes where the occlusion area has the same texture pattern as the surrounding area. The images shown in FIGS. 3A and 3B of this embodiment are such an example, where the background is a texture pattern and a person is shown in the foreground. In such a case, regarding the occlusion area, it is considered that blocks having similar colors are found in images taken from different viewpoints, and the occlusion area can be predicted with high accuracy. Next, in S208, the disparity vector of the block including the occlusion area obtained in S207 is encoded. It is desirable to use lossless encoding as in S204.

  Next, in S209, a corresponding block for predicting the non-occlusion area 501 is searched from FIG. In S209, basically, the process of obtaining the predicted block of the occlusion area performed in S207 may be replaced with a non-occlusion area. However, since it is considered that the corresponding block in the non-occlusion region exists on the epipolar constraint, the search for the disparity vector is limited to one dimension in the horizontal direction. As a result of the search, a block including a non-occlusion area is obtained as a prediction block for the block 307 in FIG. Next, in S210, a disparity vector indicating the position of the prediction block including the non-occlusion region is encoded by the same method as in S204.

  In step S211, a mixed block prediction image is generated. Details of the processing of S211 will be described using the flowchart of FIG.

  First, in step S801, a block for predicting an occlusion area in a mixed block and a block for predicting a non-occlusion area are created. An example of creating a prediction block of block 304 in FIG. 3B is shown in FIG. Blocks 307 and 308 in FIG. 6 are the same as blocks 307 and 308 in FIG. 3A, respectively, and are blocks for predicting the occlusion area and the non-occlusion area of the mixed block 304, respectively. The mask 306 is the same as the block 306 in FIG. In order to obtain a prediction block, first, as shown in FIG. 6, regions used for prediction are cut out using a mask for each of the prediction block in the non-occlusion region and the prediction block in the occlusion region (blocks 601 and 602 in FIG. 6). Next, they are integrated with each other (block 603 in FIG. 6).

  The block 603 obtained in FIG. 6 is considered to function sufficiently as the prediction of the block 304 in FIG. However, in the present embodiment, in order to further increase the accuracy as a prediction block, the boundary position between the occlusion area and the non-occlusion area is further corrected through S802 to S804. An example of boundary correction will be described by enlarging the pixels on the line 604 that is one line of the block 603.

  First, the necessity of boundary correction will be described. FIG. 7B is an enlarged view of the line 604. FIG. 7A is an enlarged view of a line for prediction on the line 604 (hereinafter, a prediction target line), and is one line of FIG. 3B 304, which is a prediction target block. Naturally, it can be said that the prediction accuracy is higher as each pixel in FIG. 7B has a color closer to that in FIG. A straight line 701 in FIG. 7A indicates a boundary between the non-occlusion portion and the occlusion portion, and the straight lines in FIGS. 7B to 7E are the same. The pixels 702 and 703 in FIG. 7A are boundary pixels of the prediction target line, and are intermediate colors between the pixels on the non-occlusion part side (black) and the boundary pixels (dots) on the occlusion part side. Yes. On the other hand, the boundary pixels 704 and 705 in FIG. 7B that perform prediction of this line have different colors. First, since the boundary pixel 704 is a pixel at the boundary when the block 307 in FIG. 6 is cut out by the mask 306, the boundary pixel 704 includes the foreground part (black) and the background part (diagonal line) in the block 307. ) Intermediate color. The boundary pixel 705 is a pixel obtained by cutting the block 308 in FIG. As described above, it is considered that the pixels 704 and 705 that predict the boundary pixels 702 and 703 cannot achieve sufficient prediction accuracy.

  In order to improve the prediction accuracy of the boundary pixels described above, the boundary pixels 704 and 705 in FIG. 7B are first deleted in S802. This is shown in FIG.

  In step S803, the boundary pixel deleted in step S802 is complemented twice. Double interpolation is a pixel having the color of a pixel adjacent to one pixel at the boundary after deletion, such as the pixels 706 and 707 in FIG. 7C, and the pixel value after deletion is filled. The result is shown in FIG.

  In step S804, a color for predicting one pixel on both sides of the boundary is determined by blending the double-interpolated pixels. This is shown in FIG. Pixels 708 and 709 in FIG. 7E are pixels complemented by blending. The blending may be performed by the average of the two colors complemented twice.

  Note that the weighting factor for blending may be predicted from a prediction target image or the like. A technique for estimating a coefficient for blending two regions is widely known as matting, and a known method may be used. When the blend coefficient is obtained from the prediction target image, since the prediction target image cannot be known on the decoding side, it is necessary to encode the blend coefficient.

  Next, the differential encoding in S212 may be performed by the same method as that for non-mixed blocks. Finally, in S213, it is determined whether all the blocks of the prediction target image have been processed. If all the blocks have not been processed, the process returns to S202 again, and the unprocessed block is encoded. If all blocks have been processed, the process ends.

  The above is the processing details of the predictive coding unit 423 in the present embodiment. FIG. 1 shows an example of a specific block configuration diagram of the predictive coding unit 423. The role of each block in FIG. 1 will be described below.

  As shown in FIG. 1, the predictive encoding unit 426 inputs an encoding target image (color image I2) in units of blocks. In addition, a reference viewpoint color image (color image I1) and a reference viewpoint distance image (distance image S1) are also input. The output is encoded data of prediction information and difference information for predicting the reference viewpoint color image. The image memory 102 stores a reference viewpoint color image (color image I1).

  The mask image generation unit 101 generates a mask image at the viewpoint position of the imaging unit 421 from the reference viewpoint distance image and the preset position (distance) information of the two imaging units 420 and 421. That is, the process of S201 in FIG. 2 is performed.

  The block attribute determination unit 108 refers to the corresponding block of the mask image to determine whether the encoding target block is a region mixed block. That is, the process of S202 is performed.

  The disparity vector search unit 103 refers to the mask image generated by the mask image generation unit 101, determines whether the pixel block of interest is a block in which an occlusion part and a non-occlusion part are mixed, and under this determination, an image A disparity vector is searched from the memory 102. That is, in the case of a non-mixed block, a single disparity vector is searched, and in the case of a mixed block, two disparity vectors are searched for each region. That is, the processes of S203, S206, S207, and S209 are performed.

  The disparity compensation prediction unit 104 generates a predicted image using the disparity vector obtained by the disparity vector search unit 103. That is, the processing of S205 and S211 is performed.

  The DCT unit 105, the quantization unit 106, and the entropy encoding unit 107 encode the difference image. That is, the process of S212 is performed. The entropy encoding unit 107 also encodes a disparity vector. That is, the processing of S204, S208, and S210 is also performed.

  FIG. 1 shows an example in the embodiment. It goes without saying that some of the processing units may be integrated into one processing unit, and may be further divided.

  According to the present embodiment as described above, the reference image data obtained from the imaging device of the reference viewpoint among the multiple viewpoints (plurality) and the distance image data in the reference image have a viewpoint different from the reference viewpoint. A predicted image for each block, which is an encoding unit, for an image can be generated with high accuracy, and encoding efficiency can be increased.

[Modification of First Embodiment]
Each unit shown in FIG. 1 may be configured by hardware, but may be implemented as software (computer program). In this case, the software is installed in a memory of a general computer such as a PC (personal computer). Then, when the CPU of this computer executes the installed software, this computer realizes the functions of the above-described image processing apparatus. That is, this computer can be applied to the above-described image processing apparatus. An example of the hardware configuration of a computer applicable to the predictive coding apparatus according to the first embodiment will be described with reference to the block diagram of FIG.

  The CPU 1501 controls the entire computer using computer programs and data stored in the RAM 1502 and the ROM 1503, and executes the above-described processes described as being performed by the multi-view image encoding apparatus.

  The RAM 1502 is an example of a computer-readable storage medium. The RAM 1502 has an area for temporarily storing computer programs and data loaded from the external storage device 1507, the storage medium drive 1505, and the network interface 1510. Further, the RAM 1502 has a work area used when the CPU 1501 executes various processes. That is, the RAM 1502 can provide various areas as appropriate. The ROM 1503 is an example of a computer-readable storage medium, and stores computer setting data, a boot program, and the like.

  A keyboard 1504 and a mouse 1505 can be operated by a computer operator to input various instructions to the CPU 1501. The display device 1506 is configured by a CRT, a liquid crystal screen, or the like, and can display a processing result by the CPU 1501 using an image, text, or the like. For example, the image to be encoded can be displayed, and the encoded result can be displayed.

  The external storage device 1507 is an example of a computer-readable storage medium, and is a large-capacity information storage device represented by a hard disk drive device. The external storage device 1507 stores an OS (Operating System), computer programs and data for causing the CPU 1501 to implement the processes shown in FIG. 1, the above-described various tables, databases, and the like. Computer programs and data stored in the external storage device 1507 are appropriately loaded into the RAM 1502 under the control of the CPU 1501 and are processed by the CPU 1501.

  The storage medium drive 1508 reads a computer program and data recorded on a storage medium such as a CD-ROM or DVD-ROM, and outputs the read computer program or data to the external storage device 1507 or the RAM 1502. Note that part or all of the information described as being stored in the external storage device 1507 may be recorded on this storage medium and read by this storage medium drive 1508.

  The I / F 1509 is an interface for inputting an encoding target color image, a reference viewpoint color image, and a reference viewpoint distance image from the outside, and is USB (Universal Serial Bus) as an example. A bus 1510 connects the above-described units.

  In the above configuration, when the computer is turned on, the CPU 1501 loads the OS from the external storage device 1507 to the RAM 1502 according to the boot program stored in the ROM 1503. As a result, an information input operation can be performed via the keyboard 1504 and the mouse 1505, and a GUI can be displayed on the display device 1506. When the user operates the keyboard 1504 or the mouse 1505 and inputs an instruction to start an application program for gray image quantization stored in the external storage device 1507, the CPU 1501 loads the program into the RAM 1502 and executes it. As a result, this computer functions as a predictive coding apparatus.

  Note that the application program for predictive encoding executed by the CPU 1501 basically includes functions corresponding to the respective units shown in FIG. Here, the encoding result is stored in the external storage device 1507. It is apparent from the following description that this computer can be similarly applied to image processing apparatuses according to the following embodiments.

[Second Embodiment]
In the second embodiment, an example in which the encoding method of the occlusion part in the first embodiment is modified is shown.

  FIG. 10 shows a flowchart of processing performed by the predictive coding unit 423 in the second embodiment. The flowchart of FIG. 10 is obtained by changing the prediction method of the occlusion part of the mixed block in the flowchart of FIG. 2 of the predictive encoding part in the first embodiment. Therefore, the processes other than S1007, S1008, S1011, S1014, and S1015 relating to the creation of the predicted image of the occlusion part are the same as the processes given the same names in the flowchart of FIG. In the following, the description will be focused on the parts that are different from the first embodiment.

  First, in the flowchart of FIG. 10, the processing up to S1006 is completed in the same procedure as in the first embodiment. That is, when a block containing both occlusion and non-occlusion is input as an encoding target block, the block is divided into an occlusion part and a non-occlusion part in S1006. Also in the second embodiment, first, in S1007, one disparity vector is obtained for the occlusion part, as in the first embodiment.

  Next, in this embodiment, in the error determination process in S1014, the error between the predicted image of the occlusion part in the block and the prediction target image obtained in S1007 is obtained. The error is evaluated by the sum of absolute differences of the occlusion pixel colors. If the difference is equal to or greater than the threshold (the number of pixels in the block × 5 or more), the process proceeds to S1015. Note that this threshold value may be arbitrarily determined by the designer based on the degradation of image quality allowed for the system.

  In S1015, the occlusion part in the block is divided. In block division, an 8 × 8 pixel block is divided into four 4 × 4 pixel blocks (hereinafter referred to as small blocks). H. Like H.264, it may be divided into blocks of different sizes.

  When the block is divided, the parallax vector is searched again in S1007. At this time, the disparity vector is searched one by one for the divided small blocks. However, when all the pixels in the small block belong to the non-occlusion part, it is not necessary to assign parallax to the small block.

  Next, the process proceeds to S1014. In this embodiment, since the division smaller than the block of 4 × 4 pixels is not performed, the division is terminated and the process proceeds to S1008.

  In the case of performing finer division, a prediction image of the occlusion part of the encoding target block is generated by combining the prediction images of the small blocks, and the prediction error of the occlusion part of the encoding target block is obtained again. If the prediction error is less than or equal to the threshold, the process further proceeds to S1015. However, although the accuracy of prediction increases when block division is performed a lot, the amount of information of disparity vectors increases, so it is desirable to terminate with appropriate division.

  Next, in S1008, the parallax vector of the occlusion part is encoded. Except that there are a plurality of disparity vectors, encoding may be performed in the same procedure as in the first embodiment.

  In S1011, a prediction image of a mixed block is generated. The basic procedure is the same as S211 in the first embodiment. However, when generating the prediction block of the occlusion part, the only difference is that the prediction image for each small block is generated in combination. This is the end of the description of the processing of the second embodiment.

  In the second embodiment, the method of dividing the block of the occlusion part is shown. Since the block division is added, the information amount of the disparity vector is increased as compared with the first embodiment. However, since the block division is performed after the occlusion portion and the non-occlusion portion are separated, it is possible to increase the possibility of performing more accurate prediction with a smaller number of divisions compared to the method of performing block division without dividing the region.

  In the second embodiment, the method of dividing the occlusion part in the block into blocks is shown, but it is also possible to easily add a process of dividing the non-occlusion part into blocks.

[Third Embodiment]
In the first embodiment, a predicted image is generated with a disparity vector for each of the non-occlusion part and the occlusion part. However, as shown in the example of FIG. 3D, for the non-occlusion part, if projection conversion is performed using the distance image of the reference viewpoint, a predicted image is generated without assigning a disparity vector for each block. It is possible. In the present embodiment, an example is shown in which projection conversion based on a distance image is used for generating a predicted image of a non-occlusion part, whereas a disparity vector is used in the first embodiment.

  FIG. 11 is a flowchart of the process of the predictive coding unit 423 in the present embodiment. The details of the processing will be described below, focusing on portions that are different from the flowchart of the first embodiment of FIG. In the following, as in the first embodiment, an example in which the color image in FIG. 3B is predictively encoded using the color image in FIG. 3A and the distance image of the viewpoint in FIG. 3C will be described. Used for.

  First, in S1101, a mask image is generated by the same method as in S201. In step S1114, a predicted image of the non-occlusion part is generated by projection conversion. An example generated by FIGS. 3A and 3C is FIG.

  Next, as in the first embodiment, the process moves to a block unit. First, in S1102, block attribute determination is performed. In the present embodiment, one of the three types of non-occlusion single block, occlusion single block, and mixed block is determined.

  If it is determined in S1102 that the block is a non-occlusion single block, a predicted image for the non-occlusion single block is generated in S1115. As shown in FIG. 3D, since a predicted image of a block is obtained by projection conversion for a non-occlusion single block, this is used as it is.

  If it is determined in S1102 that the block is an occlusion single block, the processing from S1103 to S1105 is performed. This processing may be performed in the same manner as the flowcharts S203 to S205 in FIG. 2 of the first embodiment.

  If it is determined that the block is a mixed block in S1102, the process proceeds to S1106. S1106 to S1108 perform the same processing as S206 to S208 in FIG. By the processing so far, it is possible to predict the occlusion portion in the mixed block.

  Next, in S1111, a prediction image of a mixed block is generated. The prediction of the occlusion part can be performed with the disparity vector obtained in S1108. In addition, regarding the non-occlusion portion, in FIG. 3D, prediction can be performed by using pixels at the same position from a predicted image created by projection conversion. As in the first embodiment, the occlusion part and the non-occlusion part are integrated and boundary pixels are corrected according to the flowchart of FIG.

  Above, description of the prediction image according to the block attribute determined by S1102 is finished. Next, the processing of S1112 and S1113 may be performed in the same manner as S212 and S213 of the first embodiment. The above is the processing of the third embodiment.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (8)

An image encoding device that predictively encodes image data of a target viewpoint different from the reference viewpoint based on image data at a reference viewpoint and distance image data of the reference viewpoint,
Using the distance image data, identification data generating means for generating identification data for identifying whether each pixel in the image data of the target viewpoint belongs to an occlusion part or a non-occlusion part;
Input means for inputting block image data in units of blocks of a preset size from the image data of the target viewpoint;
Determination means for determining whether the target block image data input by the input means is a mixed block in which an occlusion part and a non-occlusion part are mixed by referring to the identification data generated by the identification data generation means When,
If the determination unit determines that the block is a non-mixed block, the prediction block image data for the block image data of interest is generated with reference to the image data of the reference viewpoint,
When the determination unit determines that the block is a mixed block, referring to the identification data, the block image data of interest includes an occlusion area including occlusion pixels and a non-occlusion pixel including non-occlusion pixels. The prediction area image data corresponding to each area is generated with reference to the image data of the reference viewpoint, and the obtained prediction area image data is integrated to obtain the corresponding block image data. Predicted image generation means for generating predicted block image data;
An image encoding apparatus comprising: encoding means for encoding a difference between predicted block image data generated by the predicted image generation means and the block image data of interest.   The predicted image generation means, when the block image data of interest is the mixed block, a boundary position between the predicted area image data of the occlusion area and the predicted area image data of the non-occlusion area And generating the predicted block image data for the block image data of interest by interpolating the pixel values after deletion with reference to the pixel values adjacent to the deleted pixels in each prediction region image The image encoding device according to claim 1. The predicted image generation means includes
In the case where the target block image data is the mixed block, a difference between the occlusion region image data in the target block image data and the predicted region image data obtained by the search is equal to or greater than a preset threshold value Error determining means for determining whether or not
A division unit for dividing the block image data of interest into a plurality of small blocks smaller than the block when the error determination unit determines that the difference is equal to or greater than the threshold;
The predicted image is searched for a predicted image from the image data of the reference viewpoint in each of the regions composed of occlusion pixels in each divided small block, and the predicted image is integrated to predict the predicted region image data of the occlusion region in the block image data of interest. The image encoding apparatus according to claim 1, further comprising: means for generating Whether the block image data of interest is a non-occlusion single block made up of only non-occlusion pixels, an occlusion single block made up of only occlusion pixels, or a mixed block in which occlusion pixels and non-occlusion pixels are mixed Determine whether or not
When the block image data of interest is a non-occlusion single block composed of only non-occlusion pixels, the prediction image generation means predicts the block image of the block of interest image data by projection conversion using the distance image. Data is generated. The image coding device according to any one of claims 1 to 3 characterized by things.   5. The image encoding device according to claim 1, further comprising a unit that encodes the image data at the reference viewpoint and the distance image data. 6. A control method of an image encoding device for predictively encoding image data of a target viewpoint different from the reference viewpoint based on image data at a reference viewpoint and distance image data of the reference viewpoint,
An identification data generating step for generating identification data for identifying whether each pixel in the image data of the target viewpoint belongs to an occlusion part or a non-occlusion part, using the distance image data. When,
An input step for inputting block image data in units of blocks of a preset size from the image data of the target viewpoint;
Whether the target block image data input by the input unit is a mixed block in which an occlusion part and a non-occlusion part are mixed is determined by referring to the identification data generated by the identification data generation unit. A determination step for determining;
Predictive image generation means
If it is determined in the determination step that it is a non-mixed block, the prediction block image data for the block image data of interest is generated by referring to the image data of the reference viewpoint,
When it is determined that the block is a mixed block in the determination step, the identification block data is referred to, and the block image data of interest includes an occlusion area composed of occlusion pixels and a non-occlusion pixel composed of non-occlusion pixels. The prediction area image data corresponding to each area is generated with reference to the image data of the reference viewpoint, and the obtained prediction area image data is integrated to obtain the corresponding block image data. A predicted image generation step of generating predicted block image data;
A method for controlling an image encoding device, wherein the encoding means includes an encoding step of encoding a difference between the prediction block image data generated in the prediction image generation step and the block image data of interest.   A program for causing a computer to function as the image encoding device according to claim 1 by being read and executed by a computer.   A computer-readable storage medium storing the program according to claim 7.

Images