JP4637672B2 - Encoding processing apparatus and method - Google Patents

Description

  The present invention relates to an encoding processing apparatus and an encoding processing method for estimating or adjusting the depth degree of 3D image display data.

In recent years, development of an image processing apparatus that reproduces a three-dimensional moving image having a depth has been performed.
There are right-eye and left-eye image data for three-dimensional image display. A typical three-dimensional image display capable of binocular stereoscopic viewing provides binocular parallax by separately presenting left and right binocular images consisting of images from different viewpoint positions to the left and right eyes of the observer. It is possible to observe three-dimensionally. That is, binocular stereoscopic image data is displayed by receiving and displaying binocular stereoscopic image data including image data from two viewpoints for the left and right eyes.
As a method of generating a two-dimensional image from an arbitrary viewpoint based on such an image, Patent Documents 1 and 2 propose a method of generating a two-dimensional image that looks natural during image observation.
In Patent Document 3, a right-eye and left-eye three-dimensional moving image is synthesized for each horizontal scanning line by using a synthesizer to form a frame image, and the synthesized frame image is converted into MPEG (Moving A technique for encoding using a picture image coding experts group) encoder is disclosed.
Patent Document 4 also outputs double-sided composite image data by alternately outputting line-by-line image data sequentially output from each of the first and second reading means, and further compression-encodes the double-sided composite image data. To do. The encoded double-side composite image data is decoded by a predetermined arithmetic decoding method and output as double-side composite image data in units of lines, and the double-side composite image data is alternately separated for each line, whereby the first single-side image data and A technique for obtaining second single-sided image data is disclosed.
Further, Patent Document 5 discloses a parallax image imaging device that can optically detect a parallax amount obtained when a subject is viewed from different points and obtain accurate depth information of the subject.
JP 2002-324249 A JP 2002-56407 A JP-A-8-70475 Japanese Patent Laid-Open No. 11-55530 JP 2001-16612 A

By the way, the right eye image data and the left eye image data for displaying a three-dimensional image as described above display (output) the same object, and generally have high correlation with each other, so compression at the time of encoding is performed. The rate can be increased.
However, a stereoscopic image has a stereoscopic effect as the difference between the right-eye moving image and the left-eye moving image increases. However, if the stereoscopic effect is reproduced greatly, the correlation is generally low and encoding is performed even if they are merged with each other. The effect to do is not great. Further, it is known that, for example, when the movement is intense, the object viewed with both eyes changes, so that the correlation between both image data does not necessarily increase.
Therefore, the present invention has been made in view of such points, and provides an encoding processing device and an encoding processing method capable of improving the encoding rate of 3D image display data. It is another object of the present invention to provide an encoding processing device and an encoding processing method that can realize code amount control in consideration of depth reproduction of a stereoscopic image.

In order to achieve the above object, an invention according to claim 1 is an encoding processing device that performs encoding processing for reproducing a three-dimensional image, and inputs image data for right eye and left eye. Image data input means, depth degree calculation means for calculating the depth degree from the input right eye and left eye image data, and input right eye and left eye image data alternately in line units Based on the calculation result of the depth data calculating means, the image data synthesizing means for superimposing and synthesizing the input image data, and the synthesized image data synthesized by the image data synthesizing means is encoded based on the calculation result of the depth degree calculating means. And image data selection means for selecting whether the image data for the right eye and the left eye is to be encoded.
According to a second aspect of the present invention, in the encoding processing apparatus according to the first aspect, the depth degree calculating means includes a correlation calculating means for calculating a correlation between two input image data, and the correlation calculating means. The depth degree is calculated based on the calculation result.
According to a third aspect of the present invention, in the encoding processing apparatus according to the second aspect, the correlation calculating means calculates a correlation using a partial region of the image data .

According to a fourth aspect of the present invention, in the encoding processing apparatus according to any one of the first to third aspects, the image data synthesizing unit alternately superimposes two or more pieces of image data in line units. It is characterized by combining them together.
According to a fifth aspect of the present invention, in the encoding processing device according to any one of the first to fourth aspects, the correlation in the correlation calculation means is a horizontal line or a vertical line of the image data. It is calculated every time.
According to a sixth aspect of the present invention, in the encoding processing apparatus according to the fifth aspect, the correlation calculating unit selects a horizontal direction or a vertical direction of input image data.
A seventh aspect of the present invention provides the encoding processing apparatus according to any one of the first to sixth aspects, wherein the encoded data encoded by the encoding means is encoded based on the JPEG2000 standard. It is characterized by being.

According to an eighth aspect of the present invention, in the encoding processing device according to the seventh aspect , the data storage means for storing the frame code data obtained by encoding the image data, and the code encoded after encoding the frame image data Reconstructing means for reconstructing frame code data from the data.
According to a ninth aspect of the present invention, in the encoding processing device according to the seventh aspect, the composition of the image data is in units of color elements.
According to a tenth aspect of the present invention, in the encoding processing device according to the seventh aspect, the composition of the image data is in units of tiles.
The invention according to claim 11 is an encoding processing method for performing an encoding process for reproducing a three-dimensional image, an image data input step for inputting image data for the right eye and the left eye, and an input Depth degree calculating step for calculating the degree of depth from the right-eye and left-eye image data, and the input image data for the right eye and the left eye that are alternately overlapped and combined in line units Based on the synthesis step, the encoding step for encoding the input image data, and the calculation result of the depth degree, the synthesized image data is to be encoded, or the right eye image and the left eye image And an image data selection step for selecting whether the data is to be encoded .

According to the first and eleventh aspects of the present invention, encoding efficiency can be increased by performing encoding with high encoding efficiency. Encoding efficiency can be improved by encoding the image data for the right eye and the left eye which selectively have parallax together.
According to the second aspect of the present invention, the degree of depth of a three-dimensional image can be easily calculated.
According to the third aspect of the present invention, there is provided means for simply estimating the overall depth tendency by simply examining a partial region of an image .
According to the fourth aspect of the present invention, it is possible to easily synthesize image data by alternately superimposing two or more pieces of image data in line units.
According to the fifth aspect of the present invention, it is possible to efficiently process image data captured or scanned not only for each horizontal line but also for each vertical line.

According to the sixth aspect of the present invention, even when the order of reading the image data is not determined in advance by selectively selecting the processing line direction so as to match the order of reading the image data. It becomes possible to process efficiently.
According to the present invention as set forth in claim 7 , since it is standardized by encoding based on the JPEG2000 standard, it is general purpose.
According to the present invention described in claim 8 , the code string level editing can be easily performed in the parser. Since the code level corresponds to the area on the corresponding image space, the code level area processing can be easily performed. Frame reproduction can be controlled by editing the code data. When the image data is merged, the code data can be controlled by combining the image data for the right eye and the left eye, and efficient code data editing can be performed.
According to the ninth aspect of the present invention, the color elements are highly correlated with each other, and it can be expected that the compression rate when the merged image data is encoded is increased.
According to the tenth aspect of the present invention, when the merging of image data is performed in units of tiles, for example, when a large image is merged without being divided into units of tiles, the storage capacity for storing the image data increases. This is effective in reducing the storage capacity for storing image data.

When encoding three-dimensional image data consisting of two image data for the right eye and left eye for encoding, a method for combining and merging the two image data, and for each line There is a method of encoding sequentially while reading image data sequentially. In the former method, new image data obtained by synthesizing two image data is generated, and the image data is read and encoded all at once. Therefore, a large storage capacity is required, but the encoding control is simple. This is realized by a general encoding method in which image data is read and encoded all at once.
On the other hand, since the latter generates image data for each line, the storage capacity is small, but encoding control is somewhat complicated. Therefore, in the present invention, the image data to be encoded is selectively selected and encoded.
First, a case where an image obtained by selectively merging (synthesizing) image data for the right eye and image data for the left eye for each line will be described as a first embodiment of the present invention.
FIG. 1 is a block diagram illustrating a configuration of an encoding processing apparatus according to the first embodiment.
The encoding processing apparatus shown in FIG. 1 includes a three-dimensional image input unit (image data input unit) 1, an image depth calculation unit (depth degree calculation unit) 2, a three-dimensional (3D) image data selection unit (image data selection unit). 3) a three-dimensional (3D) image data synthesis unit (image data synthesis unit) 4, an encoding processing unit (encoding unit) 5, a depth degree storage unit 6, and an image data storage unit 7.
The three-dimensional image input unit 1 reads right-eye and left-eye image data as three-dimensional image data, and the image depth calculation unit 2 calculates the depth degree. The three-dimensional (3D) image data selection unit 3 selects whether the image data is merged as an encoding target or independent image data according to the calculated depth degree. The 3D image data composition unit 4 merges the image data to generate new image data. The encoding processing unit 5 encodes the image data from the 3D image data synthesis unit 4 or the 3D image data selection unit 3.
The depth degree storage unit 6 stores image depth data, and the image data storage unit 7 stores image data.

FIG. 2 is a diagram for explaining generation of image data obtained by combining (merging) the right-eye image data and the left-eye image data.
When two image data are combined and encoded, new image data is generated by alternately superimposing the two image data for each line. In the case of data similar to each other as seen in the right-eye and left-eye image data for displaying a three-dimensional stereoscopic image, the combination of image data is alternately synthesized for each line with high correlation between images. An image can be expected to have a high compression rate in encoding.
It is expected that the compression rate at the time of encoding can be improved by encoding data obtained by combining a plurality of data, such as right-eye and left-eye image data for displaying a three-dimensional image.
Note that the image data for displaying the three-dimensional image is not limited to still image data, and may be continuous moving image data. In that case, as shown in FIG. 3, the right-eye image data and the left-eye image data are combined, encoded, decoded, encoded data editing / region editing, etc. What is necessary is just to decompose | disassemble into the image data for left eyes.
Further, the synthesis method may be changed in order to increase the correlation so that the compression rate is increased in the synthesis of the image data.
FIG. 4 is a diagram for explaining processing for synthesizing the right-eye image data and the left-eye image data by shifting the columns. In this example, merged data is generated by shifting the horizontal direction from each other. This is because, in 3D stereoscopic display data, the right and left positions of images representing the same object are slightly shifted from each other due to parallax, such as right-eye and left-eye image data. This is effective when the continuous data moves slightly in the left-right direction in the encoding of continuous data, which will be described later.
In particular, in the case of three-dimensional image data, when a nearby object is photographed, a parallax shift is large, so that a function of shifting in the left-right direction according to the focal length can be provided. For example, it can be expected that the compression rate of the combined image data is increased by increasing the correction amount to be shifted when the focal length is short.
The image depth calculation unit 2 calculates the depth based on the correlation between the right-eye image data and the left-eye image data.

FIG. 5 is a principle diagram of a process for encoding image data obtained by selectively merging right-eye image data and left-eye image data for each line according to the present embodiment.
The correlation calculation unit 2a provided in the image depth calculation unit 2 calculates the correlation between the partial image data of the right-eye image data and the left-eye image data. In the 3D image data selection unit 3, for example, when the correlation is high, the right eye image data and the left eye image data are separately encoded based on the correlation value calculated by the correlation calculation unit 2a, and when the correlation is low, the right eye The image data obtained by merging the image data for the left eye and the image data for the left eye is encoded.
In the calculation of the correlation value in the correlation calculation unit 2a, for example, input image data is read in line units, and the correlation value is calculated in line units. At this time, three-dimensional (right eye and left eye) image data may be directly read in line units, and correlation values may be calculated in line units. When the correlation value is calculated, the image depth calculation unit 2 calculates the degree of depth using the correlation value for each line. For example, the reciprocal of the correlation value is the depth degree. The depth degree of the entire image is calculated for each line of the image and calculated as an average value for each line.

ここで、画素の（ｘ，ｙ）は座標値であり、Ｉ（ｘ，ｙ）は画像Ｉの座標（ｘ，ｙ）の値であり、Ｊ（ｘ，ｙ）は画像Ｊの座標（ｘ，ｙ）の値である。画像は、ｘの最大値がｌｘ−１、ｙの最大値がｌｙ−１よりなる。
なお、相関値は必ずしも上記に示すような算出式（式１）により求めなくてもよく、画像データ間のデータ値の近さが算出できればよく、例えば、あるｙ番目のラインの相関Ｓ（ｙ）は、以下の式２により求めてよい。その場合はより簡潔になる。
相関値Ｓ（ｙ）は、

More specifically, the correlation value between the image data for the right eye and the image data for the left eye is calculated according to the following (Equation 1).
S (y) is a correlation value of a y-th line (y is a numerical value within a range of 0 to ly-1) of an image.

Here, (x, y) of the pixel is a coordinate value, I (x, y) is a value of the coordinate (x, y) of the image I, and J (x, y) is a coordinate of the image J (x , Y). In the image, the maximum value of x is 1x-1, and the maximum value of y is ly-1.
Note that the correlation value does not necessarily have to be obtained by the calculation formula (formula 1) as described above, and it is only necessary to calculate the closeness of the data value between the image data. For example, the correlation S (y of a certain y-th line) ) May be obtained by the following equation 2. In that case it becomes more concise.
The correlation value S (y) is

を算出して、画像全体の相関値として推定してもよい。
ここで、一部の画像領域は、推定精度が重要とあるような関心領域であったり、あるいは、処理効率を優先し、先に読み込んだ画像領域の画像データの相関を計算するのであっても構わない。全ての画像データを分析しないで済ますことは、処理効率を向上させるという大きな効果がある。 Further, the depth degree is estimated by the following relational expression.
The formula for calculating the depth is
Correlation value <S → depth degree = large Correlation value> S → depth degree = small Expression 3
In this embodiment, the depth for each image area is calculated, and the average value is estimated as the depth of the entire image. Further, when the correlation value is smaller than the predetermined value S, it is estimated that the depth degree is large, and when the correlation value is larger than the predetermined value S, the depth degree is estimated to be small.
It is also possible to estimate the depth degree by examining a part of the image area. Since only a part of the image area is examined, the processing efficiency can be improved.
In the present embodiment, the calculation of the correlation value for the depth degree estimation described above is performed for, for example, the lines from the first line to several lines (yn) in (Expression 1) or (Expression 2).

May be calculated and estimated as the correlation value of the entire image.
Here, some image regions are regions of interest where estimation accuracy is important, or even if the correlation between the image data of the previously read image regions is calculated with priority given to processing efficiency. I do not care. Not having to analyze all the image data has a great effect of improving the processing efficiency.

FIG. 6 is a flowchart showing a process of selectively merging the image data for the right eye and the left eye for each line in the encoding processing apparatus according to the first embodiment.
In this case, first, image data for right eye and left eye, which is three-dimensional image data, is input (S1). Next, the correlation value of part or all of the image data for the right eye and the left eye is calculated, and the depth degree is determined in the subsequent step S3.
Here, if the correlation value is smaller than the preset threshold value in step S3 (Y in S3), the right eye image data and the left eye image data are merged (S4). After the merged image data is encoded (S5), the encoded data is decoded (S6), and the merged image data is decomposed and returned to the original image data for the right eye and the left eye. (S7) A three-dimensional image is reproduced using the right-eye and left-eye image data (S8).
On the other hand, if the correlation value is larger than the preset threshold value in step S3 (N in S3), after encoding the image data for the right eye and the left eye (S9), the encoded data is Each is decoded (S10), and in step S8, the three-dimensional image is reproduced using the image data for the right eye and the left eye.

The determination of the depth degree in step S3 is performed by, for example, comparing the correlation between data of the same luminance information when the image data is divided into luminance information and color difference information. It is assumed that the image data is similar data having the same properties. Therefore, in the determination of the depth degree, a general similarity can be used instead of the correlation.
For example, the similarity is determined based on whether the distribution (density) histograms of image data are close to each other. The similarity determination may be performed by comparing two image data input by the input processing unit directly in pixel units as in the above example, or by comparing with image characteristics (distribution representing characteristics). It doesn't matter. For example, feature amounts such as color distribution (statistical features such as average and variance) may be compared with each other. Further, it may be a specific area or the entire area. When comparing pixel by pixel, binary image data of image data to be determined may be generated and compared with each other. Further, the user may specify the image data by comparing them.
If the similarity is close, the encoding is performed by merging. If not, the encoding is performed without merging. Then, the determination result is reflected in the decoding process of steps S6 and S10. Note that the criterion for determining whether or not the similarity is close is determined by estimating whether or not the compression rate can be increased by combining and encoding, rather than individually encoding.
Thus, in the encoding processing device according to the first embodiment, the correlation (similarity) between two image data (right eye image data and left eye image data) having different fields of view as three-dimensional image data is examined, Only when the correlation is high, the coding rate can be increased by merging the image data for the right eye and the left eye with each other in line units.

Next, adjustment of code amount by image data editing considering depth will be described as a second embodiment of the present invention. FIG. 7 illustrates depth adjustment in consideration of remaining code amount as an encoding processing apparatus of the second embodiment. It is the figure which showed the structure of the block which processes. In addition, the same code | symbol is attached | subjected to the same site | part as FIG.
7 includes a three-dimensional image input unit (image data input unit) 1, an image depth calculation unit (depth calculation unit) 2, an image editing unit (image data editing unit) 11, an encoding processing unit. (Encoding unit) 5, depth degree storage unit 6, image data storage unit (image data storage unit) 7, remaining code amount calculation unit (remaining code amount calculation unit) 12, and remaining code amount storage unit 13. .
The right-eye and left-eye image data, which is three-dimensional image data, is input from the three-dimensional image input unit 1, and the depth degree when the image data input by the image depth calculation unit 2 is reproduced is calculated. The image editing unit 11 edits the image data based on the depth degree and the remaining code amount of the remaining code amount storage unit 13.
The remaining code amount is a storage capacity that can be stored in the image data storage unit 7. The remaining code amount can be calculated by measuring the storage capacity of the image data storage unit 7 in the remaining code amount calculation unit 12. In the encoding processing unit 5, the image editing unit 11 encodes the edited image data. At this time, the image data for the right eye and the left eye, which are three-dimensional image data, may or may not be merged as described above.
The coding processing apparatus according to the second embodiment configured as described above is characterized in that the image editing unit 11 determines editing of image data based on the depth information and the remaining code amount.

FIG. 8 is a principle diagram for adjusting the depth in consideration of the remaining code amount.
The correlation calculation unit 2a of the image depth calculation unit 2 calculates a correlation value between the right-eye and left-eye image data partial image data, which is a three-dimensional image, and a correlation value indicating a depth degree at the time of reproducing the three-dimensional image Based on the remaining code amount, the partial image of the image data is edited to construct new image data and encode it. In the present embodiment, the right-eye image data and the left-eye image data are encoded separately, but they may be merged to form new image data and encoded. In the present embodiment, the image editing unit 11 edits the partial region so that the right eye image data and the left eye image data are the same, and easily adjusts the depth of the three-dimensional display image to be eliminated. ing.
As described above, the second embodiment is characterized in that the depth of the reproduced image is taken into account when adjusting the code amount.
Thus, it is possible to adjust the depth by editing the image data. In general, the depth of the three-dimensional reproduction image can be adjusted by adjusting the drawing position with respect to the drawing content of the right-eye image data corresponding to the position of the drawing content of the left-eye image data. Typically, the depth can be eliminated by making the drawing positions to be drawn the same (the same drawing content).
FIG. 9 is a flowchart showing a process for adjusting the depth of the reproduced image based on the restriction on the depth code amount.
In this case, the right-eye image data and the left-eye image data which are three-dimensional images are input (S21), and it is determined whether or not there is a margin in the remaining code amount (S22). If not (N in S22), the depth degree of the image data is calculated for each image attribute (S23), and the image data is edited so as to reduce the depth degree (S24). For example, editing is performed so that the image data for the right eye and the left eye are the same. Thereafter, encoding processing (S25) and decoding processing (S26) are performed, and reproduction processing (S27) is performed.
In the processing flow shown in FIG. 9, when it is determined in step S22 that there is no remaining code capacity, the depth degree of the image data is calculated for each image attribute in step S23, and the depth degree is determined in step S24. However, if it is determined in step S22 that the remaining code capacity is not sufficient, the right eye image data and the left eye image data may be merged. In that case, after the merge process, the processes after step S23 may be performed.
Here, a case where depth adjustment processing is performed for each image attribute will be described.

FIG. 10 is a block diagram of the depth adjustment processing for each image attribute.
The depth adjustment processing block shown in FIG. 10 includes a three-dimensional image input unit 21, an image data storage unit 22, an attribute-specific region extraction unit 23, an image attribute region data storage unit 24, an image attribute-specific depth calculation unit 25, and an image attribute-specific block. A depth data storage unit 26, a depth evaluation unit 27 for each image attribute, and an image data editing unit 28 for each image attribute are configured.
Three-dimensional image data is input from the three-dimensional image input unit 21, the image data is stored in the image data storage unit 22, and the image attribute region information is extracted by the attribute region extraction unit 23. As shown in FIG. 11, the image attribute depth calculation unit 25 includes an image attribute image extraction unit 31, a line image data extraction unit 32, a correlation calculation unit 33, a depth calculation unit 34, a line image data storage unit 35, The correlation data storage unit 36 is configured to read the image data of the attribute area by the image extraction unit 31 for each image attribute, read the image data for each line by the image data extraction unit 32 for each line, and read the image data for each line by the correlation calculation unit 33. To calculate the correlation value. The depth calculation unit 34 calculates depth data. When the correlation is calculated, the correlation value for each line is stored in the correlation data storage unit 36. The depth calculation unit 34 calculates the depth degree using the correlation value for each line in which the image data of the attribute area is stored. Further, the depth calculation unit 34 uses the stored correlation values in units of lines to total the reciprocal of the correlation values for each line of the image, calculates an average value for each line, and calculates the depth degree of the entire image. ing.
The depth information for each image attribute calculated by the depth calculation unit 25 for each image attribute in this way is stored in the depth data storage unit 26 for each image attribute. The depth evaluation unit 27 for each image attribute evaluates the presence / absence of the depth of the image attribute or the depth level using the reference value. For example, large / medium / small is determined as the depth level.

FIG. 12 is a principle diagram of depth adjustment for each image area.
The three-dimensional image data of the specific region of the image attribute of the right eye image data and the left eye image data is extracted, and the correlation between the image data of the partial regions is calculated by the correlation calculation unit 33 in the depth calculation unit 25 for each image attribute.
Then, the image data editing unit 28 edits the image data using the correlation value obtained by the correlation calculation unit 33.
In the editing by the image data editing unit 28 for each image attribute, the image attribute area of one image data is edited. Means for automatically adjusting the depth of a three-dimensional display image is provided. As typical editing, editing is performed so that the image data for the right eye and the image data for the left eye are the same for each region, and adjustment means for easily adjusting the depth of the three-dimensional display image is provided. To do. That is, an adjustment unit that easily converts a three-dimensional image display into a two-dimensional image display is provided. In addition, when there is a depth in recognizing characters, eye fatigue occurs due to the large amount of parallax. Therefore, when a character image area is included, it is effective to provide an adjustment unit that adjusts the character image area so that the depth of the character image area is preferably reduced so that the amount of parallax can be eliminated.
FIG. 13 is a flowchart showing processing for adjusting the depth for each image attribute.
In this case, after performing a calculation process for calculating the depth degree of the image data for each image attribute (S31), the depth degree is calculated for the image area whose image attribute is a character (S32). If the depth degree is not “0”, the right eye image data and the left eye image data are made the same (S33), and the process is terminated.
Note that the processing flow shown in FIG. 13 describes the processing for eliminating the depth of the character image area, but editing the image data in the image attribute area is not limited to editing the character attribute area. . In this processing flow, the image attribute area is used, but it may be an image area. The image data may be adjusted so that the depth degree of the image data of a specific image region such as a region of interest remains, and the remaining region has no depth.

Here, a calculation process for calculating the depth degree executed in step S31 shown in FIG. 13 will be described using the flowchart shown in FIG.
In this case, right-eye and left-eye image data that is three-dimensional image data is input (S41), and either right-eye or left-eye three-dimensional image data is analyzed to calculate a region for each attribute. (S42).
Next, input is performed for each line of image data for the right eye and for the left eye (S43), and a correlation value is calculated for each line of image data (S44). Then, the reciprocal of the correlation value is set as the depth degree for each line of the image (S45), the reciprocal of the correlation value is totaled for each line of the image, and the average value for each line is calculated to obtain the depth degree of the area (S46). . In subsequent step S47, it is determined whether or not all the images in the region have been performed. If not performed for all the images in the region (N in S47), the process returns to step S43 for processing.
On the other hand, if it is determined in step S47 that the image has been applied to all the images in the region (Y in S47), it is determined in step S48 whether or not the operation has been performed for all the regions, and for all regions, If not (N in S48), after changing the area (S49), the process returns to step S43 to perform the process again. On the other hand, if the process is performed for each region with respect to all regions (Y in S48), the process is terminated.
In this way, in the encoding processing device according to the second embodiment, focusing on the fact that the correlation between the image data for the right eye and the left eye is inversely proportional to the depth degree during reproduction, By editing the input image data and controlling the code amount based on the remaining code amount and the depth degree, the code amount control considering the reproduction image quality of the three-dimensional image is performed. For example, it is possible to improve the compression rate in encoding by partially modifying the image data so that the two image data for the right eye and the left eye are partially the same.

Next, a case where the line direction is determined and the image data for the right eye and the left eye are merged and combined for each line will be described as a third embodiment of the present invention.
The basic configuration of the encoding processing apparatus of the present invention reads image data in units of lines, evaluates the degree of depth in units of lines, selectively merges in units of lines, and encodes the merged image data. . In the image processing apparatus of this embodiment, reading of image data, calculation and evaluation of the degree of depth, or merging of code data all process image data in units of lines. For this reason, it is desirable that the image data of each line unit matches the reading direction of the image data. For example, if image data merged in units of lines while being read in units of lines is synthesized, the processing can be made efficient.
Therefore, in the present embodiment, the reading direction (horizontal direction or vertical direction) of the image data and the direction of the line of processing in units of line of the image data in the subsequent process are configured to increase the processing efficiency. Is possible.
FIG. 15 is a diagram for describing a direction (horizontal direction or vertical direction) in which image data is read. If the captured or scanned image data can be read and processed in the same order, the processing can be performed efficiently. This is because if processing is performed after all image data has been read, a memory for temporarily storing the image data is required.
In the present embodiment, image data is read (captured or scanned) by reading image data for the right eye and the left eye for each vertical line as well as for each horizontal line and calculating the correlation between the lines. It is possible to realize efficient processing according to the order.
It should be noted that the reading direction may be configured so that the horizontal direction and the vertical direction can be selectively selected so that the input image data can be processed in order. By selectively selecting the processing line direction so as to match the reading order of the image data, it is possible to efficiently process even when the reading order of the image data is not determined in advance.

Next, depth editing at a code level will be described as a fourth embodiment of the present invention.
FIG. 16 is a diagram illustrating a case where image data to be combined is in units of tiles.
When the image data for the right eye and the left eye is divided in advance in units of blocks (tiles) and encoded independently for each block of image data, the image data to be merged is also in blocks (tiles). ) Is a unit. In particular, when a large image is merged without being divided into tile units, the image data capacity is large. Therefore, since the storage capacity for storing the image data is large, it is effective to divide and merge in block units.
FIG. 17 is a diagram illustrating a case where image data to be combined is in units of components (color elements).
In FIG. 17, the image data for the right eye and the image for the left eye are each composed of image data composed of three components. This is because the correlation between the image data is low unless the composition is performed for each color element (for example, every three RGB elements). That is, the color elements are highly correlated with each other, and it can be expected that the compression rate when the merged image data is encoded is increased. Conversely, image data between different components cannot be expected to have a high correlation. In the present embodiment, the number of components is three. However, the number of components may be four as long as it is composed of a plurality of components.

FIG. 18 is a principle diagram of depth degree adjustment by code string editing according to the fifth embodiment.
When combining two pieces of image data, the image data is divided into blocks (tiles) so that the image data to be combined is separated from each other, and each block (tile) is independently encoded and independent. Independent code data is generated for each block and edited at the code data level.
When such encoding is encoded by JPEG2000 (ISO / IEC 15444-1) described above, editing at the code level can be easily advanced. In the present embodiment, the method for encoding the synthesized image data is easier to control the code amount of the partial area because the data of both is collected as compared with the case of encoding separately and independently. We also paid attention to where it is.
According to the encoding of the JPEG2000 standard, editing at the code level can be easily realized. Since the code level corresponds to the area in the corresponding image space, the code level area processing can be easily performed. For example, the encoding is performed with different compression rates in the area and other areas. In particular, encoding is performed at different compression rates for the attention area and the other areas.
Hereinafter, encoding will be described based on the JPEG2000 standard.
The encoding of the JPEG2000 standard is performed in the following procedure.
First, frame data of an interlaced image is converted into data for each color component of Y, Cr, and Cb. Next, dimensional discrete wavelet transform is performed on the color data of each color component. The wavelet coefficients obtained in this way are subjected to scalar quantization processing prescribed in JPEG2000. Next, the entropy encoding process (arithmetic encoding process by so-called coefficient modeling) prescribed in JPEG2000 is performed on the scalar quantized data. After all the color data is processed as described above, a code string defined by JPEG2000 is generated.
The decoding process is the reverse procedure.
Of course, these processes may be realized by a hardware circuit. The processing speed can be increased. Note that there is already an image processing apparatus that implements all the encoding processing compliant with JPEG2000 with a hardware circuit.

FIG. 19 is a diagram for explaining a hierarchical encoding algorithm that is the basis of JPEG 2000. The color space transform / inverse transform unit 110, the two-dimensional wavelet transform / inverse transform unit 111, the quantization / inverse quantization unit 112, and the entropy The encoding / decoding unit 113 and the tag processing unit 114 are configured. One of the biggest differences compared to the JPEG algorithm is the conversion method. Discrete cosine transform (DCT) is used in JPEG, and discrete wavelet transform (DWT) is used in the hierarchical coding compression / decompression algorithm.
The advantage that DWT has better image quality in the high compression region than DCT is one of the major reasons adopted in JPEG2000, which is the successor algorithm of JPEG.
Another major difference is that, in the latter case, a function block called a tag processing unit 114 is added in order to perform code formation at the final stage. In this part, compressed data is generated as a code stream during the compression operation, and a code stream necessary for decompression is interpreted during the decompression operation. And with the code stream, JPEG2000 can realize various convenient functions. For example, as shown in FIG. 23, the compression / decompression operation of a still image can be freely stopped at an arbitrary hierarchy (decomposition level) corresponding to octave division in block-based DWT.
A color space conversion unit is often connected to the input / output portion of the original image. For example, the RGB color system composed of R (red) / G (green) / B (blue) components of the primary color system and the Y (yellow) / M (magenta) / C (cyan) components of the complementary color system This corresponds to the part that performs conversion from the YMC color system consisting of the above to the YUV or YCbCr color system or the reverse conversion.
Hereinafter, the JPEG2000 algorithm will be described in detail.
As shown in FIG. 20, in general, each color image component (in this case, the RGB primary color system) is divided into rectangular regions (tiles). Each tile, for example, R00, R01, ..., R15 / G00, G01, ..., G15 / B00, B01, ..., B15, is a basic unit for executing the compression / decompression process. Therefore, the compression / decompression operation is performed independently for each component and for each tile.
At the time of encoding, the data of each tile 120 of each component is input to the color space conversion unit 110 in FIG. 19 and subjected to color space conversion, and then the two-dimensional wavelet conversion unit 111 performs two-dimensional wavelet conversion (forward conversion). Is applied and spatially divided into frequency bands.

FIG. 21 shows sub-bands at each composition level when the number of composition levels is three. That is, the tile original image (0LL) (decomposition level 0) obtained by the tile division of the original image is subjected to two-dimensional wavelet transform, and sub-bands (1LL, 1HL, 1LH, 1HH). Subsequently, the low-frequency component 1LL in this hierarchy is subjected to two-dimensional wavelet transform to separate sub-bands (2LL, 2HL, 2LH, 2HH) shown in the decomposition level 2. Similarly, the low-frequency component 2LL is also subjected to two-dimensional wavelet transform to separate the sub-bands (3LL, 3HL, 3LH, 3HH) shown in the decomposition level 3.
Further, in FIG. 23, the sub-bands to be encoded at each decomposition level are represented in gray.
For example, when the number of decomposition levels is 3, the sub-bands shown in gray (3HL, 3LH, 3HH, 2HL, 2LH, 2HH, 1HL, 1LH, 1HH) are to be encoded, and the 3LL sub-band is Not encoded.
Next, the bits to be encoded are determined in the designated encoding order, and the context is generated from the bits around the target bits in the quantization unit 112 in FIG.
The wavelet coefficients that have undergone the quantization process are divided into non-overlapping rectangles called “precincts” for each sub-band. This was introduced to use memory efficiently in implementation. As shown in FIG. 23, one precinct is composed of three rectangular regions that are spatially matched. Further, each precinct is divided into non-overlapping rectangular “code blocks”. This is the basic unit for entropy coding.

The entropy encoding unit 113 (see FIG. 19) performs encoding on tiles of each component by probability estimation from the context and the target bit. In this way, encoding processing is performed in tile units for all components of the original image.
The minimum unit of code data formed by the entropy encoding unit 113 is called a packet. The packets are sequenced in progressive order, which is indicated by one of the image header segments. The packets are arranged by some progressive order data, eg, region, resolution, layer, and color component, respectively. That is, in the JPEG2000 standard, by changing the priority order of the four image elements of image quality (layer (L)), resolution (R), component (C), and position (precinct (P)), the following 5 Street progression is defined.
(1) LRCP progression: Since decoding is performed in the order of precinct, component, resolution level, and layer, the image quality of the entire image is improved each time the layer index is advanced, and the progression of image quality can be realized. Also called layer progression.
(2) RLCP progression: Since the decoding is performed in the order of precinct, component, layer, and resolution level, the progression of resolution can be realized.
RPCL progression: Since decoding is performed in the order of layer, component, precinct, and resolution level, it is a progression of resolution as in RLCP, but the priority of a specific position can be increased.
(3) PCRL progression: Since decoding is performed in the order of layer, resolution level, component, and precinct, decoding of a specific part is prioritized, and progression of spatial position can be realized.
(4) CPRL Progression: Since decoding is performed in the order of layer, resolution level, precinct, and component, for example, when a color image is progressively decoded, a component progression that first reproduces a gray image can be realized.
As described above, in the JPEG2000 standard, an image is divided into regions (image constituent elements such as tiles or precincts), resolutions, hierarchies (layers), and color components, and each is independently encoded as a packet. These packets are characterized in that they can be identified and extracted from the code stream without decoding.
Finally, the tag processing unit (code string forming unit) combines all the encoded data from the entropy coder unit into one code stream and performs processing for adding a tag thereto.

FIG. 22 simply shows the structure of the code stream. Tag information called a header is added to the head of the code stream and the head of the partial tiles constituting each tile, followed by the encoded data of each tile. A tag is again placed at the end of the code stream.
On the other hand, at the time of decoding, contrary to the time of encoding, image data is generated from the code stream of each tile of each component.
This will be briefly described with reference to FIG. In this case, the tag processing unit 114 interprets tag information added to the code stream input from the outside, decomposes the code stream into code streams of each tile of each component, and each tile of each component. Decoding processing is performed for each code stream.
The position of the bit to be decoded is determined in the order based on the tag information in the code stream, and the inverse quantization unit 112 determines from the sequence of the peripheral bits (that has already been decoded) at the target bit position. A context is created.
The entropy decoding unit 113 performs decoding by probability estimation from the context and the code stream, generates a target bit, and writes it in the position of the target bit. Since the data decoded in this way is spatially divided for each frequency band, the two-dimensional wavelet inverse transform unit 111 performs two-dimensional wavelet inverse transform on each of the tiles of each component of the image data. Restored. The restored data is converted into original color system data by the color space inverse conversion unit.
As described above, since JPEG2000 code data has a hierarchical structure in units of packets having five types of progression, code data editing processing such as deletion of packets constituting the hierarchy in units of layers can be easily performed. Can do.

It is the block diagram which showed the structure of the encoding processing apparatus as the 1st Embodiment of this invention. It is a figure for demonstrating the production | generation of the image data which each synthesize | combined (merged) the image data for right eyes and left eyes. It is a figure for demonstrating the merge in moving image data. It is a figure for demonstrating the process which shifts a row | line | column and synthesize | combines, when synthesize | combining image data for right eyes and image data for left eyes. FIG. 4 is a principle diagram of a process for selectively encoding image data obtained by merging right-eye image data and left-eye image data for each line. It is the flowchart which showed the process which merges the image data for right eyes and left eyes selectively for every line in the encoding processing apparatus of 1st Embodiment. It is the figure which showed the structure of the block which performs depth adjustment processing in consideration of the amount of remaining codes as an encoding processing apparatus of 2nd Embodiment. It is a principle figure which adjusts the depth which considered the amount of remaining codes. It is the flowchart which showed the process which adjusts the depth of a reproduction | regeneration image based on the restrictions of the back code amount. It is a block diagram of a depth adjustment process for each image attribute. It is the figure which showed the structure of the calculation process block which calculates the depth degree for every image attribute. It is a principle figure of the depth adjustment for every image area. It is the flowchart which showed the process which adjusts depth for every image attribute. It is the flowchart which showed the calculation process which calculates the depth degree for every image attribute. It is a figure for demonstrating the direction (horizontal direction or vertical direction) which reads image data. It is the figure which showed the case where the image data to synthesize | combine is a tile unit. It is the figure which showed the case where the image data to synthesize | combine is a component (element of color) unit. It is a principle figure of the depth degree adjustment by code sequence editing concerning a 5th embodiment. It is a figure for demonstrating the hierarchical encoding algorithm used as the foundation of JPEG2000. It is a figure for demonstrating the basis of tile division | segmentation. It is a figure for demonstrating a decomposition level and a sub band. It is a figure for demonstrating the structure of a code stream. It is a figure for demonstrating a precinct and a code block.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Three-dimensional image input part, 2 ... Image depth calculation part, 3 ... 3D image data selection part, 2a ... Correlation calculation part, 4 ... 3D image data synthesis part, 5 ... Coding process part, 6 ... Storage part, 7 ... Image data storage unit, 11 ... Image editing unit, 12 ... Remaining code amount calculation unit, 13 ... Remaining code amount storage unit, 21 ... Three-dimensional image input unit, 22 ... Image data storage unit, 23 ... Region extraction unit for each attribute 24 ... Image attribute area data storage unit, 25 ... Depth calculation unit for each image attribute, 26 ... Depth data storage unit for each image attribute, 27 ... Depth evaluation unit for each image attribute, 28 ... Image data editing unit for each image attribute, 31 ... Image extraction unit for each image attribute, 32 ... Image data extraction unit for each line, 33 ... Correlation calculation unit, 34 ... Calculation unit, 35 ... Image data storage unit for each line, 36 ... Correlation data storage unit, 110 ... Color space conversion / reverse Conversion unit, 111 ... 2D wavelet Transform and inverse transform unit, 112 ... quantization and inverse quantization unit, 113 ... entropy encoding and decoding unit, 114 ... tag processing unit

Claims (11)

  An encoding processing apparatus for performing encoding processing for reproducing a three-dimensional image, image data input means for inputting image data for right eye and left eye, and input for right eye and left eye Depth degree calculating means for calculating the depth degree from the image data, image data synthesizing means for alternately superimposing the inputted right eye and left eye image data in line units, and inputted image data Based on the calculation result of the encoding means for encoding and the depth degree calculation means, the synthesized image data synthesized by the image data synthesizing means is to be encoded, or the images for the right eye and the left eye An encoding processing apparatus comprising: image data selection means for selecting whether data is to be encoded.   The encoding processing apparatus according to claim 1, wherein the depth degree calculating unit includes a correlation calculating unit that calculates a correlation between two input image data, and the depth degree is calculated based on a calculation result of the correlation calculating unit. An encoding processing apparatus characterized by calculating.   The encoding processing apparatus according to claim 2, wherein the correlation calculating unit calculates a correlation using a partial region of the image data. The encoding processing apparatus according to any one of claims 1 to 3, wherein the image data synthesizing unit synthesizes two or more pieces of image data which are alternately overlapped in line units. Encoding processing device. 5. The encoding processing apparatus according to claim 1, wherein the correlation in the correlation calculation unit is calculated for each horizontal line or each vertical line of the image data. Encoding processing device. 6. The encoding processing apparatus according to claim 5, wherein the correlation calculating unit selects a horizontal direction or a vertical direction of input image data. The encoding processing apparatus according to any one of claims 1 to 6, wherein the encoded data encoded by the encoding means is encoded according to the JPEG2000 standard. Processing equipment. 8. The encoding processing apparatus according to claim 7, further comprising: a data storage unit that stores frame code data obtained by encoding image data; and frame code data that is encoded after the frame image data is encoded. An encoding processing apparatus comprising: a reconfiguring unit configured. 8. The encoding processing apparatus according to claim 7, wherein the composition of the image data is in units of color elements. 8. The encoding processing apparatus according to claim 7, wherein the synthesis of the image data is in units of tiles. An encoding processing method for performing encoding processing for reproducing a three-dimensional image, an image data input step for inputting image data for right eye and left eye, and input for right eye and left eye A depth degree calculating step for calculating a depth degree from the image data, an image data combining step for combining the input right eye image data and left eye image data alternately in line units, and input image data Based on the encoding step for encoding and the calculation result of the depth degree, it is selected whether to synthesize the synthesized image data or to encode the right-eye image data and the left-eye image data. And an image data selection step.

Images