JP5614269B2 - 3D image processing method and 3D image processing apparatus - Google Patents

Description

  The present invention relates to a three-dimensional image processing method and a three-dimensional image processing apparatus.

  In recent years, a left-eye image and a right-eye image are given to the left eye and the right eye, respectively, to provide a three-dimensional (stereoscopic) image. The left-eye image signal and the right-eye image signal are generated by a three-dimensional imaging device or a computer graphics device. The three-dimensional imaging device has two imaging devices arranged to generate parallax, and the two imaging devices generate a left-eye image signal and a right-eye image signal. The computer graphics device generates a left-eye image signal and a right-eye image signal so as to generate parallax. In other words, the left eye image signal and the right eye image signal are generated by two image signal generation systems.

  The two types of image signal generation systems are manufactured so that the characteristics match as much as possible so as not to generate a difference other than parallax. However, it is not easy to completely match, and some difference occurs.

  In particular, moving image signals for right eye and left eye used as 3D video are MPEG2 and H.264. It is recorded or transmitted after being converted into encoded video data according to a moving image encoding standard such as H.264. In general, the read or received encoded video data is decoded and reproduced as a video signal, which is then supplied to a display device.

  In encoding a moving image signal, in order to keep the converted data within a predetermined data amount (bit rate), it is generally irreversible compression, and an image at the time of reproduction includes encoding noise. Therefore, different encoding noises exist in the right-eye and left-eye video data of the 3D video that is decoded and reproduced once encoded image data.

  When there is no parallax between the left and right images and a difference occurs between the same parts, different images are input to the left and right eyes of the observer. This becomes a factor that hinders the establishment of stereoscopic vision and leads to a burden on the observer such as eye strain.

JP 09-275578 A

  According to the embodiment, in a three-dimensional image, a discrepancy between left and right image signals such as coding noise is suppressed, and a three-dimensional image with less sense of incongruity is provided.

  According to one aspect of the invention, one of the left-eye image and the right-eye image is a standard image, the other is a reference image, the standard image is divided into a plurality of blocks, and a parallax evaluation is detected for each of the divided blocks. A block that determines a block satisfying a predetermined parallax evaluation as a parallax-free part and corrects the pixel value of the corresponding reference image belonging to the parallax-free part to be close to the pixel value of the reference image. An original image processing method is provided.

  According to another aspect of the invention, one of the left-eye image and the right-eye image is a reference image, and the other is a reference image, and a disparity evaluation with the reference image is detected for each block obtained by dividing the reference image, and the disparity evaluation is performed. Blocks that satisfy the predetermined condition are corrected so that the pixel value of the corresponding image for the left eye and the pixel value of the image for the right eye belonging to the non-parallax portion are close to each other. A three-dimensional image processing apparatus including a correction unit is provided.

  According to the above aspect, it is possible to provide a three-dimensional image with a little uncomfortable feeling even if a difference not caused by parallax occurs in the left and right image signals in the three-dimensional image.

FIG. 1 is a diagram for explaining generation of three-dimensional image data used in the first embodiment. FIG. 2 is a block diagram illustrating a configuration of the three-dimensional image processing apparatus according to the first embodiment. FIG. 3 is a flowchart showing processing in the three-dimensional image processing apparatus according to the first embodiment. FIG. 4 is a diagram for explaining a block in the standard image and a search range in the reference image. FIG. 5 is a diagram for explaining the block matching process. 6A and 6B are diagrams for explaining the correction process. FIG. 6A shows a block determined to have no parallax, FIG. 6B shows a corrected left eye image, and FIG. 6C shows a corrected right eye image. Show. FIG. 7 is a diagram showing a configuration when the corrected left-eye image data and right-eye image data are encoded. FIG. 8 is a block diagram illustrating a schematic configuration of the image processing apparatus according to the second embodiment. FIG. 9 is a functional block diagram of the image processing apparatuses according to the first and second embodiments realized by a computer.

  Embodiments of the present invention will be specifically described below with reference to the drawings.

  FIG. 1 is a diagram for explaining generation of three-dimensional image data used in the first embodiment.

  As shown in FIG. 1, a person 2 is photographed with a three-dimensional imaging device (three-dimensional camera) 10 against a background of a distant mountain 3, a close mountain 4, a cloud 6, a sky 7, and a mountain 5 partially hidden in the clouds. Consider the case.

  The three-dimensional imaging device 10 has two imaging systems for the left eye and the right eye arranged so as to generate parallax. The imaging system for the left eye includes an imaging lens 11A, an imaging element 12A, a lens barrel 13A, and a signal processing circuit 14A that processes the output of the imaging element 12A and outputs image data for the left eye. Similarly, the imaging system for the right eye includes an imaging lens 11B, an imaging element 12B, a lens barrel 13B, and a signal processing circuit 14B that processes the output of the imaging element 12B and outputs image data for the right eye. Have. The three-dimensional imaging device 10 is a still image camera (digital camera) or a moving image camera (digital movie camera), and in the case of a moving image camera, outputs images continuously at a predetermined frame rate. The three-dimensional display device receives left-eye image data and right-eye image data output from the three-dimensional imaging device 10 and displays a three-dimensional image.

  As shown in FIG. 1, the left-eye imaging system and the right-eye imaging system output left-eye image data and right-eye image data that generate parallax because the optical axes are shifted. It is desirable that the left-eye imaging system and the right-eye imaging system output the same image data except that the optical axis is shifted and parallax is generated. Therefore, in the imaging system for the left eye and the imaging system for the right eye, each part of the imaging lenses 11A and 11B, the imaging elements 12A and 12B, the lens barrels 13A and 13B, and the signal processing circuits 14A and 14B is as identical as possible. It is manufactured as follows. However, in reality, they cannot be manufactured to be completely the same, so it is inevitable that a certain degree of difference occurs.

  The three-dimensional image processing apparatus according to the first embodiment processes the left-eye image data and the right-eye image data as described above, and the processed left-eye image data and right-eye image data are processed into the three-dimensional display apparatus. Output to. Further, the image data for the left eye and the image data for the right eye processed by the 3D image processing apparatus of the embodiment may be stored in a digital recorder for 3D images.

  As described above, the three-dimensional image targeted in the first embodiment is image data for right eye and left eye generated based on the principle of binocular parallax.

  FIG. 2 is a block diagram illustrating a configuration of the three-dimensional image processing apparatus according to the first embodiment. The three-dimensional image processing apparatus according to the first embodiment is realized by a computer as will be described later, for example.

  As shown in FIG. 2, the three-dimensional image processing apparatus according to the first embodiment includes an image data input unit 21, a parallax detection / determination unit 22, a correction unit 23, an image data output unit 24, and a control unit 25. A memory controller 26 and a memory 27. The memory 27 may be provided outside the 3D image processing apparatus.

  The image data input unit 21 receives left-eye image data and right-eye image data input from the outside, and stores them in the memory 27 through the memory controller 26. The parallax detection / determination unit 22 reads the image data for the left eye and the image data for the right eye from the memory 27 through the memory controller 26, and detects a portion without parallax. The correction unit 23 performs correction processing on a portion determined to have no parallax in the left-eye image data and the right-eye image data, and stores the corrected left-eye image data and right-eye image data in the memory 27. . The image data output unit 24 reads and outputs the corrected left-eye image data and right-eye image data from the memory 27 through the memory controller 26 in response to a request from the outside. The control unit 25 performs start / end control of the image data input unit 21, the parallax detection / determination unit 22, the correction unit 23, the image data output unit 24, and the memory controller 26, and interface control with the outside.

  FIG. 3 is a flowchart showing processing in the three-dimensional image processing apparatus according to the first embodiment. In the following description, it is assumed that one of the left and right images is a standard image and the other image is a reference image.

  In step S11, the reference image screen is divided into a plurality of (K) blocks. Each block has N × M pixels, for example, M = N = 16.

  In step S12, zero is set to the variable k. The variable k can take a value from 0 to K-1.

  In step S13, the k-th block BLk is matched, and the zero vector error S and the minimum vector error Smin within the search range are calculated.

  FIG. 4 is a diagram for explaining a block in the standard image and a search range in the reference image. As shown in FIG. 4, the screen of the reference image 30 is divided into a plurality of blocks. FIG. 4 shows a block 31 to be processed. In the reference image 40, the corresponding block 41 is a block at a position corresponding to the block 31. Here, the correspondence between the block of the standard image 30 and the block of the reference image 40 is referred to as a vector, and the vector of the block 31 of the standard image 30 and the corresponding block 41 in the reference image 40 is referred to as a zero vector. A sum of absolute differences of pixel values of two blocks represented by vectors is referred to as a vector error, and an error of a zero vector is represented by S.

  In the reference image 40, a rectangle 42 including a corresponding block 41 indicates a search range. Starting from the upper left block 43 within the search range 42, the minimum value of vector errors for all blocks within the search range 42 is represented by Smin while shifting the block position by one pixel. Therefore, there is a case where Smin = S, and such a case is common in a portion where the parallax is very small.

  FIG. 5 is a diagram illustrating the block matching process in step S13.

  5A and 5B show examples of the left-eye image and the right-eye image when the scene of FIG. 1 is photographed, and the left-eye image data and the right-eye image input from the outside. It is an image generated from data. In (A) and (B) of the figure, reference numbers 2A and 2B are hidden in person 2, 3A and 3B are in distant mountain 3, 4A and 4B are in close mountain 4, and 5A and 5B are partially hidden in clouds. Mountain 5, 6 A and 6 B correspond to cloud 6, and 7 A and 7 B correspond to sky 7, respectively.

  FIG. 5C shows an image in which the left-eye image and the right-eye image shown in FIGS. 5A and 5B are superimposed. The portions other than the portions 2A and 2B corresponding to the person are distant views and have almost no overlap because they have no parallax, but different images overlap in the portions 2A and 2B.

  In a block having no parallax, the zero vector error S has a very small value, and the difference between the zero vector error S and the minimum vector error value Smin also has a very small value. In other words, when both of these values are small, it can be said that the block has no parallax.

  Returning to the flowchart of FIG. 3, in step S <b> 14, it is determined whether the target block is a block without parallax. Specifically, it is determined whether the absolute value of the zero vector error S is smaller than the first threshold value sth and the absolute value of the difference between the zero vector error S and the minimum vector error value Smin is smaller than the second threshold value vth. If both are satisfied, it is determined that the block has no parallax, and the other portion is determined as a block having parallax.

  Specifically, it is determined as a block with no parallax when the following two determination formulas are satisfied.

S <sth
abs (S-Smin) <vth
If it is determined that the block has no parallax, the process proceeds to step S15; otherwise, the process proceeds to step S16.

  In step S15, the correction strength α (0 ≦ α ≦ 1.0) is set to a relatively large value Dα, and the process proceeds to step S17.

  In step S16, the correction strength α (0 ≦ α ≦ 1.0) is set to a relatively small value Sα, and the process proceeds to step S17.

  In step S17, the pixel values in the target block BLk of the standard image and the corresponding block SBLk of the reference image are corrected. In the first embodiment, the following correction is performed.

  Since the block has N × M pixels as described above, the pixels in the block are represented by pixels (i, j) (0 ≦ i <N, 0 ≦ j <M). The pixel value of the base image before correction is inS (i, j), the pixel value of the reference image before correction is inR (i, j), the pixel value of the base image after correction is outS (i, j), and after correction When the pixel value of the reference image is expressed by outR (i, j), the correction formula is expressed as follows.

outS (i, j) = α × (inS (i, j) + outR (i, j)) / 2
+ (1-α) × inS (i, j)
outR (i, j) = α × (inS (i, j) + outR (i, j)) / 2
+ (1-α) × inR (i, j)
When it is determined that there is no parallax of the block, Dα is set to a value close to 1, and the pixel values of the target block of the base image and the corresponding block of the reference image are each brought close to the pixel average value. When α = 1 is set, the left and right pixels have completely the same value, and the processing with the highest correction strength is performed. Dα when it is determined that there is no block parallax may be any value as long as the value is close to 1. If it is determined that the block has parallax, α = 0 is set, and the input pixel is used as an output pixel as it is. Therefore, substantial correction processing is not performed.

  6A and 6B are diagrams for explaining the correction process. FIG. 6A shows a block determined to have no parallax, FIG. 6B shows a corrected left-eye image, and FIG. C) shows the right eye image after correction. In the part other than the person, the pixel values of the left and right pixels at the corresponding positions are corrected so as to approach the average value of the pixel values of the left and right pixels, respectively.

  Returning to FIG. 3, in step S18, it is determined whether or not an unprocessed block remains. If it remains, the process proceeds to step S19, and if not, the process ends.

  In step S19, the variable k is incremented by 1, and the process returns to step S13. As a result, the above processing is repeated for all blocks, and correction is performed on the base image and the reference image, that is, the images for the left eye and the right eye.

  In the first embodiment, the above-described correction is performed in step S17. However, the correction method can be variously modified. Next, one modified example will be described.

  In this modification, selSR (i, j) represents the higher frequency component included in the target block of the base image and the corresponding block of the reference image. The correction formula is as follows.

outS (i, j) = α × selSR (i, j) + (1−α) × inS (i, j)
outR (i, j) = α × selSR (i, j) + (1−α) × inR (i, j)
As described above, selSR (i, j) is the one that contains more high-frequency components among the target block of the base image and the corresponding block of the reference image. Therefore, in the background portion, correction is made so that the change in pixel value approaches a more rapid one.

  How much high frequency components are included is determined by calculating the activity in the block. The activity is the sum of the horizontal adjacent difference absolute value sum and the vertical adjacent difference absolute value sum in the N × M pixel block. The activity calculation method is as follows.

  Assuming that the activity of the target block of the reference image is actS and the activity of the corresponding block of the reference image is actR, the program for calculating actS and actR is as follows.

// calculation of actS actS = 0;
for (j = 0; j <M; j ++) {
for (i = 0; i <N−1; i ++) {
actS + = abs (inS [i, j] −inS [i + 1, j])
}
}
for (j = 0; j <M−1; j ++) {
for (i = 0; i <N; i ++) {
actS + = abs (inS [i, j] −inS [i, j + 1])
}
}
// Calculation of actR actR = 0;
for (j = 0; j <M; j ++) {
for (i = 0; i <N−1; i ++) {
actR + = abs (inR [i, j] -inR [i + 1, j])
}
}
for (j = 0; j <M−1; j ++) {
for (i = 0; i <N; i ++) {
actR + = abs (inR [i, j] -inR [i, j + 1])
}
}
The actS and actR calculated as described above are compared, and if actS is greater than actR, selSR (i, j) is set to a pixel value of inS [i, j]. When actS is less than or equal to actR, selSR (i, j) is a pixel value of inR [i, j].

  The left-eye image data and right-eye image data corrected as described above are supplied to a three-dimensional display device and used for displaying a three-dimensional image. Further, when the corrected left-eye image data and right-eye image data are transmitted or stored, an encoding process is performed to compress the data amount, and then transmitted or stored.

  FIG. 7 is a diagram showing a configuration when the corrected left-eye image data and right-eye image data are encoded. The corrected left-eye image data and right-eye image data are supplied to the left-eye encoding unit 51A and the right-eye encoding unit 51B, respectively. The left-eye encoding unit 51A and the right-eye encoding unit 51B encode the left-eye image data and the right-eye image data, respectively, using a known encoding method, and the left-eye encoded image data and the right-eye image data. Encoded image data is output. The left-eye encoding unit 51A and the right-eye encoding unit 51B may encode the left-eye image data and the right-eye image data independently by a known encoding method, but as necessary. You may utilize the correlation between the image data for left eyes, and the image data for right eyes.

  In the first embodiment, the left-eye image data and the right-eye image data output from the three-dimensional imaging device are corrected. As described above, the image data is transmitted or stored in an encoded state. Therefore, the left-eye coded image data and the right-eye coded image data are supplied to the image processing apparatus, and the image processing apparatus may decode and process these data. The second embodiment is an image processing apparatus used in such a case.

  FIG. 8 is a block diagram illustrating a schematic configuration of the image processing apparatus according to the second embodiment. The image processing apparatus according to the second embodiment includes MPEG2 and H.264 from the outside. The encoded image data for the left eye and the encoded image data for the right eye encoded using an encoding method such as H.264 are supplied. The image processing apparatus according to the second embodiment performs correction processing on the encoded image data for the left eye and the encoded image data for the right eye, and then outputs the corrected image data for the left eye and the image data for the right eye. Supply to display devices.

  As shown in FIG. 8, the image processing apparatus of the second embodiment includes a moving image decoding unit 61, a parallax detection / correction unit 62, an image data output unit 64, a control unit 65, a memory controller 66, a memory 67. The image processing apparatus according to the second embodiment has the same configuration as the image processing apparatus according to the first embodiment, except that a moving image decoding unit 61 is provided instead of the image data input unit 21.

  The moving image decoding unit 61 performs a decoding process on the encoded image data for the left eye and the encoded image data for the right eye supplied from the outside by using a corresponding known method. The parallax detection / correction unit 62 detects and corrects the non-parallax portion of the decoded left-eye image data and right-eye image data using the same method as described in the first embodiment. I do.

  The image processing apparatuses according to the first and second embodiments are realized by using, for example, a computer that combines a processor or DSP and a memory.

  FIG. 9 is a functional block diagram of the image processing apparatus 70 of the first and second embodiments realized by a computer.

  As shown in FIG. 9, the image processing apparatus according to the embodiment includes a left-eye image (L image) RAM 71, a right-eye image (R image) RAM 72, a block matching calculation unit 73, an activity calculation unit 74, The determination unit 75, the correction processing unit 76, an internal bus 77, and an input / output interface (Read / Write IF) 78 are included. The parallax detection / determination unit 22 according to the first embodiment corresponds to a part obtained by combining the block matching calculation unit 73 and the determination unit 75, and the correction unit 23 corresponds to a part obtained by combining the activity calculation unit 74 and the correction processing unit 76. To do. The parallax detection / correction unit 62 according to the second embodiment corresponds to the combined portion of the block matching calculation unit 73, the activity calculation unit 74, the determination unit 75, and the correction processing unit 76.

  The L image RAM 71, R image RAM 72, block matching calculation unit 73, activity calculation unit 74, and correction processing unit 76 are connected to each other via an internal bus 77.

  As described above, one of the left-eye image and the right-eye image is used as a standard image, and the other is used as a reference image. For example, when the image for the left eye is used as the standard image and the image for the right eye is used as the reference image, the block of the standard image has N × M pixels, whereas the reference image has a search wider than N × M pixels. Since pixels for a range are required, the R image RAM 72 that stores the right-eye image has a larger capacity than the L image RAM 71 that stores the left-eye image.

  The block matching calculation unit 73 reads out the images from the L image RAM 71 and the R image RAM 72, performs the block matching process in step 13 described above, detects the zero vector error S and the minimum error Smin, and outputs them to the determination unit 75. To do.

  The activity calculation unit 74 is provided when realizing a modification in which correction is performed using the above-described selSR (i, j). The activity calculation unit 74 reads out images from the L image RAM 71 and the R image RAM 72, calculates activities actS and actR, and outputs to the correction processing unit 76 which of the activities actS and actR is higher.

  The determination unit 75 performs the determination process in step S <b> 14 and outputs a determination result indicating whether or not there is parallax to the correction processing unit 76. The correction processing unit 76 reads out images from the L image RAM 71 and the R image RAM 72, performs correction processing based on the determination result from the determination unit 75, and outputs the correction results outS and outR via the input / output interface 78. . In the modification, the correction processing unit 76 performs correction processing based on the determination result of the activity calculation unit 74.

  In a three-dimensional image display, a part without parallax, such as a background part, is often stationary, and encoding noise is more conspicuous than a moving object. By performing image processing using the image processing apparatus according to the first and second embodiments and the modified example, the left and right pixel values are brought close to the same value for a portion without parallax, so Discrepancies in video can be suppressed.

  Although the embodiment has been described above, all examples and conditions described herein are described for the purpose of helping understanding of the concept of the invention applied to the invention and the technology. It is not intended to limit the scope of the invention, and the construction of such examples in the specification does not indicate the advantages and disadvantages of the invention. Although embodiments of the invention have been described in detail, it should be understood that various changes, substitutions and modifications can be made without departing from the spirit and scope of the invention.

Hereinafter, the following additional notes will be disclosed with respect to the embodiment.
(Appendix 1)
One of the image for the left eye and the image for the right eye is a standard image, the other is a reference image,
Dividing the reference image into a plurality of blocks;
Detect parallax evaluation for each divided block,
A block for which the parallax evaluation satisfies a predetermined condition is determined as a parallax-free portion,
A three-dimensional image processing method, wherein the pixel value of the corresponding standard image belonging to the non-parallax portion is corrected so as to be close to the pixel value of the reference image.
(Appendix 2)
The three-dimensional image processing method according to claim 1, wherein in the correction, the pixel value of the corresponding standard image and the pixel value of the reference image are brought close to an average value.
(Appendix 3)
The third order according to supplementary note 1, wherein in the correction, frequency analysis is performed on each block of the reference image and the reference image, and a block containing a large amount of high-frequency components in the reference image and the reference image is selected as a correction image. Original image processing method.
(Appendix 4)
The predetermined condition is that a zero block error value in a block corresponding to the reference image and the reference image is smaller than a first threshold, and a zero block error value and a minimum block error value within a search range by block matching processing are The three-dimensional image processing method according to any one of supplementary notes 1 to 3, which is a condition for determining that there is no parallax when the absolute value of the difference is smaller than a second threshold value.
(Appendix 5)
One of the left-eye image and the right-eye image is used as a standard image, and the other is used as a reference image. Parallax evaluation with the reference image is detected for each block obtained by dividing the standard image, and the parallax evaluation satisfies a predetermined condition A parallax detection / determination unit that determines a block as a non-parallax portion;
A three-dimensional image processing apparatus comprising: a correction unit that corrects the pixel value of the corresponding left-eye image belonging to the non-parallax portion and the pixel value of the right-eye image to be close to each other.
(Appendix 6)
The three-dimensional image processing apparatus according to appendix 5, wherein the correction unit brings the pixel value of the corresponding left-eye image and the pixel value of the right-eye image close to an average value.
(Appendix 7)
The correction unit performs frequency analysis on each block of the left-eye image and the right-eye image, and corrects a block containing a large amount of high-frequency components from the left-eye image and the right-eye image. The three-dimensional image processing apparatus according to appendix 5, which is selected as
(Appendix 8)
The predetermined condition is that a zero block error value in a corresponding block of the left eye image and the right eye image is smaller than a first threshold value, and the minimum block within the search range by the block matching process with the zero block error value The three-dimensional image processing apparatus according to any one of appendices 5 to 7, which is a condition for determining that there is no parallax when the absolute value of the difference from the error value is smaller than the second threshold value.

DESCRIPTION OF SYMBOLS 21 Image data input part 22 Parallax detection / determination part 23 Correction | amendment part 24 Image data output part 25 Control part 26 Memory controller 27 Memory

Claims (2)

One of the image for the left eye and the image for the right eye is a standard image, the other is a reference image,
Dividing the reference image into a plurality of blocks;
The corresponding block is detected by block matching between the reference image and the reference image, the sum of absolute differences of the pixel values of the corresponding block is set as a zero block error value, and the blocks for all blocks within the search range of the block matching Perform a disparity evaluation with the minimum value of the sum of absolute differences of pixel values as the minimum block error value,
The zero-block error value is smaller than the first threshold value, and when the absolute value of the difference between the minimum block error value and the zero block error value is less than the second threshold value, determines that the parallax free portion,
Perform frequency analysis on each block of the corresponding standard image and the reference image belonging to the non-parallax portion, select a block containing a lot of high frequency components from the standard image and the reference image as a correction image, and A three-dimensional image processing method, wherein a pixel value of an image not selected as an image is corrected so as to approach a pixel value of the corrected image. One of the left-eye image and the right-eye image is a standard image and the other is a reference image. For each block obtained by dividing the standard image, a corresponding block is detected by block matching of the standard image and the reference image. A parallax in which the sum of absolute differences of block pixel values is a zero block error value, and a minimum value of the sum of absolute differences of block pixel values for all blocks within the block matching search range is a minimum block error value. evaluated, the zero block error value is smaller than the first threshold value, and when the absolute value of the difference between the minimum block error value and the zero block error value is less than the second threshold value, determines that the parallax free portion A parallax detection / determination unit;
Perform frequency analysis on each block of the corresponding standard image and the reference image belonging to the non-parallax portion, select a block containing a lot of high frequency components from the standard image and the reference image as a correction image, and A three-dimensional image processing apparatus comprising: a correction unit that corrects a pixel value of an image not selected as an image so as to approach the pixel value of the corrected image.

Images