JP2017085598A - Image processing apparatus, image processing method, image processing program, and stereoscopic image display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable stable mapping even in a region with a few textures.SOLUTION: An image processing apparatus according to the embodiment, acquires a plurality of candidate vectors from a plurality of candidate blocks close to a target block, sets one or more particles indicating a position of a correspondence image closed to the plurality of candidate correspondence blocks, arranges a plurality of particle vectors indicating each of particles, calculates a weight on the basis of a difference between a pixel value of the target block on the target image and the pixel value of the block on a reference image regulated by each particle, applies the calculated weight to the plurality of particle vectors, and obtains the correspondence vector regarding to the target block by performing a statistical processing of the plurality of particle vectors on the basis of the weight.SELECTED DRAWING: Figure 1

Description

  Embodiments described herein relate generally to an image processing device, an image processing method, an image processing program, and a stereoscopic image display device.

  2. Description of the Related Art Conventionally, there is a corresponding point detection device that associates two images by searching for a region having a high correlation from one image to another. For example, in Patent Document 1, a vector of a block that has already been calculated in a past frame or space is set as a candidate vector, a search is limited to only a region obtained by adding an update vector to the candidate vector, and recursive processing is performed on the entire screen. There is something.

Japanese Patent No. 3147893

  In the following embodiments, an image processing device, an image processing method, an image processing program, and a stereoscopic image display device that enable stable association even in a region with a small texture are provided.

  An image processing apparatus according to an embodiment is an image processing apparatus that obtains a correspondence vector indicating a correspondence from a target block on a first image to a corresponding block on a second image different from the first image, and An acquisition unit that acquires a plurality of candidate vectors from already calculated corresponding vectors for each of a plurality of candidate blocks in the vicinity of the block, and a plurality of candidate corresponding blocks that each of the acquired candidate vectors points to with respect to the target block One or more particles indicating the position on the second image in the vicinity of each other, an arrangement unit for arranging a plurality of particle vectors indicating the particles with reference to the target block, and the first image on the first image A plurality of pixel values on the second image defined by the pixel value of the target block and each of the particles A weight is calculated based on a difference between each pixel value of each lock, a calculation unit that assigns the calculated weight to each of the plurality of particle vectors, and the weight based on the weight assigned to each of the plurality of particle vectors A processor that statistically processes a plurality of particle vectors to obtain a solution of a corresponding vector related to the target block.

FIG. 1 is a block diagram illustrating a schematic configuration example of an image processing apparatus according to a first embodiment. FIG. 2 is a flowchart showing an operation example of the image processing apparatus shown in FIG. FIG. 3 is a diagram illustrating an example when there are two processed blocks for the target block in the first embodiment. FIG. 4 is a diagram illustrating an example of candidate blocks in the first embodiment. FIG. 5 is a diagram illustrating an example of weights of particle vectors in the first embodiment. FIG. 6 is a diagram illustrating an example of a result of statistical processing in the first embodiment. FIG. 7 is a definition diagram of a problem for obtaining a corresponding block on the reference image having a high correlation with the target block on the target image. FIG. 8 is a diagram obtained by projecting the definition of FIG. 7 onto a two-dimensional plane. FIG. 9 is a diagram illustrating a processing target handled in one process in the first embodiment. FIG. 10 is a diagram showing an example of the scan order of the processing used in step S103 of FIG. FIG. 11 is a diagram showing processed blocks when processing is performed in the scan order shown in FIG. 10. FIG. 12 is a diagram showing examples of temporal candidate blocks, hierarchical candidate blocks, and spatial candidate blocks in the first embodiment. FIG. 13 is a diagram illustrating an example of particle vectors arranged for one candidate vector in the first embodiment. FIG. 14 is a diagram illustrating another arrangement example of the particle vectors according to the first embodiment. FIG. 15 is a diagram illustrating an example of an MSE used in the first embodiment. FIG. 16 is a diagram illustrating an example of a Gaussian function used in the first embodiment. FIG. 17 is a diagram showing an example of another function that can be used instead of the Gaussian function shown in FIG. FIG. 18 is a block diagram illustrating a schematic configuration example of a stereoscopic image display apparatus according to the second embodiment. FIG. 19 is a flowchart illustrating an operation example of the stereoscopic image display apparatus illustrated in FIG. 18. FIG. 20 is a conceptual diagram when generating a multi-parallax image according to the second embodiment.

  Hereinafter, an image processing device, an image processing method, an image processing program, and a stereoscopic image display device according to exemplary embodiments will be described in detail with reference to the drawings. The embodiment illustrated below is for solving the problem of obtaining a partial region having a high correlation with the partial region in the target image from the reference image. For example, this is a problem of obtaining a motion vector from one frame of a moving image to another frame (also referred to as motion estimation), tracking of an object included in the moving image, frame interpolation of the moving image, and stereoscopic video by the naked eye. The present invention can be applied to the problem of obtaining the correspondence of the stereo image to be realized from the left-eye image to the right-eye image.

(Embodiment 1)
First, an image processing apparatus, an image processing method, and an image processing program according to the first embodiment will be described in detail with reference to the drawings. In the first embodiment, a case where a problem (also referred to as motion estimation) for obtaining a motion vector from one frame (target image) of a moving image to another frame (reference image) is illustrated.

  FIG. 1 is a block diagram illustrating a schematic configuration example of the image processing apparatus according to the first embodiment. The image processing apparatus 10 shown in FIG. 1 obtains a correspondence vector V indicating the association from a partial area on the target image 100 to a partial area on the reference image 200 in the motion estimation process on the moving image. In the following description, a partial region including one or more pixels is referred to as a block. In addition, a processing target block on the target image 100 is referred to as a target block, and a block on the reference image 200 that is associated with the target block is referred to as a corresponding block.

  The target image 100 corresponds to a certain frame in the moving image, and the reference image 200 corresponds to another one frame in the moving image. As illustrated in FIG. 1, the image processing apparatus 10 includes a candidate vector acquisition unit 11, a particle vector arrangement unit 12, a particle vector weight calculation unit 13, and a particle vector processing unit 14.

  Candidate vector acquisition unit 11 receives target image 100 from, for example, an external host device. The candidate vector acquisition unit 11 identifies one or more blocks (hereinafter referred to as candidate blocks) in the vicinity of the target block on the input target image 100, and has already calculated corresponding vectors for each of these candidate blocks. To acquire one or more corresponding vectors as candidate vectors (candidate vector acquisition step).

  The particle vector arrangement unit 12 sets one or more particles in the vicinity of each block (hereinafter referred to as a candidate corresponding block) to which each of the one or more candidate vectors acquired by the candidate vector acquisition unit 11 points with the target block as a reference. Then, a particle vector indicating each particle from the target block is arranged (particle vector arranging step). The corresponding vector, candidate vector, and particle vector each start from a predetermined position (for example, the upper left pixel or the center pixel) of the reference block. Hereinafter, for simplification of description, it is simply described as “based on a block”.

  In this description, the particle vector is a vector indicating the position of a particle arranged according to a predetermined rule with respect to an image. Here, the particles are the images pointed to by the candidate vector (reference image 200 or superimposed image 100A of target image 100 and reference image 200 (see FIG. 8): Here, for simplification, reference image 200 is used. And the superimposed image 100A are not distinguished). This particle can be handled as one pixel in the image for processing. In the first embodiment, one or more particles are set around the candidate block indicated by the candidate vector. Therefore, in the first embodiment, one or more particle vectors are arranged for one candidate vector. However, when a plurality of candidate vectors are acquired for the target block, at least one particle vector may be arranged for at least one candidate vector among these candidate vectors.

  The reason why the particle vector to be arranged in the first embodiment is at least one particle vector in the vicinity of the candidate vector other than the position of the candidate vector is that when no particle vector is arranged, the solution does not converge and the corresponding vector This is because there is a possibility that the value of V is always “0”.

  The particle vector weight calculation unit 13 calculates the correlation between the target block and each block specified by each particle, and assigns a weight corresponding to the calculated correlation to each corresponding particle vector (particle vector weight) Calculation step). The correlation may be an index based on pixel values such as brightness, RGB color component values, and color space values such as YUV. In that case, for example, a small difference (difference in pixel values) means a high correlation. It is assumed that the particle is located at a predetermined position (for example, the upper left pixel or the center pixel) in the block, similarly to the start point of the corresponding vector or candidate vector. In other words, the block indicated by the particle vector for which the correlation with the target block is calculated is a block (partial region) defined by the particle. The size (size) of the block defined by the particles may be the same as the target block and the candidate block.

  The particle vector processing unit 14 statistically processes one or more particle vectors according to the weights calculated for each particle vector, thereby obtaining a corresponding vector V related to the target block (particle vector processing step).

  Next, an exemplary operation of the image processing apparatus 10 shown in FIG. 1 will be described in detail with reference to the flowchart of FIG. As illustrated in FIG. 2, the candidate vector acquisition unit 11 downsamples the target image 100 input from the higher-level device by one layer or more (step S <b> 101). As a result, a target image having a hierarchical structure is generated based on the target image 100.

  Next, the candidate vector acquisition unit 11 divides each of the original target image 100 and the target image obtained by downsampling into a plurality of target blocks (step S102). Subsequently, the candidate vector acquisition unit 11 selects one of the plurality of target blocks according to a predetermined scan order (step S103).

  Next, the candidate vector acquisition unit 11 determines whether there is another block for which the corresponding vector has already been calculated (hereinafter referred to as a processed block) in the vicinity of the selected target block (step S104). If the result of this determination is that there is no processed block (step S104; NO), the candidate vector acquisition unit 11 acquires a predetermined vector (for example, a 0 (zero) vector) as a candidate vector for the selected target block. (Step S106). On the other hand, when a processed block exists (step S104; YES), the candidate vector acquisition unit 11 selects at least one of the processed blocks as a candidate block, and one of the selected one or more candidate blocks. A candidate vector acquisition step of acquiring one or more corresponding vectors as candidate vectors from the above corresponding vectors is executed (step S105). For example, FIG. 3 illustrates a case where two processed blocks 102 and 103 exist for the target block 101. In this case, in step S105, as shown in FIG. 4, at least one of these two processed blocks 102 and 103 (two in FIG. 4) is selected as a candidate block, and the correspondence already calculated for these is selected. At least one of the vectors 112 and 113 (two in FIG. 4) is acquired as a candidate vector.

  Next, the particle vector arrangement unit 12 executes a particle vector arrangement step of arranging one or more particle vectors for the selected target block (step S107). Next, the particle vector weight calculation unit 13 executes a particle vector weight calculation step of calculating a weight to be given to one or more arranged particle vectors (step S108). For example, FIG. 5 illustrates a case where the weight 122 that is the probability distribution of the candidate vector 112 is calculated.

  Next, the particle vector processing unit 14 statistically processes one or more particle vectors according to the weight calculated for each particle vector, thereby executing a particle vector processing step for obtaining a corresponding vector for the selected target block. (Step S109). That is, as shown in FIG. 5, the probability distribution 120 as shown in FIG. 6 can be obtained by statistically processing one or more particle vectors according to the weights calculated for each of the one or more candidate vectors 112 and 113. it can. As shown in FIG. 6, the correspondence vector 111 related to the target block 101 can be acquired based on a probability distribution 120 (for example, its center of gravity or center).

  Thereafter, the image processing apparatus 10 determines whether or not the process of acquiring the corresponding vectors for all target blocks has been completed (step S110), and if completed (step S110; YES), ends the operation. . On the other hand, if it has not been completed (step S110; NO), the image processing apparatus 10 returns to step S103, selects the next target block according to a predetermined scan order, and executes the subsequent operations.

Here, if the coordinates of each pixel based on a certain pixel (for example, the lower left pixel) in an image having a hierarchical structure is (X, h) ^T = (x, y, h) ^T , one image (for example, The pixel value of each pixel on the target image 100) can be described as A _o (X, h), and the pixel value of each pixel on the other image (for example, the reference image 200) can be described as A _r (X, h). . T represents transposition of a matrix vector. The column vector is shown in capital letters. These pixel values A _o (X, h) and A _r (X, h) can also be expressed as A _o (x, y, h) and A _r (x, y, h), respectively.

  H represents a hierarchy. A hierarchy is a pyramid hierarchy of images. As the most basic pyramid hierarchy, for example, there is a Gaussian pyramid represented by the following formula (1).

  In Expression (1), the upward arrow represents downsampling, and G () represents a Gaussian filter. That is, the Gaussian pyramid refers to a pyramid hierarchy including one or more images obtained by downsampling while smoothing an image with a Gaussian filter and the original image. Therefore, h = 0 indicates an original image, and h = 1 indicates an image obtained by performing downsampling once on the original image.

  Further, in the following description, the downsampling scale, that is, the fraction of downsampling with respect to the original image is represented by ρ. For example, if the original image is down-sampled to half the length and breadth, ρ = 2.

  The first embodiment is not limited to the one using a pyramid hierarchy. That is, it may be configured to use only h = 0.

  The association between the target block on the target image 100 and the corresponding block 201 on the reference image 200 having a high correlation with the target block 101 is defined using a corresponding vector. Here, the problem of obtaining the corresponding block 201 on the reference image 200 having a high correlation with the target block 101 on the target image 100 can be defined as shown in FIG. In FIG. 7, a vector 111 is a vector indicating from the standard pixel in the target block 101 in the target image 100 to the corresponding pixel in the corresponding block 201 on the reference image 200 having a high correlation with the target block 101. When the relationship shown in FIG. 7 is projected onto a two-dimensional plane, it is as shown in FIG. In this description, a vector 111 that points from the base pixel of the target block 101 to the corresponding pixel of the corresponding block 201 in the superimposed image 100A of the target image 100 and the reference image 200 is defined as a corresponding vector. This is the same definition used for calculating motion vectors for moving images and corresponding points in stereo matching for multi-parallax images.

Here, when the image has a pyramid hierarchy, it is assumed that the corresponding vector V (see FIG. 1) also has the same number of pyramid hierarchies as the image. In that case, the corresponding vector V (X, h) of the hierarchy h at the coordinate X can be described by the following equation (2).

Further, when the time t given to the image is further taken into consideration, that is, when the processing is executed in consideration of the time series, the corresponding vector V (X, h, t) can be described by the following equation (3). it can.

  Note that when processing a moving image, the target image 100 and the reference image 200 change at different times t. However, when processing a still image, the target image 100 and the reference image 200 are different even at different times t. does not change.

  As shown in FIG. 9, the processing target handled in one process is a block unit in which one or more pixels are collected. These blocks may be obtained by dividing one image by a predetermined number of divisions in the vertical direction and the horizontal direction, or a size in which the number of pixels in the vertical and horizontal directions is determined in advance for the image. May be obtained by filling (arranging) these sections from a predetermined position (for example, upper left).

If the block is filled from the upper left of the image, the horizontal and vertical sizes of each block are (B _x , B _y ), and the block to be processed is I = (i, j) ^T , then I coordinates X _L of the upper left in the block can be represented by the following formula (4).

The center coordinates X _C of I blocks can be represented by the following formula (5).

Here, the operator c (I) for calculating the center coordinates of the block can be defined as the following equation (6). The operator c (I) can also be described as c (i, j).

<Details of each step>
The details of each step shown in FIG. 2 will be described below in detail with reference to the drawings in accordance with the above definitions.

Step S103: Process Scan Order FIG. 10 is a diagram showing an example of the process scan order used in step S103 of FIG. FIG. 11 shows processed blocks when processed in the scan order shown in FIG. In the example shown in FIGS. 10 and 11, the original target image at time t−1 is 100 (t−1), and the original target image at time t is 100 t. An image obtained by down-sampling the target image 100 (t−1) once at ρ = 2 is set as the target image 10 (t−1), and an image obtained by down-sampling the target image 100t once at ρ = 2 is set as the target image 10t. It is said.

  As shown in the example shown in FIG. 10, for example, in the scan order, blocks are selected as needed in the order of time from the past to the future and hierarchy from top to bottom. Further, on the same plane (image), for example, it is conceivable to perform processing from the upper left to the lower right. However, the present invention is not limited to these, and various modifications are possible such as changing the time from the future to the past, the hierarchy from the lower layer to the upper layer, or the selection order on the same plane from the lower right to the upper left. In the following description, as shown in FIG. 10, the scanning order in the same plane is from the upper left to the lower right, and the block on the upper side and on the left is spatially higher.

  Further, by selecting the blocks in the scan order shown in FIG. 10, as shown in FIG. 11, blocks that are temporally past and hierarchically and spatially higher than the target block to be processed are already present. This is the processed block for which the correspondence vector has been calculated.

Step S105: Candidate Vector Acquisition Step The candidate vector acquisition step shown in step S105 of FIG. 2 will be described. In the candidate vector acquisition step, a correspondence vector of a block located in the temporally, hierarchically, and spatially vicinity of the target block among the processed blocks is acquired as a candidate vector. FIG. 12 shows an example in which correspondence vectors of processed blocks (temporal candidate blocks, hierarchical candidate blocks, spatial candidate blocks) that are temporally, hierarchically or spatially adjacent to the target block are acquired as candidate vectors. Indicates. However, FIG. 12 is merely an example, and any processed block located in the vicinity of the target block may be used. A block located in the vicinity is not limited to a block that is temporally, hierarchically, or spatially adjacent to the target block, and is located at an oblique position temporally, hierarchically, or spatially with respect to the target block. A block that is located second or third by tracing an adjacent block from a certain block or target block may be included.

In the example shown in FIG. 12, a set of candidate vectors (candidate vector group) as shown in the following formula (7) is acquired.

Here, since the candidate vector group is a set, this is defined as a candidate vector set W as shown in the following equation (8).

  Here, all of the temporal element, the hierarchical element, and the spatial element are used as candidates, but the present invention is not limited to this. That is, at least one element among temporal elements, hierarchical elements, and spatial elements may be used as a candidate.

Step S107: Particle Vector Placement Step The particle vector placement step shown in step S107 in FIG. 2 will be described. In the particle vector arrangement step, as described above, one or more particles are set in the vicinity of each of one or more candidate corresponding blocks each of which one or more candidate vectors indicate with reference to the target block, and one or more particles A particle vector based on the target block is arranged for each. In other words, one or more particle vectors indicating one or more particles in the vicinity of the candidate corresponding block indicated by each candidate vector are arranged in the vicinity of each candidate vector.

  At least one particle vector needs to be arranged in the vicinity of the candidate vector so as not to overlap the candidate vector. However, it is not always necessary to arrange one or more particle vectors for all candidate vectors, and it is sufficient that at least one particle vector is arranged in the vicinity of at least one candidate vector.

  FIG. 13 shows an example of particle vectors arranged for one candidate vector. In the example shown in FIG. 13, one or more (four in FIG. 13) particles 131 belonging to a circle with a predetermined radius (for example, 1) are set around the point 142 pointed to by the candidate vector 112, and the particle vector pointing to them is set. An example in which 132 is arranged with respect to the candidate vector 112 is shown. Note that the point 142 may be the same as the particle 131.

A particle 131 (including a point 142) in a circle having a radius of 1 can be represented as a neighborhood set N as shown in the following equation (9).

Here, ‖ 長 represents the length. Therefore, as shown in the following formula (10), when n is an integer 1, the number of particles 131 (including the point 142) belonging to the neighborhood set N is five.

In this case, assuming that the candidate vector 112 is wεW, the particle vector p (132) is expressed by the following equation (11), and the particle vector set P in which all the particle vectors p (132) are collected is It can be expressed as equation (12). Here, ∀ represents all elements belonging to the set.

  The particle vector 132 set for each candidate vector 112 is not limited to the example shown in FIG. For example, as shown in FIG. 14A, a total of eight particles surrounding the point 142 pointed to by the candidate vector 112 may be set, the particle 131 may be set for each, and the particle vector 132 pointing to these may be arranged. Further, as shown in FIG. 14B, for example, a total of 12 particles 131 belonging to a circle having a radius of 2 may be set around the point 142 indicated by the candidate vector 112, and the particle vector 132 indicating them may be arranged. Good. Alternatively, as shown in FIG. 14C, the particle vector 132 is arranged for one or more particles 131 randomly set in a circle having a predetermined radius centered on the point 142 indicated by the candidate vector 112. Also good. For convenience of explanation, the particle vector 132 is not shown in FIG.

Step S108: Particle Vector Weight Calculation Step The particle vector weight calculation step shown in step S108 in FIG. 2 will be described. In the particle vector weight calculation step, weights are calculated for all particle vectors. A displacement block difference (DBD) can be used as the basis for calculating the weight for each particle vector. As one of the displacement block difference DBDs, there is an MSE (Mean Squared Error) as shown in FIG.

Here, if the particle vector is pεP, the MSE can be defined as the following equation (13).

According to the equation (13), the average of the square difference between the pixel value of the target image in the block I = (i, j) ^T and the pixel value of the reference image in the block displaced by the particle vector is obtained. it can.

In addition to the MSE, there is an absolute value difference average MAD (Mean Absolute Difference). Similarly, when the particle vector is pεP, MAD can be defined as the following equation (14).

  Next, processing for calculating the weight of each particle vector using MSE or MAD will be described. The displacement block difference DBD has a smaller error as the value is smaller in both the case where the MSE is used and the case where the MAD is used. This means that the particle vector is close to the true corresponding vector. Therefore, the weight of each particle vector should be such that the smaller the displacement block difference DBD is, the larger the weight is, and the larger the weight is, the smaller the weight is.

As a function (weight function) for giving such weights, for example, a Gaussian function as shown in FIG. 16 can be used. In the case of MSE, a weight using a Gaussian function can be calculated as in the following equation (15).

On the other hand, in the case of MAD, it can be calculated as in the following equation (16).

  Here, σ represents the standard deviation of the Gaussian function, that is, the noise intensity. Therefore, if the image has a lot of noise or the images to be associated are not similar, this value σ is increased. This may be set in advance according to the target image or the like, or may be set according to the input image.

Further, a function as shown in FIG. 17 may be used. In this case, the weight using the MSE can be calculated as in the following equation (17).

On the other hand, the weight using the MAD can be calculated as in the following equation (18).

  Here, ε is a constant for preventing '0 (zero)' division. An appropriate constant such as 0.1 or 1.0 may be set in advance for ε. As a characteristic, the larger the ε, the smoother the weight function, and the smaller the ε, the steeper weight function.

Step S109: Particle Vector Processing Step In the particle vector processing step shown in Step S109 of FIG. 2, the corresponding vector of the target block is obtained by performing statistical processing according to the weight corresponding to the particle vector.

In the statistical processing, for example, a weighted average as shown in the following equation (19) may be used.

The correspondence vector obtained using Expression (19) is a correspondence vector represented by the target block. Therefore, these are assigned as corresponding vectors in the target block as shown in the following equation (20).

Further, a weighted median (median value) shown in the following equation (21) may be used instead of the weighted average shown in equation (19). In equation (21), arg min _x E (where x is below arg min) indicates the value of x that minimizes E.

  By executing the above steps recursively according to the flow shown in FIG. 2, it is possible to stably acquire the corresponding vector even in a region with a small texture.

  For example, in Patent Document 1, a vector of a block that has already been calculated in a past frame or space is set as a candidate vector, a search is limited to only a region obtained by adding an update vector to the candidate vector, and recursive processing is performed on the entire screen. Thereby, it is possible to stably perform association with a small amount of calculation. This is based on the grounds that assuming that the object has a certain size or larger, the solution vector of the target block is given by spatially adjacent blocks or temporally adjacent blocks. ing. By performing these processes recursively throughout the entire screen, it is possible to associate a large movement with a small amount of calculation.

  However, in Patent Document 1, since a solution is selectively selected on the basis of a small number of candidate vectors, there is a problem that the association is likely to be unstable in a region having a small texture.

  Therefore, in the first embodiment, as described above, a particle vector is arranged in the vicinity of a candidate vector, a weight (also referred to as probability density) of each particle vector is calculated, and statistical processing is performed according to each weight to obtain a corresponding vector. This makes it possible to estimate a true corresponding vector based on a statistical distribution even if a single solution is not determined stably in a region with less texture. It becomes possible.

(Embodiment 2)
Next, an image processing device, an image processing method, an image processing program, and a stereoscopic image display device according to Embodiment 2 will be described in detail with reference to the drawings. The second embodiment exemplifies a case where the problem of obtaining a correspondence from a left-eye image to a right-eye image of a stereo image forming a stereoscopic video is solved. Moreover, in the following description, about the structure and operation | movement similar to embodiment mentioned above, the overlapping description is abbreviate | omitted by citing it. In addition, since the target image 100 and the reference image 200 in the first embodiment can be a left-eye image and a right-eye image in the second embodiment, they will be described as the left-eye image 100 and the right-eye image 200 in the following description. However, the present invention is not limited to this, and the right eye image and the left eye image may be interchanged. In the following description, the corresponding vector in the first embodiment is replaced with a disparity vector.

  FIG. 18 is a block diagram illustrating a schematic configuration example of the stereoscopic image display apparatus 20 according to the second embodiment. A stereoscopic image display device 20 shown in FIG. 18 generates a multi-parallax image used by the naked eye 3D or the like from a stereo image. As is apparent from a comparison between FIG. 18 and FIG. 1, the stereoscopic image display device 20 has a configuration similar to that of the image processing device 10 and a multi-parallax image based on the corresponding vector V output from the particle vector processing unit 14. Is further provided with a parallax image generation unit 21 that generates the image and a multi-parallax image display unit 22 that displays the multi-parallax image generated by the parallax image generation unit 21.

  FIG. 19 shows an operation example of the stereoscopic image display apparatus 20 shown in FIG. As is clear when FIG. 19 is compared with FIG. 2, the stereoscopic image display device 20 performs the same operation (steps S101 to S110) as the operation illustrated in FIG. 2 to calculate a disparity vector for the target block. Thereafter, the parallax image generation unit 21 generates parallax images from the left-eye image 100 and the parallax vector V for the number of parallax images desired to be generated (step S201). As a result, a multi-parallax image used by the naked eye 3D or the like is generated.

Here, as shown in FIG. 20, assuming that the left-eye image 100 is obtained from an intermediate viewpoint between the left eye and the right eye, each parallax number is represented by O at the center of both eyes and minus on the right side and plus k on the left side. can do. In that case, a multi-parallax image can be generated using a formula (22) from a parallax vector V20 obtained by multiplying the parallax vector d by k.

Each parallax image can be generated by moving the pixel value I _o (x) of the left eye image 100 according to d _k . However, there is a possibility that a hole will be formed simply by moving. In that case, the image may be filled by interpolating the hole area from the peripheral parallax vector V. Since other configurations, operations, and effects are the same as those of the first embodiment, detailed description thereof is omitted here.

  In addition, the above-described embodiment and its modifications are merely examples for carrying out the present invention, and the present invention is not limited to these, and various modifications according to specifications and the like are within the scope of the present invention. Furthermore, it is obvious from the above description that various other embodiments are possible within the scope of the present invention. For example, it is needless to say that the modification examples illustrated as appropriate for each embodiment can be applied to other embodiments.

DESCRIPTION OF SYMBOLS 10 Image processing apparatus 11 Candidate vector acquisition part 12 Particle vector arrangement | positioning part 13 Particle vector weight calculation part 14 Particle vector processing part 20 Stereo image display apparatus 21 Parallax image generation part 22 Multi parallax image display part 100 Target image (left eye image)
100A superimposed image 101 target block 102, 103 processed block 111 corresponding vector 112, 113 candidate vector 120 probability distribution 122 weight 131 particle 132 particle vector 142 point 200 reference image (right eye image)

Claims (13)

An image processing apparatus for obtaining a correspondence vector indicating a correspondence from a target block on a first image to a corresponding block on a second image different from the first image,
An acquisition unit for acquiring a plurality of candidate vectors from corresponding vectors already calculated for each of a plurality of candidate blocks in the vicinity of the target block;
One or more particles indicating positions on the second image are set in the vicinity of each of the plurality of candidate corresponding blocks each of which the obtained candidate vectors point to with respect to the target block, and the target block is set as a reference An arrangement unit for arranging a plurality of particle vectors indicating the particles;
A weight is calculated based on a difference between a pixel value of the target block on the first image and a pixel value of each of a plurality of blocks on the second image defined by each of the particles, and the calculated weight is A calculation unit for assigning each of the plurality of particle vectors;
An image processing apparatus comprising: a processor that statistically processes the plurality of particle vectors based on the weights assigned to each of the plurality of particle vectors to obtain a solution of a corresponding vector related to the target block. The calculation unit is configured based on the square difference average MSE obtained from the pixel value of the target block and the pixel value of each block defined by each of the one or more particles using the following formula (1). The image processing apparatus according to claim 1, wherein the weight given to each of the two or more particle vectors is calculated.

3. The image processing according to claim 2, wherein the calculation unit calculates the weight to be given to each of the one or more particle vectors by using a Gaussian function g shown in the following formula (2) in which σ is a standard deviation. apparatus.

The image processing apparatus according to claim 2, wherein the calculation unit calculates the weight to be given to each of the one or more particle vectors using a function represented by the following expression (3), where ε is a constant.

The calculation unit, based on the absolute value difference average MAD obtained from the pixel value of the target block and the pixel value of each block defined by each of the one or more particles using the following formula (4), The image processing apparatus according to claim 1, wherein the weight given to each of the one or more particle vectors is calculated.

The image processing according to claim 5, wherein the calculation unit calculates the weight to be given to each of the one or more particle vectors by using a Gaussian function g shown in the following formula (5) where σ is a standard deviation. apparatus.

The image processing apparatus according to claim 5, wherein the calculation unit calculates the weight to be given to each of the one or more particle vectors using a function represented by the following formula (6), where ε is a constant.

  The image processing device according to claim 1, wherein at least one of the particle vectors indicates the particle at a position different from the candidate corresponding block indicated by the candidate vector.   The image processing apparatus according to claim 1, further comprising: a generation unit that generates one or more parallax images from the first image using the correspondence vector obtained by the processing unit.   The image processing apparatus according to claim 1, wherein the plurality of candidate correspondence blocks include one or more of a spatial neighborhood area, a past neighborhood area, and a hierarchical neighborhood area of the target block. An image processing method for obtaining a correspondence vector indicating a correspondence from a target block on a first image to a corresponding block on a second image different from the first image,
Obtaining a plurality of candidate vectors from corresponding vectors already calculated for each of a plurality of candidate blocks in the vicinity of the target block;
One or more particles each indicating a position on the second image are set in the vicinity of each candidate corresponding block to which each of the obtained candidate vectors points to the target block as a reference;
Arranging a plurality of particle vectors indicating each of the particles with respect to the target block;
Calculating a weight based on a difference between a pixel value of the target block on the first image and a pixel value of each of the plurality of blocks on the second image defined by each of the particles;
Assign each of the calculated weights to the plurality of particle vectors;
An image processing method comprising: obtaining a solution of a corresponding vector related to the target block by statistically processing the plurality of particle vectors based on the weights assigned to the plurality of particle vectors. An image processing program for causing a computer to obtain a correspondence vector indicating a correspondence from a target block on a first image to a corresponding block on a second image different from the first image,
A process of obtaining a plurality of candidate vectors from corresponding vectors already calculated for each of a plurality of candidate blocks in the vicinity of the target block;
One or more particles each indicating a position on the second image are set in the vicinity of each candidate corresponding block in which each of the acquired candidate vectors points to the target block as a reference, and the particle is set based on the target block. A process of arranging a plurality of particle vectors pointing to each,
A weight is calculated based on a difference between a pixel value of the target block on the first image and a pixel value of each of a plurality of blocks on the second image defined by each of the particles, and the calculated weight is A process of assigning each of the plurality of particle vectors;
An image processing program for causing the computer to execute processing for obtaining a solution of a corresponding vector related to the target block by statistically processing the plurality of particle vectors based on the weights assigned to each of the plurality of particle vectors . A stereoscopic image display device for obtaining a disparity vector indicating a correspondence from a target block on a first image to a corresponding block on a second image different from the first image,
An acquisition unit that acquires a plurality of candidate vectors from disparity vectors that have already been calculated for each of a plurality of candidate blocks in the vicinity of the target block;
One or more particles each indicating a position on the second image are set in the vicinity of each candidate corresponding block in which each of the acquired candidate vectors points to the target block as a reference, and the particle is set based on the target block. An arrangement part for arranging a plurality of particle vectors pointing to each;
A weight is calculated based on a difference between a pixel value of the target block on the first image and a pixel value of each of a plurality of blocks on the second image defined by each of the particles, and the calculated weight is A calculation unit for assigning each of the plurality of particle vectors;
A processor that statistically processes the plurality of particle vectors based on the weights assigned to each of the one or more particle vectors to obtain a parallax vector solution for the target block;
A generation unit that generates one or more parallax images from the first image using the parallax vector obtained by the processing unit;
A stereoscopic image display device comprising: a display unit that displays the one or more parallax images generated by the generation unit.

Images