JP2019106215A - Data processor, imaging device, and data processing method - Google Patents

Abstract

To correctly acquire the reliability of information having an error.SOLUTION: A data processor for calculating reliability data representing reliability to pixel value of at least a portion of pixels of second image data from first image data and the second image data includes a similarity acquisition part for acquiring similarity between the pixel value of a first pixel of the first image data corresponding to a target pixel of the reliability data and the pixel value of each of a plurality of second pixels in a prescribed peripheral area of the first pixel, and a reliability acquisition part for acquiring a reliability being the pixel value of the target pixel of the reliability data from the similarity about each of the plurality of second pixels, the pixel value of a third pixel of the second image data corresponding to the first pixel, and the pixel value of each of plurality of fourth pixels located in the prescribed peripheral area of the third pixel to correspond to each of the plurality of second pixels.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a data processing device, an imaging device, and a data processing method.

  There has been proposed a method for acquiring or calculating a distance image or motion image representing the distribution of distance information or motion information from a captured image. Here, the distance information and motion information acquired from the photographed image often include an error. For example, when distance information or motion information is obtained by acquiring the correspondence between two images by the template matching technique, a large error may occur at the boundary of objects in the image. Such an error occurs when one template contains objects with different distances or movements. The obtained distance information and motion information are often false information having an intermediate value of the distance and motion of each object. The size of the area having an error depends on the size of the template. Of course, even when distance information or motion information is obtained by a method other than template matching technology, similar errors may occur at object boundaries.

  There are the following methods for correcting the information of each pixel of the distance image and the motion image including an error.

  In Patent Document 1, the reliability of distance information of each pixel is calculated, and the distance information is corrected using the reliability. More specifically, the reliability of the distance information is calculated based on the magnitude of the luminance value, the magnitude of the luminance change, the frequency characteristic, or the motion information in the captured image. Therefore, it is possible to reduce the reliability of a place having no texture or a place having a large movement. However, in the case where each object at the object boundary has texture and movement is small, the reliability is also calculated largely at the object boundary. That is, in the method of Patent Document 1, the reliability is calculated without considering the object boundary.

  In Patent Document 2, correction of distance information is performed as follows. First, clustering is performed based on the pixel value of a captured image or the depth value of a distance image to classify pixels into a plurality of classes. The class of the correction target pixel is determined using the pixel value statistics (average value and the like) of each class and the correction target pixel and the pixel values in the vicinity thereof. Then, the depth value of the correction target pixel is replaced with the representative depth value of the class (such as the average value of the depth values in the class). Such processing can perform correction in consideration of spatial continuity of pixel values. However, when the depth value in the class includes an error, the representative depth value of the class deviates from the correct depth value, and sufficient correction can not be performed.

  In Non-Patent Document 1, a distance image is corrected by a weighted cross bilateral filter using distance information, luminance information of a captured image, and reliability information derived therefrom. That is, peripheral pixels having a large difference between the distance value of the correction target pixel and the luminance value are not used for the correction processing as being unreliable. However, the reliability calculated in this manner is not the reliability of the distance information itself, but only the relative reliability between pixels. Further, in this method, since the correction target pixel including an error has a large distance difference from the peripheral non-error pixel, the correct distance information not including the peripheral error is not used for the correction process. .

JP, 2013-239119, A JP, 2012-078942, A

Yuya Matsuo and 2 others, "Improving the accuracy of depth estimation with weighted cross bilateral filters," Journal of the Institute of Image Information and Television Engineers Vol. 66, no. 11, pp. J434-J443 (2012)

  As described above, in any prior art, it is not possible to accurately correct information including an error at an object boundary. This is because the degree of reliability of each pixel (how much error is included) can not be evaluated correctly. In order to properly perform the correction, it is necessary to obtain the reliability of the information itself indicating how reliable the information including the error caused by the boundary of the object is.

  An object of the present invention is to provide a data processing apparatus that generates reliability information that makes pixels with errors due to object boundaries less reliable and makes other correct pixels more reliable.

According to a first aspect of the present invention, there is provided data calculating reliability data representing reliability of pixel values of at least a part of pixels of the second image data from the first image data and the second image data. A processing device,
A pixel value of a first pixel of the first image data corresponding to a target pixel of the reliability data, and a pixel value of each of a plurality of second pixels in a predetermined peripheral region of the first pixel; A similarity acquisition unit for acquiring the similarity of
The degree of similarity for each of the plurality of second pixels, the pixel value of a third pixel of the second image data corresponding to the first pixel, and the predetermined peripheral region of the third pixel A reliability acquisition unit for acquiring a reliability, which is a pixel value of a pixel of interest of the reliability data, from the pixel values of the plurality of fourth pixels corresponding to the plurality of second pixels and located inside the When,
And the like.

According to a second aspect of the present invention, there is provided data calculating reliability data representing the reliability of the pixel value of at least a part of pixels of the second image data from the first image data and the second image data. A processing device,
A correction unit that corrects the second image data based on the first image data and the second image data, and generates a second image data after correction;
A reliability acquisition unit that acquires a reliability by comparing the second image data after correction and the second image data before correction;
And the like.

A third aspect of the present invention calculates reliability data representing reliability of pixel values of at least a part of pixels of the second image data from the first image data and the second image data. A data processing method performed by the data processing device,
A pixel value of a first pixel of the first image data corresponding to a target pixel of the reliability data, and a pixel value of each of a plurality of second pixels in a predetermined peripheral region of the first pixel; Similarity step to obtain the similarity of
The degree of similarity for each of the plurality of second pixels, the pixel value of a third pixel of the second image data corresponding to the first pixel, and the predetermined peripheral region of the third pixel A reliability acquisition step of acquiring the reliability, which is the pixel value of the pixel of interest of the reliability data, from the pixel values of each of the plurality of fourth pixels corresponding to the plurality of second pixels located within When,
It is characterized by including.

A fourth aspect of the present invention calculates reliability data representing the reliability of the pixel value of at least a part of pixels of the second image data from the first image data and the second image data. A data processing method performed by the data processing device,
Correcting the second image data based on the first image data and the second image data, and generating a second image data after the correction;
A reliability step of acquiring reliability by comparing the second image data after correction and the second image data before correction;
It is characterized by including.

  According to the present invention, it is possible to correctly set the reliability of information having an error.

1 is a block diagram showing an example of an imaging device including a data processing device according to Embodiment 1. FIG. 3 is a flowchart of an imaging processing method and a data processing method according to the first embodiment. FIG. 6 is a diagram for explaining reliability calculation and image correction in the first embodiment. FIG. 6 is a diagram for explaining an error included in distance image data according to the first embodiment. FIG. 7 is a diagram for explaining calculation of reliability in the first embodiment. FIG. 7 is a view for explaining calculation of reliability in the case where repetitive processing is performed in the first embodiment. FIG. 8 is a view for explaining reliability calculation and image correction according to a modification of the first embodiment. FIG. 7 is a block diagram showing an example of a data processing apparatus according to a second embodiment. FIG. 14 is a view for explaining an error included in distance image data in the third embodiment. FIG. 14 is a view for explaining calculation of reliability in the third embodiment.

  Embodiments of the present invention will be described in detail with reference to the drawings, but the present invention is not limited to the configurations of the respective embodiments. Also, the embodiments may be combined as appropriate.

(Embodiment 1)
FIG. 1 schematically shows the configuration of an imaging device according to Embodiment 1 of the present invention. The imaging apparatus 1 includes an imaging optical system 10, an imaging element 11, a control unit 12, a data processing device 13, a storage unit 14, an input unit 15, and a display unit 16. FIG. 2A is a flow chart showing processing from shooting processing to output / recording processing of a shot image. Each configuration of the imaging device 1 will be described with reference to the flowchart of FIG. 2A.

  The imaging optical system 10 includes a plurality of lenses, and focuses incident light on the image plane of the imaging element 11. The imaging device 11 includes an image sensor such as a CCD or a CMOS. The image sensor may have a color filter, may not have a color filter, and may be a three-plate type. The imaging device 1 performs imaging processing S <b> 20 by acquiring a signal of each pixel of the imaging element 11.

The data processing device 13 includes a signal processing unit 130, a memory 131, a distance map generation unit 132, a reliability data calculation unit 133, and a distance map correction unit 134. The signal processing unit 130 is a functional unit that performs the image processing S21. The image processing S21 includes various signal processing such as AD conversion, noise removal, demosaicing, luminance signal conversion, aberration correction, white balance adjustment, color correction and the like of an analog signal output from the imaging device 11. Digital image data output from the signal processing unit 130 is stored in the memory 131 and used for display on the display unit 16, recording (storage) in the storage unit 14, calculation of distance information, generation of distance image data, etc. Ru.

  The distance map generation unit 132 is a functional unit that performs distance generation processing S22. The distance map generation unit 132 acquires digital image data (hereinafter, also simply referred to as photographed image data) of the photographed image output from the signal processing unit 130, and generates a distance map representing distance information of the subject from the photographed image data. Do. The distance map is data consisting of a plurality of distance information, and can be considered as an image having a distance value as a pixel value, and hence it is also referred to as a distance image or distance image data below. As a method of acquiring distance information of the subject, a method (Depth From Defocus method: DFD method) using photographed image data of different blurs photographed by changing photographing conditions or a method (stereo method) using photographed image data having different parallax Can be mentioned. Other methods include Time of Flight method and Depth From Focus method. The distance image data generated by the distance map generation unit 132 is stored in the storage unit 14 or temporarily stored in the memory 131, and is used for processing in the subsequent stage.

  The distance information may be a relative distance from the focus position or an absolute distance from the imaging device at the time of shooting. When obtaining distance information from two images, the relative distance from the focus position may be a relative distance from the middle position of the focus positions of the two images to the subject, or the relative distance from the focus position of any one image to the subject It may be a long distance. Also, the absolute distance or relative distance may be either the distance on the image plane side or the distance on the object side. Further, the distance may be represented by the distance in the real space, or may be represented by an amount that can be converted to the distance in the real space, such as the defocus amount and the parallax amount.

  The reliability data calculation unit 133 has a function of calculating reliability data representing the reliability of each pixel value (distance value) of the distance map generated by the distance map generation unit 132. The reliability data is information serving as an index as to whether the distance information of each pixel of the distance map is a correct value. The reliability data calculation unit 133 does not have to obtain the reliability for all the pixels of the distance map, and may calculate the reliability for only some of the pixels. As shown in FIG. 3, the reliability data calculation unit 133 calculates the reliability data from the statistics of peripheral pixels and the statistics using the similarity, and the similarity acquisition unit 1331 which calculates the similarity of luminance values. And a reliability acquisition unit 1332. The reliability acquisition unit 1332 includes a statistic acquisition unit 1333 that calculates a statistic, and a conversion unit 1334 that converts the statistic to a reliability.

  The distance map correction unit 134 uses the acquired captured image data, the distance image data generated by the distance map generation unit 132, and the reliability information generated by the reliability data calculation unit 133 to obtain the distance image data. It has a function of correcting the distance information of each pixel.

  The processing performed by the reliability data calculation unit 133 and the distance map correction unit 134 corresponds to the data processing S23 in FIG. 2A. Details of the data processing S23 will be described later.

The storage unit 14 is a non-volatile storage medium in which photographed image data, distance image data, reliability information data, corrected distance image data, parameter data used by the imaging device 1 and the like are stored. The storage unit 14 may be any storage medium that can read and write at high speed and has a large capacity. For example, the storage unit 14 preferably includes a flash memory or the like. The input unit 15 is an interface operated by the user to input information to the imaging apparatus 1 and change settings. For example, the input unit 15 can include a dial, a button, a switch, a touch panel, and the like. The display unit 16 is a display unit configured by a liquid crystal display, an organic EL display, or the like. The display unit 16 is used for composition confirmation at the time of photographing, browsing of photographed and recorded images, display of various setting screens and message information, and the like. The output / recording processing S24 includes recording processing of the photographed image data, distance image data, reliability image data, corrected distance image data, etc. in the storage unit 14 and display processing of these data on the display unit 16 .

  The control unit 12 is a function of controlling each unit of the imaging device 1. The functions of the control unit 12 include, for example, automatic focusing by autofocus (AF), change of focus position, change of F value (aperture), capture of image, control of shutter and flash (all not shown), storage There is control of the unit 14, the input unit 15 and the display unit 16.

  Next, the main errors appearing on the distance image data will be described. The first is an error that occurs at a point where distance information changes significantly (referred to as an object boundary). This type of error appears in the vicinity of the boundary where objects overlap on the front and back sides. When a method is used to calculate the distance from image similarity, such as the DFD method, the error in the calculated distance information is calculated at the object boundary because the information on the front and back sides is mixed at the time of distance calculation. growing.

  The second is an error (data loss) in which distance information can not be obtained. The cause of this error depends on the distance acquisition method. Taking the stereo method as an example, there are a region where there is no pattern (texture) in the subject, a dark region, and a region (occlusion region) which can not be seen from one side due to a difference in parallax. Also, this type of error often occurs in an area including a plurality of consecutive pixels.

  The third is an error caused by noise in distance information. When noise occurs, even if an object is at the same distance, distance information in the object is dispersed.

  The data processing S23 will be described in more detail below. In the data processing S23 of the present embodiment, among the above-mentioned errors, the reliability obtained by mainly evaluating the presence or absence of an error at the first distance boundary (object boundary) is calculated, and distance image data is calculated based on the reliability. Correct the Hereinafter, the data processing S23 will be described in more detail with reference to the drawings. FIG. 2B is a more detailed flowchart of data processing S23. FIG. 3 is a diagram showing functional blocks of the reliability data calculation unit 133 and the distance map correction unit 134 together with the flow of data. 4A to 4C are diagrams showing examples of processing data.

  In step S30, the reliability data calculation unit 133 acquires the photographed image data 41 (first image data) and the distance image data 42 (second image data). FIG. 4A shows an example of the photographed image data 41, and FIG. 4B shows an example of the distance image data. The photographed image data 41 and the distance image data 42 are images of approximately the same viewpoint. For example, the distance image data 42 is image data including distance information calculated from a plurality of photographed image data of different photographing conditions, and the photographed image data 41 is any of a plurality of photographed image data based on the distance information calculation. Or image data obtained by combining them. The photographed image data 41 may be a monochrome image (luminance image) or a color image.

As described above, the distance image data 42 contains an error at the object boundary. The distance information of the object boundary portion (the AA ′ portion in FIG. 4B) of the distance image data 42 is an object (human body) on the front side of the distance information of the rear object (background) near the object boundary as shown in FIG. Includes errors under the influence of distance information. More specifically, it includes an error in which distance information of the rear object is calculated closer than actual. This error occurs during the distance map generation process. The impact of errors by the front object diminishes as one moves away from the object boundary. If an area in which the distance information differs from the correct distance value is defined as an error area, in the present embodiment, the area of the rear object within a predetermined range from the object boundary becomes the error area. As shown in FIG. 4D, the reliability data calculation unit 133 performs processing to obtain reliability information for correctly extracting an error area. More specifically, the reliability data calculation unit 133 has reliability information that is low reliability (displayed in black in FIG. 4D) in the error area and reliability (displayed in white in FIG. 4D) in the other areas. Do the process that can be obtained.

Ｓは類似度（値が小さいほど類似度が高く、値が大きいほど類似度が低い）、Ｉは撮影画像の輝度値を表し、ｐは信頼度算出対象画素の位置、ｑは信頼度算出対象画素ｐの周辺画素の位置である。 In step S31, the similarity acquiring unit 1331 calculates the pixel value of the pixel (first pixel) in the captured image corresponding to the calculation target pixel (regarding pixel) of the reliability and a plurality of pixels (plural pixels in the peripheral region). The similarity S of the pixel values of the second pixel of For example, if the photographed image is a monochrome luminance image, the similarity S is calculated as a luminance difference as follows.

S indicates the degree of similarity (the smaller the value, the higher the degree of similarity; the larger the value, the lower the degree of similarity), I represents the luminance value of the photographed image, p is the position of the reliability calculation target pixel, and q is the reliability calculation target It is the position of the peripheral pixel of the pixel p.

Ｒ、Ｇ、Ｂはそれぞれ赤、緑、青のカラーチャンネルを表す。類似度算出手法は上記記載した距離計算だけではなく、マンハッタン距離等、どのような手法を用いても構わない。また、カラー画像からＣＩＥＬａｂ色空間やＹＵＶ色空間など、他の色空間に変換し、上記類似度計算を行ってもよい。 If the photographed image is a color image, the similarity S is calculated as the Euclidean distance of color difference as follows.

R, G and B represent red, green and blue color channels, respectively. The similarity calculation method is not limited to the above-described distance calculation, and any method such as Manhattan distance may be used. Alternatively, the color image may be converted to another color space such as CIELab color space or YUV color space, and the similarity calculation may be performed.

式３によれば、閾値Ｕよりも類似度Ｓが小さい（類似している）周辺画素の重みＷが１に決定され、それ以外の周辺画素の重みＷを０に決定される。なお、このようにして類似度Ｓと重みＷは関連付けられるため、類似度Ｓと重みＷは同一視することもできる。 The similarity acquisition unit 1331 further calculates the weight W used in the statistic calculation (S32) from the similarity S. The weight W takes a value between 0 and 1, and is determined to be closer to 1 as the similarity S is higher and to be closer to 0 as the similarity S is lower. That is, the weight W is set to 1 when the value of the similarity S is 0 in Equations 1 and 2, and is set to approach 0 as the value of the similarity S increases. In addition, a threshold U may be provided for the similarity S, and the weight W may be determined as follows.

According to Equation 3, the weight W of peripheral pixels whose similarity S is smaller than (or similar to) the threshold value U is determined to be 1, and the weights W of other peripheral pixels are determined to be 0. Since the similarity S is associated with the weight W in this manner, the similarity S and the weight W can be regarded as identical.

Next, in step S32, the statistic acquisition unit 1333 calculates the statistic T of the pixel (third pixel) in the distance image data 42 corresponding to the reliability calculation target pixel. The statistic T is a value for evaluating how far the distance value in the distance image data 42 is from the estimated true value. The statistic T is a distance value estimated to be correct at the target pixel based on the distance value between the calculation target pixel (third pixel) and its peripheral pixels (multiple fourth pixels) and the similarity S in the photographed image. Is calculated as the value corresponding to the difference between the and the actual distance value. At this time, the statistical amount T is calculated by performing weighted averaging on the distance value of the target pixel and the distance values of peripheral pixels using the weight W obtained above as a weight. If the weight W is determined as Expression 3, the statistic T is calculated using only distance values of pixels (pixels of the same object) having similar pixel values in the photographed image among peripheral pixels.

Ｔは統計量、Ｄは距離値、Ｑは周辺画素範囲（画素ｑの集合）を表す。周辺画素範囲Ｑは、広すぎると輝度（色）が類似している距離情報は値が近いという前提が崩れ、新たなエラーを生む可能性がある。また演算量も増えるという問題もある。一方、周辺画素範囲Ｑが狭すぎると、エラー領域における統計量Ｔの計算において周辺画素範囲Ｑ内に含まれる正しい距離を持った画素が少なくなり、適切にステップＳ３３の信頼度計算が行えなくなる可能性が生じる。したがって、周辺画素範囲Ｑの大きさは、オブジェクト境界において生じるエラー領域の大きさを基にあらかじめ決定しておくことが望ましい。例えば、距離算出においてテンプレートマッチングを採用する場合、エラー領域の大きさはテンプレートマッチングにおけるウィンドウサイズによって決まる。例えば、周辺画素範囲Ｑの大きさはウィンドウサイズの２倍程度とすることが考えられる。 The statistic T can be obtained, for example, as an absolute value of a difference between a weighted average value of distance values of peripheral pixels as described below and a distance value of a target pixel.

T represents a statistic, D represents a distance value, and Q represents a peripheral pixel range (set of pixels q). If the peripheral pixel range Q is too wide, the premise that the distance information having similar luminance (color) has a similar value is broken, which may cause a new error. There is also the problem that the amount of computation also increases. On the other hand, when the peripheral pixel range Q is too narrow, the calculation of the statistic T in the error area decreases the number of pixels having the correct distance included in the peripheral pixel range Q, and the reliability calculation in step S33 can not be appropriately performed. Sex occurs. Therefore, it is desirable that the size of the peripheral pixel range Q be previously determined based on the size of the error area generated at the object boundary. For example, when template matching is employed in distance calculation, the size of the error area is determined by the window size in template matching. For example, it is conceivable that the size of the peripheral pixel range Q is about twice the window size.

式５にしたがうと、統計量Ｔは、補正対象距離Ｄ（ｐ）と周辺画素距離Ｄ（ｑ）の差の絶対値の重み付き平均値として求められる。このような算出方法によっても、統計量Ｔは、対象画素において正しいと推定される距離値と実際の距離値の差を表すといえる。 The first term in the absolute value on the right side of Equation 4 can be regarded as a distance value estimated to be correct in the target pixel obtained from the distance values of peripheral pixels, using the similarity in the photographed image as a weight. Therefore, Equation 4 is a value corresponding to the difference between the distance value estimated to be correct in the target pixel and the actual distance value. As described above, the statistic T needs to be calculated by Equation 4 if it can be evaluated how much the distance information of the target pixel is away from the true value in consideration of the similarity in the photographed image. There is no. For example, the statistic T may be calculated according to the following Equation 5.

According to the equation 5, the statistic T is obtained as a weighted average value of absolute values of differences between the correction target distance D (p) and the peripheral pixel distance D (q). Also by such a calculation method, it can be said that the statistic T represents the difference between the distance value estimated to be correct at the target pixel and the actual distance value.

  The statistic T is a weight represented by a Gaussian distribution or the like having a certain dispersion according to the spatial distance | p−q | between the target pixel p and the peripheral pixels q in Equation 4 or Equation 5. It can also be determined according to other calculation formulas such as adding. In addition, part of the weight calculation (S31) and the statistic calculation (S32) may be realized simultaneously by filter processing. When the weight W is expressed by 1 or 0 as in Equation 3, the statistic T may be calculated by selecting only the pixels for which the weight W is 1.

Ｔｍｉｎは統計量Ｔの値の最小値、Ｔｍａｘは統計量Ｔの値の最大値を表す。統計量ＴがＴｍａｘの場合に信頼度Ｃは１となり、統計量ＴがＴｍｉｎの場合に０となり、統計量Ｔが０と１の間では信頼度Ｃは連続的に変化する。 Next, in step S33, the converting unit 1334 calculates the reliability C. The reliability C is determined in accordance with the aforementioned statistic T. For example, in the case where the statistic T is calculated according to the equation 4, it is judged to be unreliable when the statistic T is large, and it is judged to be more reliable when the statistic T is small. Naturally, depending on the method of calculating the statistic T, the method of obtaining the reliability C from the statistic T may differ, but for example, the following conversion equation can be adopted.

Tmin represents the minimum value of the statistic T, and Tmax represents the maximum value of the statistic T. The reliability C is 1 when the statistic T is Tmax, 0 when the statistic T is Tmin, and the reliability C changes continuously between 0 and 1 of the statistic T.

Ｌは閾値を表す。つまり、統計量Ｔが式４によって求められている場合は、周辺画素距離の重み付き平均値と補正対象距離との差の絶対値と、閾値Ｌとの比較によって信頼度が決定される。具体的には、統計量Ｔが閾値よりも大きければ距離値は信頼できないと判定され、閾値以下であれば距離値が信頼できると判定される。 Alternatively, the reliability C may be expressed as a binary value based on the comparison between the statistic T and the threshold. That is, assuming that the reliable reliability is 1 and the unreliable reliability is 0, the reliability C may be obtained as a binary value as follows.

L represents a threshold. That is, when the statistic T is obtained by Equation 4, the reliability is determined by comparing the absolute value of the difference between the weighted average value of the peripheral pixel distances and the correction target distance with the threshold L. Specifically, the distance value is determined to be unreliable if the statistic T is greater than the threshold, and the distance value is determined to be reliable if it is less than the threshold.

  5A to 5C are diagrams showing examples of the distance value D in the acquired distance image data, the statistic T obtained in step S32, and the reliability C obtained in step S33. Each of these figures shows the value of the A-A 'cross-sectional part of FIG. 4B. In FIG. 5A, the distance value before correction in the A-A 'cross section of FIG. 4B is indicated by a dotted line, and the weighted average distance value of the peripheral pixel distances is indicated by a solid line. Since the weighted average value of the peripheral pixel distance is calculated using the weight W based on the similarity S in the luminance image, the distance values are not averaged between the regions where the objects are different. However, it is assumed that different objects have different luminance distributions (color distributions), and different object areas are not similar. In FIG. 5A, a point A1 is an object boundary, a point A2 is a point where the pre-correction distance value and the weighted average value are the same, and a point A3 is a boundary between an error area and a non-error area.

  FIG. 5B shows the statistic T in the A-A ′ cross section of FIG. 4B. The statistic T here is obtained in accordance with the absolute value of the difference between the pre-correction distance (dotted line in FIG. 5A) and the weighted average value of the peripheral pixel distances (solid line in FIG. 5A). A point A4 represents a point having the same statistic as the statistic T (A3) at the point A3. Referring to FIG. 5B, when the threshold L of the equation 7 is set to the value of the statistic T (A3) at the point A3, the reliability becomes 0 in the region of A1 to A4, and becomes 1 in the other regions. . That is, by setting the threshold L to the statistic T (A3) at the point A3, the reliability C as shown in FIG. 5C can be obtained. Since the areas A1 to A4 are error areas, it can be seen that the reliability for this area can be calculated correctly. If the threshold L is set to a value larger than the value of the statistic T (A3), the low reliability region becomes a narrower region. Also, if the threshold L is set to a value smaller than the value of the statistic T (A3), a wider area can be made unreliable, but a part that is not an error area near the point A3 will also be unreliable. . In consideration of such a thing, it is preferable to use the statistic T (A3) at the point A3 as the threshold L in the equation 7.

  However, the threshold L does not necessarily have to be the statistic T (A3) at the point A3. Even if the value of the threshold L is larger or smaller than the statistic T (A3), the corresponding effect can be obtained. Therefore, the threshold L may be determined according to the situation.

  Further, the specific value of the statistic T (A3) at the boundary (point A3) between the error area and the non-error area changes due to the difference in distance between the foreground and background objects contained in the photographed image. For example, when the distance difference between objects is smaller, the value of the statistic T (A3) at the point A3 is smaller than when the distance difference is large. Therefore, in the reliability calculation process, in order to avoid that the reliable distance information is not determined to be unreliable, the value of the statistic T at the point A3 in the case where the distance difference between objects is the largest is obtained in advance The value may be adopted as the threshold L. The maximum distance difference is, for example, the difference between the distance corresponding to 0 and the distance corresponding to 255 if information corresponding to the distance is represented by 8 bits. Further, the position of the point A3, that is, how many pixels the boundary between the error area and the non-error area is away from the object boundary can be estimated in advance based on the method of calculating the distance map. For example, when template matching is used, it can be estimated that the position away from the object boundary by the number of pixels equal to the size of the window size is the boundary between the error area and the non-error area.

  When the threshold L is determined in this way, in the boundary area where the distance difference between the objects is small, the area determined to be unreliable becomes smaller, but the credible area is erroneously determined to be unreliable. Absent. As described above, the threshold L may be determined in advance by calculation, simulation or the like in accordance with a desired case.

  Instead of using a predetermined value as the threshold L, the threshold L may be changed dynamically. As described above, the value of the statistic at the boundary between the error area and the non-error area mainly depends on the distance difference between the foreground object and the background object. Therefore, the threshold L may be determined based on the difference between the maximum value and the minimum value of the distance around the reliability calculation target pixel. This makes it possible to calculate the reliability more appropriately.

ここで、添字ｉによってｉ回目（ｉは１以上の整数）の処理において算出される値であることを示す。重みＷ（および類似度Ｓ）および距離値Ｄは、繰り返し回数によらず同一の値を取る。 Although the reliability can be acquired (that is, the error area can be determined) by the above-described method, the area having reliability despite the unreliable information (from the point A4 to the point A3 in FIG. 5C) Area is left. In order to improve the accuracy of the reliability calculation, it is preferable to repeat the reliability acquisition process (the statistic calculation process S32 and the reliability calculation process S33). A flowchart for repeating is shown in FIG. 2C. Although the determination process S40 is included after the reliability calculation process S33, the other processes are basically the same as FIG. 2B. However, in the case of the second and subsequent processes, the reliability calculated in the immediately preceding process is used as the temporary reliability. For example, assuming that the reliability C _i is used as the temporary reliability at the time of calculating the statistic T _{i + 1} , the following equation 8 can be adopted as a calculation equation of the statistic.

Here, the subscript i indicates that the value is calculated in the i-th process (i is an integer of 1 or more). Weight W (and similarity S) and distance value D take the same value regardless of the number of repetitions.

In the present embodiment, as described above, it is assumed that the calculation result of the reliability of the pixel determined to have no reliability is correct. Therefore, with regard to a pixel determined to have no reliability by the previous repeated processing, the reliability may be determined without performing calculation again. That is, for a pixel p for which C _i (p) = 0, C _{i + 1} (p) = 0 may be set. As a result, not only the amount of calculation can be reduced, but also false detection can be prevented, in which an unreliable pixel can be erroneously trusted.

  The influence of adding the reliability C of the peripheral pixels as a weight will be described with reference to FIGS. 6A to 6C. 6A to 6C respectively represent the distance value D, the statistic T, and the reliability C, similarly to FIGS. 5A to 5C. Here, it is assumed that the area from point A1 to point A4 shown in FIG. 6A is low in reliability, and the other areas are high in reliability. The solid line in FIG. 6A is a weighted average value of the peripheral pixel distances (the first term in the absolute value of equation 8) in which the reliability is also used as a weight. As compared with FIG. 5A, the weighted average value is closer to the correct value because the distance value of the non-reliability area already calculated is not used to calculate the weighted average value. Since the weighted average value is closer to the true value, the statistic T calculated by the equation 8 becomes a larger value than that in FIG. 5B in the error area as shown in FIG. 6B. Therefore, as described above, when the value of the statistic T at the point A3 is set as the threshold L, as shown in FIG. 6C, it is possible to set no reliability in a wider area (area from the point A1 to the point A4 ′). it can. By repeating the above process, it is possible to more accurately match the area determined to have no reliability with the error area. A plurality of values can be determined in advance by calculation, simulation, or the like as described above, as the threshold L in the repetitive processing. Under such conditions, it is desirable to reduce the threshold each time the number of repeated processing is increased. In addition, the determination of repetition end in the determination process S70 is a determination of whether or not the number of times determined in advance has been reached, a determination of whether or not the number of pixels newly determined to have no reliability is less than a predetermined number, and a reliability For example, it may be determined whether or not the variance of the statistic in the region of is less than a predetermined value. Further, since an error remaining after the correction process can be roughly calculated by the value of the statistic, it may be determined whether or not the process is repeatedly performed based on the error.

  Equation 8 is not only used for the second and subsequent iterations of the iterative process, but may be employed instead of equation 4 for the first calculation process (including the case where the iterative process is not performed). In that case, it is necessary to calculate the provisional reliability in advance by some method different from the method described above. The method is not particularly limited. For example, the reliability obtained when the distance map generation unit 132 calculates distance information can be used as the temporary reliability. For example, the distance map generation unit 132 can calculate the reliability based on the degree of the texture of the subject in the captured image, the luminance value, and the like. More specifically, it is possible to determine that distance information in an area with low texture or an area with low luminance has low reliability.

  Also, the above reliability calculation method can also be used to evaluate errors other than errors at object boundaries. For example, it can be used to evaluate an error that occurs in a region of a certain size when calculating a distance. By this method, it is also possible to determine the reliability of the area where a large error occurs in the same object.

式９において、Ｄは補正前の距離値、Ｄ’は補正後の距離値である。Ｉは撮影画像データ内の画素の輝度値または色情報である。ｐは距離画像データ内の補正対象画素の位置、ｑは補正対象画素ｐの周辺画素の位置である。Ｇはガウス関数（σ_ｓ、σ_ｒは分散値）で、Ｇ_σｓおよびＧ_σｒは異なるガウス関数でもよいし、同じガウス関数でもよい。Ｑ’は計算範囲（画素ｑの集合）であり、Ｓが大きいと、周辺画素ｑの個数も大きくなる。Ｃは最終的に得られた信頼度を表す。Ｔｈは、信頼度Ｃが高い場合には１、信頼度Ｃが低い場
合には０に設定される。例えば、信頼度Ｃが０以上１以下の数値範囲をとる場合、信頼度Ｃが０．５以上のときにＴｈは１に設定され、Ｃが０．５未満の場合にＴｈは０に設定されるようにする。なお、Ｃが０と１の２値の値しかとらない場合には、Ｔｈ（Ｃ）はＣと置換してもよい（すなわちＴｈ（Ｃ）＝Ｃとしてよい）。 Next, in step S34, the distance map correction unit 134 corrects the distance map using the reliability calculated in step S33. The correction process is, for example, the following filter process as an example.

In Equation 9, D is a distance value before correction, and D ′ is a distance value after correction. I is the luminance value or color information of the pixel in the photographed image data. p is the position of the correction target pixel in the distance image data, and q is the position of the peripheral pixel of the correction target pixel p. G is a Gaussian function (σ _s , σ _r is a variance value), and G _σs and G _σr may be different Gaussian functions or may be the same Gaussian function. Q ′ is a calculation range (set of pixels q), and when S is large, the number of peripheral pixels q also increases. C represents the finally obtained reliability. Th is set to 1 when the reliability C is high, and to 0 when the reliability C is low. For example, in the case where the reliability C takes a numerical range of 0 or more and 1 or less, Th is set to 1 when the reliability C is 0.5 or more, and Th is set to 0 when C is less than 0.5. To make When C takes only two values of 0 and 1, Th (C) may be replaced with C (that is, Th (C) may be C).

  According to the present embodiment, it is possible to set the correct reliability in the distance image data. Further, by performing the correction process of the distance information based on the correct reliability information, it is possible to correct the distance image data with higher accuracy.

(Modification of Embodiment 1)
The first term of equation 8 above and the form of equation 9 are very similar. Since Equation 9 is distance information after correction, the statistic T defined by Equation 8 (and Equation 4) can be considered as an absolute value of the difference between the distance information after correction and the distance information before correction. Therefore, the reliability calculation process in the first embodiment can also be realized by a process as shown in FIG. That is, first, the similarity acquisition unit 1331 calculates the similarity S between the pixel p of interest in the captured image and its neighboring pixels q (Equation 1). The distance map correction unit 134 uses the weight W based on the calculated similarity S (for example, G _σs (| p−q |) × G _{σ r} (S)) to calculate the distance value D of the peripheral pixel q in the distance image Correction for obtaining the distance value of the pixel of interest p is performed using the equation (9). Then, the statistic acquisition unit 1333 calculates, as the statistic T, the absolute value of the difference between the distance value D before correction and the distance value D ′ after correction in the pixel of interest p (corresponding to equation 8). Finally, the converting unit 1334 converts the statistic T into the reliability C (Equation 6 or 7).

  The distance image correction process is not limited to a specific calculation method. The correction process of the distance image is the correction process as long as it reduces the difference between the true value of the distance information in the target pixel estimated in consideration of the degree of similarity in the photographed image and the distance information of peripheral pixels. Arbitrary correction processing can be adopted other than the correction shown. That is, the reliability of the distance information may be set by comparing the distance information corrected using any means with the distance information before correction. When the above-described repetitive processing is performed, corrected distance information is obtained, and thereafter, the reliability is obtained. That is, if the correction is sufficient, the end can be determined at that stage, so that the determination of the number of repetitions can be made based on the correction result.

  As described above, when the reliability calculation process is repeated, the distance image may be corrected using the reliability information obtained in the previous process. Note that for pixels determined to be unreliable by the previous processing, it can be determined that there is no reliability regardless of the value of the statistic. As a specific example of the determination of the end of the repetitive processing, for example, a determination of whether or not the number of times determined in advance has been reached, a determination of whether or not the maximum value of the correction amount by the correction processing of the distance image becomes less than a predetermined value For example, it may be determined whether the number of pixels newly determined to have no reliability is equal to or less than a predetermined number.

  In the method of the first embodiment, the calculation process of the statistic and the correction process of the distance image are different contents, but in the method of the present modification, since the main parts of these two calculations can be made common, the same arithmetic circuit is used. The implementation used is possible and the cost can be reduced. Further, since the distance image is corrected at the time of calculating the statistical amount, it is possible to reduce the calculation amount even at the point that the distance image does not need to be corrected again after the calculation of the reliability information.

Second Embodiment
In the first embodiment, generation of reliability information data and correction of distance information are performed using captured image data and distance image data. On the other hand, in the present embodiment, it is shown that generation and correction processing of reliability information can be performed using something other than photographed image data and distance image data. FIG. 8 shows a data processing device of the present embodiment. The flowchart is the same as the flowchart of the first embodiment. An image processing method according to the present embodiment will be described focusing on differences from the first embodiment.

  The data processing device 81 includes a first image data input unit 810, a second image data input unit 811, a reliability data calculation unit 812, and a second image data correction unit 813. The first image data input unit 810 receives image data (first image data) to be a reference. The first image data may be luminance image data, and in addition to this, for example, distance image data already corrected, distance image data obtained by a method such that an error does not occur in principle near an object edge, etc. It may be The first image data may also be information such as infrared light or polarization. In addition, when acquiring a luminance image or the like as the first image data, as shown in FIG. 1, the data processing device 81 can be included in the imaging device. In addition, when the first image data is distance image data and the like, the data processing device 81 may include a data generation unit for generating distance image data and the like. Assuming that the first image data is distance image data, I in Equation 1 corresponds to a distance value (does not include an error at an object boundary).

  The second image data input unit 811 receives image data (second image data) to be subjected to calculation of the reliability. Preferably, the second image data is image data having substantially the same viewpoint as the first image data, and is image data representing different information. The second image data is image data that includes an error at an object boundary. The first image data and the second image data may be data representing the same content information calculated by different calculation methods. Further, the first image data and the second image data do not necessarily have to have the same angle of view, as long as the second image data is included in the first image data. That is, the angle of view of the first image data may be equal to or larger than the angle of view of the second image data.

  As an example other than the distance image data of the second image data, motion data (Optical Flow) can be mentioned. Motion data is data representing the motion of an object (sometimes including the motion of a camera). For example, motion data holds the velocity in the horizontal direction (x direction) and the vertical direction (y direction) as data for each pixel. Motion information of the subject generally takes two luminance images at a certain time interval, and by template matching of the two images, a likely corresponding position is calculated, and from the movement amount and the photographing time interval Each velocity in the lateral direction can be calculated. The speed calculated in this manner is such that when pixels of different speeds are mixed in the window of template matching, these plural speeds are mixed to obtain an intermediate speed. That is, at the boundaries of objects moving at different speeds, the calculated motion information (speed) will have errors. Assuming that the second image data is motion data, D in Equation 4 corresponds to the lateral velocity value and the longitudinal velocity value.

  The second image data may be an infrared image or a polarized image. Infrared images and polarized images may have lower resolution than RGB images due to the effects of lens chromatic aberrations and special sensor structures. Also in this case, it can be considered that an error occurs at the object boundary.

  As in the example described here, the second image data may be data that has an error at the boundary between different objects. Further, enlargement (data interpolation) processing may be performed on the first image data when the second image data has a small size (small amount of data). In that case, again, the enlargement process generates an error near the object boundary. Image data after such enlargement processing can also be used as second image data. For example, an image sensor for acquiring the above-mentioned infrared image or polarization image may have an infrared color filter or polarization filter only at a specific pixel. In this case, since the acquired infrared image or polarized image has a smaller size than the RGB image, enlargement processing may be performed. The second image data may be generated by an apparatus different from the data processing apparatus 81 and input to the data processing apparatus 81, or the data processing apparatus 81 may be generated from the first image data or other information. .

  As described above, if there is an error in the second image data and the first image data is data serving as a reference for correction, the weight calculation S31, the statistic calculation S32, the reliability calculation S33, and the correction processing S34 are performed. It can be calculated as in the first embodiment. Depending on the data, there may be multiple correction target data, but the basic processing does not change. For example, in the case of motion data, the correction processing of motion data in the horizontal direction and the correction processing of data in the vertical direction are independently performed using the same method.

  According to the present embodiment, not only distance image data but reliability can be set for various data having an error. Further, by performing the correction processing based on the reliability, it is possible to correct data with higher accuracy.

(Embodiment 3)
In the first embodiment, as shown in FIG. 4C, the error at the object boundary gradually changes. In the present embodiment, as shown in FIG. 9, the image (second image) including an error such that the distance value of one object in the object boundary portion is substantially the same as the distance value of the other object Calculate the degree and correct the information. The present embodiment is directed to a distance image including an error such that the distance value of the background object is calculated as the distance value of the foreground object at the object boundary. Note that what kind of error the distance image has is determined by the method of calculating the distance image.

  The process in this embodiment is as shown in the flowcharts of FIGS. 2A to 2C, as in the first embodiment. The data processing method according to the present embodiment will be described with reference to FIG. 2B, focusing on differences from the first embodiment. In the present embodiment, an example in which the luminance information and the distance information are targets of processing will be described. However, as described in the second embodiment, similar calculations can be performed for other data.

図１０Ａの実線が式１０における第一項の周辺画素距離の重み付き平均値を表し、図１０Ｂが式１０によって求められる統計量Ｔを表している。もちろん統計量Ｔは必ずしも式１０によって求められる必要はない。また統計量Ｔは、式８のようにあらかじめ算出した信頼度情報を用いて計算してもよい。図１０Ａおよび図１０Ｂにおいて、点Ｂ１はオブジェクト境界、点Ｂ２はエラー領域とエラーでない領域の境界を表している。輝度画像の類似度を基に重みが決定されているため、点Ｂ１を跨いで、距離が平均化されることはない。 The similarity calculation process S31 is equivalent to that of the first embodiment. In the statistic calculation processing S32, for example, the difference between the weighted average value of the peripheral pixel distances and the distance of the correction target pixel is set as the statistic T as follows.

The solid line in FIG. 10A represents the weighted average value of the peripheral pixel distances of the first term in Equation 10, and FIG. 10B represents the statistic T obtained by Equation 10. Of course, the statistic T does not necessarily have to be determined by equation 10. Also, the statistic T may be calculated using reliability information calculated in advance as shown in Equation 8. In FIGS. 10A and 10B, a point B1 represents an object boundary, and a point B2 represents a boundary between an error area and a non-error area. Since the weight is determined based on the degree of similarity of the luminance image, the distance is not averaged across the point B1.

In the reliability calculation process S33, the reliability C is determined in accordance with the aforementioned statistic. As described above, in the present embodiment, since the distance image in which the distance value of the foreground object is calculated as the distance value of the background object is targeted, for example, when the statistic T as shown in equation 10 is calculated, The degree of reliability can be determined by the positive or negative of the statistic T. Assuming that the reliability representing reliability is 1 and the reliability representing unreliability is 0, the reliability C may be determined from the sign of the statistic T as shown in Equation 11.

  By such a calculation method, it is possible to determine that the distance value in the range from the point B1 to the point B2 is unreliable. Although the reliability is determined based on the positive or negative of the statistic T in Equation 11, the weighted average distance value of the peripheral pixels and the distance value of the correction target pixel may be different due to noise or the like even outside the error region. In that case, the value of Equation 10 can be positive or negative. In such a case, C (p) = 0 may be set when T <a (a <0) so that a pixel not including an error is not erroneously determined to be unreliable.

  According to the present embodiment, it is possible to set the correct reliability by calculating the statistic and the reliability in accordance with the shape of the error at the object boundary.

  In addition, it is necessary to appropriately change the calculation method of the reliability degree according to the method of the occurrence of the error in the reliability calculation target image. For example, if the distance value of the foreground object is a distance image including an error calculated as the distance value of the background object, it should be determined that the statistical value T according to Equation 10 is unreliable when it is positive.

(Other embodiments)
The data processing apparatus and method of the present invention described above can be preferably applied to, for example, an imaging apparatus such as a digital camera or camcorder, or an image processing apparatus or computer that performs image processing on image data obtained by the imaging apparatus. The technology of the present invention is also applicable to various electronic devices (including mobile phones, smart phones, slate-type terminals, and personal computers) incorporating such imaging devices or image processing devices. Although the configuration in which the function of the image processing apparatus is incorporated in the main body of the imaging device has been described in the above embodiment, the function of the image processing apparatus may be configured in any way. For example, the image processing apparatus may be incorporated into a computer having an imaging device, and the computer may acquire an image captured by the imaging device and execute the image processing method based thereon. Alternatively, the image processing apparatus may be incorporated in a computer capable of network access by wire or wireless, and the computer may acquire a plurality of images via the network and execute the image processing method based thereon. . The obtained distance information can be used, for example, for various image processing such as area division of an image, generation of a stereoscopic image or a depth image, and emulation of a blur effect.

  The specific implementation to the above apparatus can be implemented either by software (program) or hardware. For example, even if a program is stored in the memory of a computer (microcomputer, FPGA, etc.) incorporated in an imaging device etc. and the computer is executed by the computer, various processes for achieving the object of the present invention can be realized. Good. It is also preferable to provide a dedicated processor such as an ASIC that implements all or part of the processing of the present invention by a logic circuit.

  The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

133 Reliability Data Calculation Unit 1331 Similarity Acquisition Unit 1332 Reliability Acquisition Unit

According to a first aspect of the present invention, there is provided an acquisition means for acquiring an image data and a distance map indicating a distribution of distance information corresponding to the image data, and each of the distance maps with reference to a corresponding region of the image data. And correcting means for applying a filtering process to the area to generate a correction distance map, wherein the correction means is configured to calculate the distance of the first position on the distance map relative to the distance information of the target area on the distance map. It has a first pixel value of the image data by giving greater weight to distance information of a second position closer to the target area on the distance map than the first position on the distance map than information. The distance information corresponding to the position of a pixel having a second pixel value closer to the pixel value of the image data corresponding to the target area than the first pixel value is larger than the distance information corresponding to the position of the pixel Data processing apparatus characterized by correcting distance information of the target region by performing weighted addition using distance information around the target region, and generating the corrected distance map. .

According to a second aspect of the present invention, there is provided an acquiring step of acquiring image data and a distance map indicating distribution of distance information corresponding to the image data, and each of the distance maps with reference to a corresponding region of the image data. And a correction step of applying a filtering process to the area to generate a correction distance map, wherein in the correction step, the distance of the first position on the distance map with respect to the distance information of the target area in the distance map It has a first pixel value of the image data by giving greater weight to distance information of a second position closer to the target area on the distance map than the first position on the distance map than information. The distance information corresponding to the position of a pixel having a second pixel value closer to the pixel value of the image data corresponding to the target area than the first pixel value than the distance information corresponding to the position of the pixel Data processing characterized by correcting distance information of the target area by performing weighted addition using distance information on the periphery of the target area by giving a larger weight to It is a method.

According to a third aspect of the present invention, there is provided an acquisition means for acquiring a first image data and a map in which each data corresponds to each area of the first image data, and an area corresponding to the first image data. And correction means for applying a filtering process to each area of the map with reference to
And the correction means is configured to set the target area on the map more than the first position on the map than the data on the first position on the map with respect to data of the target area on the map. Image data corresponding to the target area rather than the first pixel value rather than data corresponding to the position of the pixel having the first pixel value of the image data given more weight to the data of the closer second position The data in the target area is corrected by performing weighted addition using data in the vicinity of the target area by giving greater weight to data corresponding to the position of the pixel having the second pixel value close to the pixel value of And generating the correction map.

A fourth aspect of the present invention is an acquisition step of acquiring first image data and a map in which each data corresponds to each area of the first image data, and an area corresponding to the first image data. And correcting the respective areas of the map with reference to the correction process to generate a correction map, wherein in the correction step, data of a target area in the map is compared with the first on the map. The data of the second position closer to the target area on the map than the first position on the map is weighted more than the data on the first position, and has a first pixel value of the image data More weight is given to data corresponding to the position of a pixel having a second pixel value closer to the pixel value of the image data corresponding to the target area than the data corresponding to the position of the pixel than the data corresponding to the position of the pixel. Only, the data of the target area is corrected by performing a weighted addition using data around the serial target area, a data processing method and generates the correction map.

Claims (34)

A data processing apparatus for calculating reliability data representing reliability of pixel values of at least a part of pixels of the second image data from the first image data and the second image data,
A pixel value of a first pixel of the first image data corresponding to a target pixel of the reliability data, and a pixel value of each of a plurality of second pixels in a predetermined peripheral region of the first pixel; A similarity acquisition unit for acquiring the similarity of
The degree of similarity for each of the plurality of second pixels, the pixel value of a third pixel of the second image data corresponding to the first pixel, and the predetermined peripheral region of the third pixel A reliability acquisition unit for acquiring a reliability, which is a pixel value of a pixel of interest of the reliability data, from the pixel values of the plurality of fourth pixels corresponding to the plurality of second pixels and located inside the When,
A data processing apparatus comprising: The reliability acquisition unit performs weighted averaging processing on each pixel value of the plurality of fourth pixels or a value obtained based on the pixel value, using the similarity for each of the plurality of second pixels as a weight. To get the confidence,
The data processing apparatus according to claim 1. The reliability acquisition unit is configured based on the degree of similarity for each of the plurality of second pixels, the pixel value of each of the plurality of fourth pixels, and the pixel value of the third pixel. Obtaining a value corresponding to a difference between a pixel value estimated to be correct in the third pixel and an actual pixel value, and acquiring the reliability based on the value;
The data processing apparatus according to claim 1. The reliability acquisition unit calculates the reliability based on a value calculated by the following equation:
The data processing apparatus according to any one of claims 1 to 3.

Here, p is the third pixel, q is the fourth pixel, Q is a set of the fourth pixels, and W (q) is the similarity in the second pixel corresponding to the fourth pixel. , D (q) is the pixel value of the fourth pixel, and D (p) is the pixel value of the third pixel. When the absolute value of the value calculated by the expression A is larger than the threshold, the reliability acquisition unit acquires a value indicating that the reliability is low as the pixel value of the focused pixel, and the calculation is performed by the expression A When the absolute value of the calculated value is equal to or less than the threshold value, a value indicating that the reliability is high is acquired as the pixel value of the pixel of interest.
The data processing device according to claim 4. The reliability acquisition unit calculates the reliability based on a value calculated by the following equation:
The data processing apparatus according to any one of claims 1 to 3.

Here, p is the third pixel, q is the fourth pixel, Q is a set of the fourth pixels, and W (q) is the similarity in the second pixel corresponding to the fourth pixel. , D (q) is the pixel value of the fourth pixel, and D (p) is the pixel value of the third pixel. The reliability acquisition unit acquires a value indicating that the reliability is low as the pixel value of the pixel of interest when the value calculated by the expression B is larger than the threshold, and the value calculated by the expression B In the case where the absolute value of is smaller than the threshold value, a value indicating that the reliability is high is acquired as the pixel value of the target pixel,
The data processing apparatus according to claim 6. The reliability acquisition unit further acquires the reliability by further using the temporary reliability of the pixel value of the third pixel and the pixel value of each of the plurality of fourth pixels.
The data processing device according to any one of claims 1 to 7. The reliability acquisition unit repeatedly executes acquisition processing of the reliability,
The provisional reliability is the reliability acquired by the immediately preceding iterative process,
The data processing apparatus according to claim 8. The reliability acquisition unit uses the reliability obtained in advance by a method different from the acquisition processing as the temporary reliability in the acquisition processing of the first reliability in the repeated processing.
The data processing device according to claim 9. The reliability acquisition unit determines the number of repetitions of the repetitive processing based on the result of the reliability.
A data processing apparatus according to claim 9 or 10. The reliability acquisition unit calculates the reliability based on a value calculated by the following equation:
A data processing apparatus according to any one of claims 8 to 11.

Here, p is the third pixel, q is the fourth pixel, Q is a set of the fourth pixels, and W (q) is the similarity in the second pixel corresponding to the fourth pixel. , D (q) is the pixel value of the fourth pixel, D (p) is the pixel value of the third pixel, and C (q) is a value indicating the provisional reliability of the pixel value of the fourth pixel. is there. The reliability acquisition unit acquires a value indicating that the reliability is low as the pixel value of the pixel of interest when the absolute value of the value calculated by the expression C is larger than a threshold, and the calculation is performed by the expression C. If the absolute value of the calculated value is equal to or less than the threshold value, a value indicating that the reliability is low is acquired as the pixel value of the pixel of interest.
A data processing apparatus according to claim 12. The reliability acquisition unit repeatedly executes acquisition processing of the reliability,
The threshold value is set smaller as the number of repetitions increases.
The data processing device according to claim 13. The threshold value is set to a value of Formula A, Formula B or Formula C at a pixel separated by a position separated by a predetermined number of pixels from an object boundary.
A data processing apparatus according to any one of claims 5, 7, 13, 14. The reliability acquisition unit uses, as the temporary reliability, a reliability obtained in advance by a method different from the acquisition processing.
The data processing apparatus according to claim 8. The provisional reliability is calculated based on the degree of texture of the subject.
The data processing apparatus according to claim 16. The similarity is a value of 0 or 1.
The data processing apparatus according to any one of claims 1 to 17. The size of the predetermined peripheral area is larger than the size of the area including the error in the pixel value of the second image data at the object boundary.
A data processing apparatus according to any one of the preceding claims. The reliability acquisition unit acquires a value indicating that the reliability is low as the pixel value of the pixel of interest when the value calculated by the expression A or C is negative, and the expression A or C is obtained. If the value calculated in is positive, a value indicating that the reliability is high is acquired as the pixel value of the pixel of interest.
A data processing apparatus according to claim 4 or 12. The image processing apparatus further includes a correction unit that corrects second image data using the reliability acquired by the reliability acquisition unit.
21. A data processing apparatus according to any one of the preceding claims. A data processing apparatus for calculating reliability data representing reliability of pixel values of at least a part of pixels of the second image data from the first image data and the second image data,
A correction unit that corrects the second image data based on the first image data and the second image data, and generates a second image data after correction;
A reliability acquisition unit that acquires a reliability by comparing the second image data after correction and the second image data before correction;
A data processing apparatus comprising: The correction unit further corrects the second image data using the temporary reliability.
A data processing apparatus according to claim 22. Repeatedly executing processing by the correction unit and the reliability acquisition unit;
The correction unit uses the reliability obtained in advance as the temporary reliability in the first process, and uses the reliability acquired immediately before by the reliability acquiring unit as the temporary reliability in the second and subsequent processes.
A data processing apparatus according to claim 23. The number of repetitions of the repetitive processing is determined according to the correction result of the second image data by the correction unit.
The data processing device according to claim 24. The first image data and the second image data are image data representing different information of substantially the same viewpoint,
The angle of view of the first image data is equal to or greater than the angle of view of the second image data.
26. A data processing apparatus according to any one of the preceding claims. The second image data is image data including an error at an object boundary.
The data processing device according to any one of claims 1 to 26. The first image data is image data having luminance or color information as a pixel value,
The second image data is image data having distance information as a pixel value,
28. A data processing apparatus according to any one of the preceding claims. The first image data is image data having luminance or color information as a pixel value,
The second image data is image data having motion information as a pixel value,
28. A data processing apparatus according to any one of the preceding claims. The first image data is image data having distance information as a pixel value,
The second image data is image data having motion information as a pixel value,
28. A data processing apparatus according to any one of the preceding claims. The similarity acquisition unit acquires the similarity in each of a plurality of second pixels based on a luminance difference or a color difference of each of the first pixel and the plurality of second pixels.
A data processing apparatus according to claim 28 or 29. An imaging device,
32. A data processing apparatus according to any one of the preceding claims,
An imaging device having A data processing method performed by a data processing apparatus, which calculates reliability data representing reliability of pixel values of at least a part of pixels of the second image data from first image data and second image data. There,
A pixel value of a first pixel of the first image data corresponding to a target pixel of the reliability data, and a pixel value of each of a plurality of second pixels in a predetermined peripheral region of the first pixel; Similarity step to obtain the similarity of
The degree of similarity for each of the plurality of second pixels, the pixel value of a third pixel of the second image data corresponding to the first pixel, and the predetermined peripheral region of the third pixel A reliability acquisition step of acquiring the reliability, which is the pixel value of the pixel of interest of the reliability data, from the pixel values of each of the plurality of fourth pixels corresponding to the plurality of second pixels located within When,
A data processing method comprising: A data processing method performed by a data processing apparatus, which calculates reliability data representing reliability of pixel values of at least a part of pixels of the second image data from first image data and second image data. There,
Correcting the second image data based on the first image data and the second image data, and generating a second image data after the correction;
A reliability step of acquiring reliability by comparing the second image data after correction and the second image data before correction;
A data processing method comprising:

Images