JP6838918B2 - Image data processing device and method - Google Patents

Description

The present invention relates to an image data processing apparatus and method.

At the same time as acquiring the captured image, a method of acquiring or calculating a distance image representing the distribution of distance information from the imaging device to the subject and a motion image representing the distribution of relative motion information between the imaging device and the subject has been proposed. There is. The acquired information often has a lower spatial resolution than the captured image, and therefore it is necessary to perform upsampling processing (enlargement processing) according to the captured image to increase the amount of information. However, upsampling causes a problem that information cannot be obtained correctly, especially at the boundary of distance and movement. A method to solve this problem has been proposed.

In Patent Document 1, a joint bilateral filter is used to correct distance information at the same time as upsampling. In Non-Patent Document 1, a Gaussian filter is applied to the edge strength of a distance image, and a reliability map showing the reliability of information is generated according to the value. Distance information, reliability map, and brightness information are used to upsample and correct distance information. In Patent Document 2, high-resolution image data is generated by upsampling low-resolution image data using a plurality of interpolation methods different from each other.

U.S. Pat. No. 7889949 Japanese Unexamined Patent Publication No. 11-168621

Garcia, F, et. Al., "Pixel weighted average strategy for depth sensor data fusion", 17th IEEE International Conference on Image Processing (ICIP) 2010, 2805-2808

However, Patent Document 1 has a problem that the distance boundary portion is blurred. This is because when the averaging process is performed using pixels corresponding to similar colors in the luminance image, the averaging process is performed using the wrong distance (distance between different objects having similar colors) near the boundary. The main cause is that it ends up.

Further, although Non-Patent Document 1 avoids the error as in Patent Document 1 in the form of reliability information, there is a problem that the accuracy is lowered due to the influence of the edge extraction method, the Gaussian filter, and the parameter setting of the threshold processing. .. There is also a problem that the amount of calculation such as edge extraction and filtering becomes large.

An object of the present invention is to provide an image data processing apparatus capable of generating the reliability of information of each pixel in enlarged image data with high accuracy and a small amount of calculation.

Image data processing apparatus according to an embodiment of the present invention, a high-resolution image data which is a low-resolution image data, the image data obtained by up-sampling the low-resolution image data, the low-resolution
Reliability data determination for determining the reliability data representing the reliability of the pixel value of the high-resolution image data and the image data acquisition means for acquiring the degree image data and the third image data corresponding to the high-resolution image data. The reliability data determining means is provided with means and correction means for correcting the high-resolution image data, and the reliability data determining means is in the vicinity of the pixel position in the low-resolution image data corresponding to the pixel position of the target pixel of the high-resolution image data. For comparison between the first evaluation value determined based on the pixel value of the pixel and the second evaluation value determined based on the pixel value of the neighboring pixel by a method different from the first evaluation value. Based on this, the reliability of the pixel value of the target pixel of the high-resolution image data is determined, and the correction means obtains the high-resolution image data based on the reliability data and the third image data. It is characterized in that correction is performed using the pixel value of the resolution image data.

Image data processing method according to an embodiment of the present invention is an image data processing method by the image data processing apparatus performs the high resolution image data which is a low-resolution image data, the image data of the low-resolution image data up-sampling And the image data acquisition step of acquiring the low-resolution image data and the third image data corresponding to the high-resolution image data, and the reliability data representing the reliability of the pixel value of the high-resolution image data are determined. The reliability data determination step includes a reliability data determination step and a correction step for correcting the high-resolution image data. In the reliability data determination step, the low-resolution image data corresponding to the pixel position of the target pixel of the high-resolution image data. The first evaluation value is determined from the nearby pixels of the pixels, and the first evaluation value is obtained from the nearby pixels of the pixels in the low resolution image data corresponding to the pixel positions of the target pixels of the high resolution image data. the different methods to determine the second evaluation value, based on a comparison of the first evaluation value and the second evaluation value, to determine the reliability of the target pixel of the high resolution image data, the The correction step is characterized in that the high-resolution image data is corrected by using the pixel value of the high-resolution image data based on the reliability data and the third image data .

According to the present invention, it is possible to generate the reliability of the information of each pixel in the enlarged image data with high accuracy and a small amount of calculation.

The block diagram of an example of the image pickup apparatus which includes the image data processing apparatus which concerns on embodiment. An example of a flowchart of the image data processing method according to the first embodiment. An example of captured image data and distance image data in the first embodiment. The figure explaining the reliability determination method in Embodiment 1. FIG. The figure explaining the reliability determination method in Embodiment 1. FIG. An example of a flowchart of the image data processing method according to the second embodiment. The figure explaining the reliability determination method in Embodiment 2. An example of a flowchart of the image data processing method according to the third embodiment. The figure explaining the reliability determination method using the neighborhood 4 pixels in Embodiment 3. FIG. The figure explaining the reliability determination method using the neighborhood 2 pixels in Embodiment 3. FIG. The schematic diagram which shows an example of the data processing apparatus in Embodiment 4.

In the present specification, the "image data" means two-dimensional array data in which numerical data is logically arranged two-dimensionally. Further, in the present specification, "image data" is also referred to as "map". The "pixel" of the image data means the position of the image data in the two-dimensional array. The numerical data constituting the image data is not limited to the data representing specific information, and includes, for example, data representing brightness information, distance information, motion information, and reliability information. The format of the numerical data is not limited to a specific format, and may be a scalar, a vector, a matrix, or the like. The image data is referred to as brightness image data, distance image data, motion image data, reliability image data, or the like, depending on the information to be handled.

The distance information in the present specification may be a relative distance from the focus position to the subject, or an absolute distance from the image pickup apparatus at the time of shooting to the subject. The absolute distance or the relative distance may be either the distance on the image plane side or the distance on the object side. Further, the distance may be expressed as a real space distance, or may be expressed as an amount that can be converted into a real space distance, such as a defocus amount and a parallax amount.

The present invention will be described in detail with reference to embodiments and drawings, but the present invention is not limited to the configuration of each embodiment. Moreover, each embodiment may be combined appropriately.

(Embodiment 1)
<Composition>
FIG. 1 schematically shows the configuration of the image pickup apparatus 1 according to the first embodiment of the present invention. The image pickup device 1 includes an image pickup optical system 10, an image pickup element 11, a control unit 12, an image data processing device 13, a storage unit 14, an input unit 15, and a display unit 16.

The image pickup optical system 10 is an optical system composed of a plurality of lenses and for forming an image of incident light on the image plane of the image pickup element 11. The image sensor 11 is an image sensor having an image sensor such as a CCD or CMOS. The image sensor 11 may be an image sensor having a color filter, a monochrome image sensor, or a three-plate image sensor.

The image data processing device 13 includes a signal processing unit 130, a memory 131, a distance map generation unit 132, an upsampling unit 133, a reliability data determination unit 134, and a distance map correction unit 135. The signal processing unit 130 is a functional unit that performs various signal processing such as AD conversion, noise removal, demosaiking, brightness signal conversion, aberration correction, white balance adjustment, and color correction of the analog signal output from the image pickup element 11. The digital image data output from the signal processing unit 130 is stored in the memory 131 and used for display on the display unit 16, recording (storing) on the storage unit 14, calculation of distance information, generation of distance image data, and the like. .. The distance image data represents the distribution of distance information.

The distance map generation unit 132 acquires photographed image data (brightness image data) from the signal processing unit 130, and acquires subject distance information and distance image data from the photographed image data. The method of acquiring the distance information of the subject is not particularly limited. Examples of methods for acquiring distance information include a method using captured image data with different blurring methods taken under different shooting conditions (Dept from Defocus method: DFD method) and a method using captured image data with different parallax (stereo method). ). Other examples include the Time of Focus method and the Depth From Focus method (DFF method). The distance image data generated by the distance map generation unit 132 is stored in the storage unit 14 or temporarily stored in the memory 131 and used for the subsequent processing. Further, the distance image data may be corrected according to the acquisition method.

In the present embodiment, the resolution of the distance map generated by the distance map generation unit 132 is lower than the resolution of the captured image data (FIGS. 3A and 3B). That is, the distance map generation unit 132 in the present embodiment does not obtain the distance information for all the pixels of the captured image data, but obtains the distance information only for the pixels at predetermined intervals.

The upsampling unit 133 has a function of upsampling the distance map generated by the distance map generation unit 132. Upsampling is a process for increasing the resolution of image data, and is also called upscaling, upconversion, or image enlargement processing. The upsampling unit 133 may make the distance map of any size by upsampling. However, if the size after upsampling is larger than the captured image data from which the distance image data is generated, the correction effect cannot be sufficiently obtained at the time of distance map correction. If the size after upsampling is smaller than the captured image data, an operation such as downsampling the captured image data at the time of distance map correction is required. In the present embodiment, the upsampling unit 133 enlarges the distance map to the same size as the captured image data from which the distance map is generated.

In the present specification, the distance image data before upsampling is also referred to as low resolution distance image data or low resolution image data. Further, the distance image data after upsampling is also referred to as high resolution distance image data, high resolution image data, enlarged distance image data, or enlarged image data.

The upsampling unit 133 can execute the upsampling process by two or more different algorithms. The upsampling (image enlargement) algorithm may be any existing algorithm. Examples of upsampling algorithms include Nearest Neighbor, Binilia, Bicubic, Lanczos-3 and the like. As the algorithm, interpolation using a polynomial approximation higher than bicubic or interpolation using a non-polynomial approximation other than Lanczos may be used.

The reliability data determination unit 134 has a function of calculating the reliability data (reliability map) of the distance map generated by the upsampling unit 133. The reliability data is data (map) consisting of the reliability of each pixel of the distance map. The reliability is information that is an index of the certainty of the distance information of each pixel of the distance map. The reliability may be multi-valued or binary. In this embodiment, the reliability is binary (reliable or unreliable).

Upsampling a distance map causes pixels, especially at the distance boundaries, to have unreliable values. The reliability data determination unit 134 determines the reliability of each pixel of the distance map after upsampling. The reliability determination method by the reliability data determination unit 134 will be described later.

The distance map correction unit 135 has a function of correcting the distance information of each pixel of the distance image data. The correction of the distance information is performed using the acquired captured image data, the distance image data made the same size as the captured image data by the upsampling unit 133, and the reliability data generated by the reliability data determination unit 134. It is said. The specific correction method will be described later.

The storage unit 14 is a non-volatile storage medium that stores captured image data, distance image data, reliability data, corrected distance image data, parameter data used in the image pickup apparatus 1, and the like. As the storage unit 14, any storage medium that can read and write at high speed and has a large capacity may be used. For example, a flash memory is preferable.

The input unit 15 is an interface for the user to operate and input information or change settings to the image pickup apparatus 1. For example, a dial, a button, a switch, a touch panel, or the like can be used as the input unit 15.

The display unit 16 is composed of a liquid crystal display, an organic EL display, or the like. The display unit 16 is used for confirming the composition at the time of shooting, viewing the shot / recorded image, displaying various setting screens and message information, and the like.

The control unit 12 is a function of controlling each unit of the image pickup apparatus 1. The functions of the control unit 12 include, for example, automatic focusing by autofocus (AF), change of focus position, change of F value (aperture), image capture, control of shutter and flash (all not shown), and storage. There are controls such as a unit 14, an input unit 15, and a display unit 16.

<Method>
FIG. 2 is a flowchart showing the flow of image processing according to the present embodiment. FIG. 3A is an example of captured image data to be processed, and FIG. 3B is an example of distance image data to be processed. FIG. 4 is a diagram showing a data flow in a part of pixels in the reliability calculation process. 5A to 5C are examples of the distance image data after the first upsampling, the distance image data after the second upsampling, and the reliability map, respectively. Hereinafter, the image processing performed by the image data processing apparatus 13 according to the present embodiment will be described with reference to these drawings.

In step S20, the upsampling unit 133 acquires the captured image data 301 (FIG. 3A) and the distance image data 302 (FIG. 3B). Specifically, the captured image data 301 can be acquired from the signal processing unit 130, and the distance image data 302 can be acquired from the distance map generation unit 132.

In FIGS. 3A and 3B, shading represents a pixel value (luminance value or distance value). In FIG. 3B, for the sake of simplification of the explanation, it is assumed that all the people (foreground) have the same distance and all the backgrounds have the same distance. However, the distance image data 302 actually holds a multi-valued distance information. Further, as described above, the distance image data 302 generated by the distance map generation unit 132 has a lower resolution than the original captured image data 301.

Next, in step S21, the upsampling unit 133 performs the first upsampling process on the distance image data 302. In this embodiment, nearest neighbor interpolation is used for the first upsampling process. In nearest neighbor interpolation, the value of the closest pixel with information is set for the pixel without information after upsampling.

The image 401 in FIG. 4 represents a part of the distance image data 302 (4 × 4 pixels). In the first upsampling process, when 3 × 3 times upsampling is performed using nearest neighbor interpolation, the image 401 becomes like the image 402. The pixel 411 surrounded by the thick line in the image 402 is a pixel corresponding to the pixel of the image 401, and has the same pixel value as the corresponding pixel. For the other pixels, the same pixel value as the closest pixel among the pixels 411 is set.

In this way, in the image 402, pixels that do not correspond to the image 401 are generated and inserted by interpolation. In particular, when focusing on the distance boundary portion, since the value of the closest pixel is selected as described above, the distance boundary in the image 402 after upsampling is the intermediate position (equidistant position) of the pixels 411. This is not always correct, and it is highly possible that the position of the distance boundary and the pixel value near the distance boundary in the image 402 are incorrect.

FIG. 5A shows the distance image data 501 after upsampling, which is obtained by performing the first upsampling process (nearest neighbor interpolation) on the distance image data 302.

Next, in step S22, the upsampling unit 133 performs a second upsampling process on the distance image data 302. In this embodiment, bilinear interpolation is used for the second upsampling process. The second upsampling processing algorithm may be any method, but it needs to be a method different from the first upsampling processing algorithm. In bilinear interpolation, a weighted average value of the pixel values of neighboring pixels having information is set for pixels without information after upsampling. In the weighted average, the distance between the interpolated pixel and the neighboring pixel is used as the weight.

The image 403 in FIG. 4 is an image obtained by upsampling the image 401 by 3 × 3 times using bilinear interpolation. The shading in FIG. 4 represents the magnitude of the pixel value (distance value). Focusing on the distance boundary, the distance changes linearly (linearly). In the upsampled image 403, the distance value is blurred and there is no clear distance boundary. This is not always correct, and it is highly possible that the pixel values in the above region in the image 403 are incorrect.

FIG. 5B shows the distance image data 502 after upsampling, which is obtained by subjecting the distance image data 302 to a second upsampling process (bilinear interpolation).

There are various upsampling methods, but any method may be used in steps S21 and S22. An important point in the upsampling method to be adopted is a pixel interpolation method without information generated by increasing the number of pixels. Examples of the interpolation method include near-rest neighbor interpolation, bilinear interpolation, and bicubic interpolation as general methods.

Next, in step S23, the reliability data determination unit 134 obtains the reliability for each pixel of the distance image data and generates the reliability data. As described above, in the present embodiment, the reliability data is binary (reliable or unreliable). The reliability data determination unit 134 compares the distance values of the same pixel positions of the two distance image data after the first and second upsampling, determines that the distance value is unreliable if the difference is larger than the threshold value, and determines that the distance value is unreliable. If it is less than or equal to, it is judged that the distance value is reliable. Such a reliability determination method is effective because it is considered that the correct distance value cannot be calculated by upsampling for pixels having different results when the upsampling method is different. The pixel position means the coordinates of the pixel in the image.

More specifically, the reliability data determination unit 134 obtains the reliability W (p) of the pixel position p as follows.

Here, _Du1 represents the distance image data after the first upsampling, and _Du2 represents the distance image data after the second upsampling. _Du1 (p) and _Du2 (p) each represent a pixel value at the pixel position p. E represents the difference between the distance values of the same pixel positions of the two distance image data after the first and second upsampling. U is a reliability determination threshold.

In this case, W = 1 indicates that it is reliable, and W = 0 indicates that it is unreliable. The determination threshold value U can be determined from the distance resolution (data resolution) of the distance image data and the like. For example, when the variation accuracy of the distance value can be approximated as a Gaussian distribution, 3σ (three times the standard deviation) can be set as the determination threshold value U. The determination threshold value U can also be determined based on the amount of error at the allowable distance boundary. For example, when it is desired to set the allowable error amount at the distance boundary portion within 1% of the maximum measurement distance difference, the threshold value U may be set to a value of 1% of the maximum measurement distance difference.

FIG. 5C shows a reliability map obtained based on the distance image data after upsampling of FIGS. 5A and 5B. In FIG. 5C, the portion drawn in black is an unreliable pixel, and the portion drawn in white is a reliable pixel. Image 404 of FIG. 4 is an enlarged view of a part of the reliability map (the part corresponding to images 401, 402, 403). Image 405 of FIG. 4 represents the corresponding captured image data. Assuming that the brightness boundary of the distance image data 405 represents the correct distance boundary, the reliability map 404 can correctly determine that the distance value is an unreliable region near the distance boundary (dotted line 412 in the reliability map 404). Represents the actual correct distance boundary).

In the present embodiment, the reliability is represented by a binary value, but the reliability may be a multi-value. When the reliability is set to multiple values, the reliability may be determined so that the larger the difference amount E, the lower the reliability.

In step S24, the distance map correction unit 135 corrects the upsampled distance image data based at least partially based on the reliability map calculated in step S23. The correction target may be the distance image data to which either the first or second upsampling has been performed, but the distance image data upsampled by using a more accurate method among these may be the correction target. It is good to do. In the present embodiment, correction processing is performed on the distance image data 502 upsampled by using bilinear interpolation.

As an example of the correction process, the following filter process can be mentioned.

In the formula (3), D is the distance information before the correction, and D'is the distance information after the correction. I is the luminance value or color information of the pixels in the captured image data. p is the position of the correction target pixel in the distance image data, and q is the position of the peripheral pixels of the correction target pixel p. G is a Gaussian function (σ is a variance value), and G _σs and G _σr may be different Gaussian functions or the same Gaussian function. Q is a calculation range and is appropriately set according to the upsampling rate.

By this filtering process, the distance data can be corrected by using only the reliable distance information without using the unreliable distance information. Further, since the weight is set to be larger as the luminance information or color information in the captured image data is similar, the distance data can be accurately corrected according to the luminance boundary or the color boundary in the captured image data. Further, when W (p) is 1, that is, when the reliable pixel is the pixel to be corrected, the correction does not need to be performed. Therefore, the calculation is performed by correcting only the pixel p in which W (p) is 0. The amount can be reduced.

The filter processing (correction processing) according to the above equation (3) can be similarly applied even when the reliability is multi-valued.

<Advantageous effect of this embodiment>
According to this embodiment, upsampling is performed using different upsampling methods, and the distance boundary (region with low reliability of pixel values) can be extracted with high accuracy by analyzing the difference in pixel values after upsampling.

Although the existing edge extraction method extracts edge pixels, it is not possible to extract all the pixels that may contain an error as in the reliability map 404 shown in FIG. In addition, when using the difference between the application results of Gaussian filters with different variance values such as Difference of Gaussian, it is possible to extract the vicinity of the distance boundary, but it is necessary to set each variance value appropriately for accurate extraction. is there. It is difficult to set an appropriate variance value, and if an inappropriate variance value is used, the correct distance boundary such as the reliability map 404 will not be extracted, and a range wider or narrower than the correct answer will be selected as the vicinity of the distance boundary. .. In addition, since filtering is required to extract the distance boundary, the amount of calculation is large.

According to the present embodiment, since the reliability required for correcting the error at the distance boundary portion due to upsampling can be correctly determined, it is possible to correct the distance image data with higher accuracy. Moreover, since edge extraction processing and filter processing are not required, the amount of calculation can be reduced.

(Embodiment 2)
In the first embodiment, the reliability is determined by using a different upsampling method, and the distance information is corrected. On the other hand, in the present embodiment, reliability data is generated by analyzing the relationship between neighboring pixels of the distance map before upsampling. The configuration of the image data processing device is the same as that of the data processing device (FIG. 1) of the first embodiment, and the processing contents of the upsampling unit 133 and the reliability data determining unit 134 are different. The processing content will be described with reference to the flowchart of FIG. The image processing method of the present embodiment will be described focusing on the differences from the first embodiment.

In step S60, the upsampling unit 133 acquires the captured image data 301 (FIG. 3A) and the distance image data 302 (FIG. 3B). This process is the same as in the first embodiment (step S20 in FIG. 3).

In step S61, the upsampling unit 133 performs upsampling processing on the distance image data 302. In the first embodiment, the interpolation performed together with the upsampling was important. In the present embodiment, the interpolation process is not indispensable from the viewpoint of obtaining the reliability data. In the upsampling process S61 of the present embodiment, the size of the image data (total number of pixels) is increased, but the number of pixels having a value is the same as the number of pixels of the image data before upsampling, and the other pixels are Has no value. The phrase "pixel has no value" also includes that the pixel has a value meaning "no value". However, in step S61, the upsampling unit 133 may execute an upsampling process accompanied by interpolation (for example, bilinear interpolation) to generate enlarged distance image data.

The process of step S61 will be described in more detail with reference to FIG. 7. In FIG. 7, the image 701 represents a part of the distance image data 302 (4 × 4 pixels), and the image 702 represents the region of the distance image data after the upsampling process corresponding to the image 701. Here, it is assumed that the upsampling process is performed to magnify the image by 3 × 3 times. In image 702, only 16 pixels corresponding to the pixels of image 701 have values, and the other pixels have no values. Pixels with no value are shown in gray in image 702.

Next, in step S62, the reliability data determination unit 134 obtains the reliability for each pixel of the distance image data and generates the reliability data. In the present embodiment, the evaluation values are obtained by two methods using pixels having values in the vicinity of the calculation target pixel p of the distance image data after upsampling, and the evaluation values are compared. The reliability with respect to the pixel p is obtained by.

The evaluation value H (p) for the pixel p is a weighted average value using only the pixels having a value in the vicinity of the pixel p and using a weighting coefficient according to the inter-pixel distance, as described below. Specifically, the evaluation value H (p) can be defined as follows.

In the equation (4), Q'represents a set of pixels in the vicinity of the pixel p and having a value. The set Q'may be set as appropriate, and the distance from the pixel p can be a pixel within a predetermined value. For example, a set consisting of pixels having values at the upper left, upper right, lower left, and lower right of the pixel p, or a set consisting of these two pixels when the pixel p is between two adjacent pixels in the vertical or horizontal direction. However, it may be set to include a larger number of pixels. F represents a function for calculating the weight according to the inter-pixel distance (| p−q |). Examples of F include a Gaussian function and a function in which only the closest pixel in the set of Q'is set to 1. F (| p−q |) obtained by such a function F can be regarded as calculating the weighting coefficient for the pixel q.

The reliability data determination unit 134 determines two evaluation values H (referred to as H1 and H2) for each pixel by using the weighting coefficients obtained by two different functions F (referred to as F1 and F2). The reliability data determination unit 134 determines that the distance value of the pixel p is unreliable if the difference | H1-H2 | between the two evaluation values H1 and H2 is larger than the judgment threshold value, and the difference | H1-H2 | is equal to or less than the judgment threshold value. If so, it is determined that the distance value of the pixel p is reliable.

For example, suppose that one function F1 is a Gaussian function having a variance σ, and the other function F2 is a function in which only the closest pixel is 1 in the pixel set Q'. Further, the set Q'is a set composed of pixels having four surrounding values. For simplicity, as shown in the images 701 and 702 of FIG. 7, the pixel value (distance value) of the pixel shown in white is set to 1, and the pixel value (distance value) of the pixel shown in black is set to 0.

The pixel values of the four neighboring points included in the set Q'corresponding to the pixel p1 in the image 702 of FIG. 7 are all 1. That is, the evaluation value for the pixel p is 1 for both the evaluation value H1 using the weight based on the Gaussian function F1 and the evaluation value H2 using the weight based on the function F2 for selecting only the closest pixel (H1). = H2 = 1). Therefore, since the difference between the evaluation values H1 and H2 is 0, it is judged that the pixel value (distance value) of the pixel p1 is reliable.

On the other hand, in the pixel p2, two nearby pixels included in the corresponding set Q'have a pixel value of "1" and two pixels have a pixel value of "0". Therefore, the evaluation value H1 using the weight based on the Gaussian function F1 takes an intermediate value between 0 and 1. On the other hand, since the pixel value of the closest pixel of pixel p is 0, the evaluation value H2 using the weight based on the function F2 that selects only the closest pixel is 0. If the difference between the evaluation values H1 and H2 | H1-H2 | is larger than the threshold value, it is determined that the pixel value (distance value) of the pixel p2 is unreliable.

The above-mentioned determination threshold value can be set to 0. In this case, if the values of the neighboring pixels included in the set Q'are different (not all are the same), it is determined that the pixels of the distance image data after upsampling are unreliable.

However, the determination threshold value is preferably a value having a certain magnitude larger than 0. This is because if the determination threshold value is too small, the region having a gradual change in distance is determined as a distance boundary, that is, an unreliable region. Therefore, it is desirable to set the threshold value according to the allowable distance change.

Image 703 in FIG. 7 shows a portion of the distance map after upsampling with interpolation corresponding to images 701 and 702. Image 704 in FIG. 7 is an example of a reliability map obtained by obtaining reliability for all pixels of image 702 (703) as described above. As can be seen from the figure, the vicinity of the distance boundary is appropriately extracted as a region where the distance value is unreliable.

In step S63, the distance map correction unit 135 corrects the enlarged distance image data using the reliability map obtained in step S62. Since the distance map correction process S63 is the same as that of the first embodiment (step S24 of FIG. 2), the repeated description will be omitted.

The function F does not have to be the Gaussian function described above or the function for selecting the closest pixel, but may be a function that outputs a predetermined weight that is inversely proportional to the distance. Further, the set Q'does not have to be 4 pixels in the vicinity, and 9 pixels or 16 pixels in the vicinity can be appropriately set.

Further, in the present embodiment, the reliability is assumed to be binary, but the reliability may be multi-valued as in the first embodiment. In this case, the reliability may be determined so that the larger the difference between the two evaluation values (| H1-H2 |), the lower the reliability.

Further, when the pixel value of the image data after upsampling is obtained by using interpolation as in the first embodiment, the value obtained by the interpolation can be regarded as corresponding to the evaluation value H in the present embodiment. The pixel value obtained by interpolation is the weighted average value of neighboring pixels (corresponding to pixels with values) before upsampling regardless of the interpolation method such as nearest neighbor, bilinear, or bicubic. This is because it can be regarded.

(Embodiment 3)
In the second embodiment, the reliability is obtained based on the pixel value (set Q') of the distance image data before upsampling corresponding to the vicinity of the pixel for which the reliability is calculated. Similarly, in the present embodiment, the reliability is obtained based on the set Q', but the reliability is obtained more simply based on the identity of the pixel values in the set Q'.

The configuration of the image data processing device is the same as that of the data processing device of the second embodiment, and the processing content of the reliability data determination unit 134 is different. The processing content will be described with reference to the flowchart of FIG. 8A. The image processing method of the present embodiment will be described focusing on the differences from the second embodiment.

In step S80, the upsampling unit 133 acquires the captured image data 301 (FIG. 3A) and the distance image data 302 (FIG. 3B). This process is the same as in the second embodiment (step S60 of FIG. 6).

In step S81, the upsampling unit 133 performs upsampling processing on the distance image data 302. In the present embodiment, the upsampling process S81 is unnecessary from the viewpoint of obtaining reliability data. The image data processing device may, for example, acquire the distance image data obtained by upsampling (enlarging) the distance image data acquired in step S80 from an external device.

In step S82, the reliability data determination unit 134 prepares the reliability map. The reliability map has the same size as the distance image data after upsampling. In the present embodiment, the reliability is binary, and the initial value of the reliability of each pixel of the reliability map prepared in step S82 is 0 (indicating that it is unreliable).

In step S83, the reliability data determination unit 134 determines the maximum difference between the pixel values (maximum and minimum values of the pixel values in the local region) for each local region (for example, 4 pixels) of the distance image data before upsampling. Difference) is obtained, and the reliability is determined based on this maximum difference. More specifically, if the maximum difference between the pixel values in the local region is equal to or less than the threshold value, the reliability data determination unit 134 sets the reliability of the region in the reliability map corresponding to the local region to 1 (reliable). Represents). If the maximum difference between the pixel values in the local area is greater than the threshold, the confidence map is not updated. The reliability data determination unit 134 performs the above processing on the entire distance image data while shifting the local region in the distance image data before upsampling. When shifting the local region, the reliability data determination unit 134 may allow duplication and shift in units of one pixel.

A more specific description will be given with reference to FIGS. 9A to 9D. In FIGS. 9A-9D, the image 901 represents a part (4 × 4 pixels) of the distance image data (distance map) before upsampling. The images 902 to 905 represent a reliability map of the portion corresponding to the image 901. In the reliability map in the figure, gray pixels indicate reliability 0 (unreliable), and white pixels indicate reliability 1 (reliable). Further, in the distance map in the figure, the pixel value (distance value) of the white pixel is set to 1 and the pixel value (distance value) of the black pixel is set to 0 for simplification of the explanation.

Further, in the following description, for the sake of simplicity, the determination threshold value is set to a sufficiently small value close to 0. However, in reality, it is preferable that the determination threshold value has a certain magnitude in consideration of the case where the subject distance is gradually changing. If the determination threshold is set to a value that is too small, a region having a gradual change in distance will be determined as a distance boundary, that is, an unreliable region. Therefore, it is desirable to set the threshold value according to the allowable distance change.

The pixel values of the four pixels in the local region p3 in the distance map 901 of FIG. 9A are all 1. Therefore, the maximum difference between the pixel values in the local region p3 is 0, that is, equal to or less than the determination threshold value. Therefore, the reliability of the region p4 (the region corresponding to the local region p3) in the distance map 902 is updated to 1. The region p4 is a square region (4 × 4 pixels) having 4 pixels after upsampling corresponding to 4 pixels included in the local region p3 (2 × 2 pixels) as vertices.

Of the 4 pixels in the local region p5 in the distance map 901 of FIG. 9B, 3 pixels are 1 and 1 pixel is 0. Therefore, the maximum difference between the pixel values in the local region p5 is 1, that is, the determination threshold value. Greater. Therefore, the reliability of the region p6 (the region corresponding to the local region p5) in the distance map 903 is not updated.

The pixel values of the four pixels in the local region p7 in the distance map 901 of FIG. 9C are all 0. Therefore, the maximum difference between the pixel values in the local region p7 is 0, that is, equal to or less than the determination threshold value. Therefore, the reliability of the region p8 (the region corresponding to the local region p7) in the distance map 904 is updated to 1.

FIG. 9D shows the reliability map 905 finally obtained by performing the above processing on the entire distance map. It can be seen that the distance boundary in the distance map can be properly extracted as an unreliable area. The distance map and the reliability map shown in FIGS. 9A to 9D each show a part of the area, and the reliability of the peripheral portion of the reliability map 905 is a local area straddling the distance map 901 and the distance map (not shown). Determined from the area.

In step S84, the distance map correction unit 135 corrects the enlarged distance image data using the reliability map obtained in step S83. Since the distance map correction process S84 is the same as that of the first embodiment (step S24 of FIG. 2), the repeated description will be omitted.

In the above description, the initial value of the distance map is set to 0, and when the maximum value of the pixel value in the local area is equal to or less than the threshold value (that is, reliable), the reliability of the corresponding area in the distance map is set. It has been updated to 1. However, when the initial value of the distance map is set to the reliability 1 and the maximum value of the pixel value in the local area is larger than the threshold value (that is, when it is unreliable), the reliability of the corresponding area in the distance map is updated to 0. You may.

Further, in the above description, the local region is a square composed of 4 pixels, but the shape of the local region is not limited to this. For example, the local area can be a larger square. Further, the local region may be a rectangle instead of a square. For example, a region composed of two adjacent pixels may be used as a local region. In this case, it is preferable to shift the local region consisting of two pixels arranged in the horizontal direction and the local region composed of two pixels arranged in the vertical direction to generate the reliability map. Further, the region for updating the reliability (region corresponding to the local region) does not have to be a part of the region obtained by upsampling the local region as described above, and may have other shapes.

Hereinafter, an operation example in the case where the local region is composed of two adjacent pixels will be described with reference to FIGS. 8B and 10A to 10F. FIG. 8B is a flowchart of an image processing method according to a modified example of the present embodiment. The difference from the flowchart of FIG. 8A is that a reliability map having an initial value of 1 is prepared in step S85, and the reliability is updated to 0 when the maximum difference is larger than the threshold value in step S86.

In FIGS. 10A to 10F, the image 1001 represents a part (4 × 4 pixels) of the distance image data (distance map) before upsampling. Images 1002 to 1007 represent a reliability map of a portion corresponding to image 1001. In the reliability map in the figure, gray pixels indicate reliability 0 (unreliable), and white pixels indicate reliability 1 (reliable). Further, in the distance map in the figure, the pixel value (distance value) of the white pixel is set to 1 and the pixel value (distance value) of the black pixel is set to 0 for simplification of the explanation. Further, for the sake of simplicity, the determination threshold value is set to a sufficiently small value close to 0.

Here, the determination is first made using the local region consisting of two pixels arranged in the horizontal direction (FIGS. 10A to 10C), and then the determination is made using the local region consisting of two pixels arranged in the vertical direction (FIGS. 10A to 10C). 10D-10F). However, this order may be either first, and the determination may be made by alternately using the local pixels in the horizontal direction and the local pixels in the vertical direction.

In FIG. 10A, since the pixel values in the local region p9 are all 1, the reliability map 1002 is not updated. In FIG. 10B, since the pixel values in the local region p10 are different, the reliability map 1003 is updated so that the reliability in the region p11 corresponding to the local region p10 is 0 (unreliable). Here, the region p11 is sandwiched between two pixels p10'corresponding to two pixels of the local region p10, and a rectangular region having four pixels corresponding to four adjacent pixels above and below the two pixels of the local region p10 as vertices. Area (excluding peripheral parts) contained inside (2 ×)
5 pixels). This is because when the local regions arranged in the horizontal direction are adopted, the presence or absence of edges in the vertical direction is determined. The reliability map 1004 shown in FIG. 10C shows a reliability map obtained by performing the above operation on the entire distance image data.

Next, the same operation is performed using the local regions arranged in the vertical direction. In FIG. 10D, since the pixel values in the local region p12 are all 1, the reliability map 1005 is not updated. In FIG. 10E, since the pixel values in the local region p13 are different, the reliability map 1006 is updated so that the reliability in the region p14 corresponding to the local region p13 is 0 (unreliable). Individually, the region p14 corresponding to the local region p13 is defined in the same manner as the region p11 corresponding to the local region p10. The reliability map 1007 shown in FIG. 10F is a final reliability map obtained by performing the above operation on the entire distance image data.

According to this embodiment, the distance boundary after upsampling can be extracted only by obtaining the maximum difference between neighboring pixels in the distance image data. That is, the reliability can be determined with a smaller amount of calculation.

This embodiment can be regarded as an embodiment in which the two evaluation values in the second embodiment are determined as follows. That is, in the present embodiment, the maximum difference between the pixel values in the local region (neighboring pixels) of the distance image data before upsampling corresponding to the reliability calculation target pixel is set as one evaluation value, regardless of the reliability calculation target pixel. It can be considered that 0 is set as the evaluation value of the other.

However, in the present embodiment, for one pixel (reliability calculation target pixel) of the distance image data after upsampling, there are a plurality of corresponding local regions (near regions) of the distance image data before upsampling. There is. Therefore, in the present embodiment, the reliability of one reliability calculation target pixel may be determined based on the evaluation value difference of a plurality of neighboring regions (local regions). When a reliability map with a reliability of 0 is prepared as an initial value (FIG. 8A), the reliability is calculated if any one of the evaluation value differences in the plurality of neighboring regions is equal to or less than the threshold value. The reliability of the target pixel is determined as 1. On the other hand, when a reliability map having a reliability of 1 is prepared as an initial value (FIG. 8B), if any one of the evaluation value differences of the plurality of neighboring regions is larger than the threshold value, the reliability is calculated. The reliability of the target pixel is determined as 0.

(Embodiment 4)
In the first to third embodiments, the captured image data and the distance image data are used to generate reliability data and correct the distance information. On the other hand, in the present embodiment, it is shown that the reliability data can be generated and the correction process can be performed by using other than the captured image data and the distance image data. FIG. 11 shows a functional block diagram of the image data processing device 111 according to the present embodiment. The flowchart of the image processing method performed by the image data processing apparatus 111 according to the present embodiment is the same as the flowchart of the first to third embodiments. The image processing method of the present embodiment will be described focusing on the differences from the first to third embodiments.

The image data processing device 111 includes a first image data input unit 1110, a second image data input unit 1111, an upsampling unit 1112, a reliability data determination unit 1113, and a second image data correction unit 1114. Image data to be used as a reference (hereinafter referred to as first image data) is input to the first image data input unit 1110. The first image data may be any image data. In addition to the luminance image data, the first image data may be, for example, distance image data that has already been upsampled and corrected. Another example of the first image data is image data having information such as infrared light and polarized light. When inputting a luminance image or the like, it may be an imaging device as shown in FIG. 1, and in the case of distance image data or the like, the data input unit may include a data generation unit.

Image data (hereinafter, referred to as second image data) corresponding to the first image data and having a resolution lower than that of the first image data is input to the second image data input unit 1111. The second image data may be any image data as long as the above conditions are satisfied, for example, distance image data, image data having information such as infrared light and polarization, motion image (Optical Flow), and segmentation. The image is mentioned. A motion image is data representing the motion of a subject (sometimes including the motion of a camera). The motion image has, for example, a data format in which the speeds in the horizontal direction (x direction) and the vertical direction (y direction) are held by each pixel. A motion image can generally be generated by taking two luminance images at a certain time interval and calculating a plausible position by template matching of the two images.
The segmentation image is data divided for each object of the subject. For example, it is an image in which a person, a car, a building, etc. in the subject are divided in pixel units and an index is given to each. In addition, the segmentation image is not the unit of an object such as a person or a car, but the data divided according to the attributes such as similar colors. Further, although the distance value is not used such as the foreground and the background or the intermediate layer thereof, the data divided according to the distance is also called a segmentation image.
The second image data may be input from a device other than the image data processing device 111, or may be calculated by the image data processing device 111 from other information including the first image data.

As described above, if the second image data has a lower resolution than the first image data and the first image data is the reference data for the correction, the data processing is performed in the same manner as in the first to third embodiments. Can be done. Depending on the data, there may be a plurality of data to be corrected, but the basic processing does not change for each. For example, in the case of a motion image, the horizontal motion image correction process and the vertical image correction process are independently performed by using the same method.

According to this embodiment, the reliability can be set by applying it to data having an error due to upsampling regardless of the distance image data, and by performing correction processing based on the reliability, it is efficient. Moreover, it is possible to correct the data with higher accuracy.

(Embodiment 5)
In the first to fourth embodiments, the image data processing apparatus acquires or generates the first image data (for example, brightness image data) and the second image data (for example, distance image data) having a resolution lower than that of the first image data. Upsampling was performed on the second image data. In other words, in the first to fourth embodiments, the distance map generation unit 132 and the upsampling unit 133 correspond to the image data acquisition means and acquire the high resolution second image data.

However, it suffices if the enlarged high-resolution second image data can be used, and the acquisition method thereof is not particularly limited. For example, the image data processing device may acquire low-resolution second image data and enlarge it to generate high-resolution second image data. Further, the image data processing device may acquire the enlarged second image data from the external device. In this case, in the first and second embodiments in which upsampling is performed by two types of methods to perform reliability, the image data processing apparatus may perform upsampling processing by only one method different from the input data. .. This is because the reliability map can be generated by using the input post-upsampling image data and the generated post-upsampling image data. Further, in the third embodiment in which the image data after upsampling is not required for the generation of the reliability map, the image data processing apparatus does not have to have an upsampling unit.

Further, when the reliability map is only generated and the correction processing of the second image data after upsampling is not performed, or when the first image data is not used for the correction processing, the image data processing device does not necessarily use the first image. There is no need to get the data.

(Embodiment 6)
The image processing method of the present invention described above can be preferably applied to, for example, an image pickup device such as a digital camera or a camcorder, or an image processing device or a computer that performs image processing on image data obtained by the image pickup device. Further, the technique of the present invention can be applied to various electronic devices (including mobile phones, smartphones, slate terminals, personal computers) incorporating such an image pickup device or an image processing device. In the above embodiment, the configuration in which the function of the image processing device is incorporated in the main body of the image processing device is shown, but the function of the image processing device may be configured in any way. For example, an image processing device may be incorporated into a computer having an image pickup device, the computer may acquire an image captured by the image pickup device, and the image processing method may be executed based on the acquired image. Further, an image processing device may be incorporated into a computer that can access the network by wire or wirelessly, and the computer may acquire a plurality of images via the network and execute the above image processing method based on the acquisition of a plurality of images. .. The obtained distance information can be used for various image processing such as region division of an image, generation of a stereoscopic image or a depth image, and emulation of a blur effect.

The specific mounting on the above device can be either software (program) mounting or hardware mounting. For example, even if a program is stored in the memory of a computer (microcomputer, FPGA, etc.) built in an imaging device or the like and the program is executed by the computer, various processes for achieving the object of the present invention can be realized. Good. It is also preferable to provide a dedicated processor such as an ASIC that realizes all or part of the processing of the present invention by a logic circuit.

13 Image data processing device 132 Distance map generation unit 134 Reliability data determination unit

Claims (18)

Image data obtained with the low-resolution image data, and a high-resolution image data is image data obtained by up-sampling the low-resolution image data, and third image data corresponding to the low resolution image data and the high resolution image data Acquisition method and
A reliability data determining means for determining reliability data representing the reliability of pixel values of the high-resolution image data, and
A correction means for correcting the high-resolution image data and
With
The reliability data determination means is
The first evaluation value determined based on the pixel values of pixels in the vicinity of the pixel position in the low-resolution image data corresponding to the pixel position of the target pixel of the high-resolution image data is different from the first evaluation value. The reliability of the pixel value of the target pixel of the high-resolution image data is determined based on the comparison with the second evaluation value determined based on the pixel value of the neighboring pixel by the method .
The correction means corrects the high-resolution image data using the pixel values of the high-resolution image data based on the reliability data and the third image data.
An image data processing device characterized by this. Further equipped with upsampling means for upsampling image data,
The image data acquisition means acquires the first high-resolution image data by upsampling the low-resolution image data by the upsampling means using the first interpolation method, and obtains the low-resolution image data. The second high-resolution image data is acquired by upsampling by the upsampling means using a second interpolation method different from the first interpolation method.
The reliability data determining means uses the pixel value of the target pixel of the first high-resolution image data as the first evaluation value, and the pixel value of the second high-resolution image data corresponding to the target pixel. Is used as the second evaluation value.
The image data processing apparatus according to claim 1. The first interpolation method is bilinear interpolation, and the second interpolation method is nearest neighbor interpolation.
The image data processing apparatus according to claim 2. The image data acquisition means acquires either the first high-resolution image data or the second high-resolution image data as the high-resolution image data.
The image data processing apparatus according to claim 2 or 3. The correction means corrects only the pixels of the high-resolution image data whose reliability data shows a reliability equal to or less than a threshold value.
The image data processing apparatus according to any one of claims 1 to 4. The correction means sets the pixel value of the target pixel of the high-resolution image data to the brightness information or color information of the third image data corresponding to the target pixel and the first pixel corresponding to the peripheral pixels of the target pixel. Correct by weighting and adding based on the difference from the brightness information or color information of the image data of 3.
The image data processing apparatus according to any one of claims 1 to 5. The low-resolution image data and the high-resolution image data are distance image data, motion image data, or segmentation image.
The image data processing apparatus according to any one of claims 1 to 6. The reliability data determination means is
The weighted average value of the pixel values of the neighboring pixels in the low-resolution image data using the first weighting coefficient is determined as the first evaluation value.
The weighted average value of the pixel values of the neighboring pixels in the low-resolution image data using a second weighting coefficient different from the first weighting coefficient is determined as the second evaluation value.
The image data processing apparatus according to claim 1. The first weighting factor and the second weighting factor are determined according to the distance from the pixel in the low resolution image data corresponding to the target pixel of the high resolution image data.
The image data processing apparatus according to claim 8. One of the first weighting coefficient and the second weighting coefficient is the weight of the pixel closest to the pixel in the low-resolution image data corresponding to the target pixel of the high-resolution image data among the neighboring pixels. Is determined to be 1 and the weights of the other pixels are determined to be 0,
The image data processing apparatus according to claim 9. At least one of the first weighting coefficient and the second weighting coefficient is determined by a Gaussian function according to the distance from the pixel in the low-resolution image data corresponding to the target pixel of the high-resolution image data. ,
The image data processing apparatus according to claim 9 or 10. One of the first weighting coefficient and the second weighting coefficient is determined as a value inversely proportional to the distance from the pixel in the low-resolution image data corresponding to the target pixel of the high-resolution image data.
The image data processing apparatus according to any one of claims 9 to 11. The reliability data determining means sets the difference between the maximum value and the minimum value of the pixel values of the pixels in the vicinity of the pixels in the low resolution image data corresponding to the target pixels of the high resolution image data as the first evaluation value, and sets the target. 0 is set as the second evaluation value regardless of the pixel.
The image data processing apparatus according to claim 1. The reliability is binary information indicating whether the pixel value is reliable or unreliable.
If the difference between the first evaluation value and the second evaluation value is equal to or less than a predetermined threshold value, the reliability data determining means determines the reliability of the target pixel as a reliability indicating that the reliability is reliable. If the difference is larger than the predetermined threshold value, the reliability of the target pixel is determined as the reliability indicating that the target pixel is unreliable.
The image data processing apparatus according to any one of claims 1 to 13. The reliability data determining means determines the reliability of the target pixel so that the larger the difference between the first evaluation value and the second evaluation value, the lower the reliability.
The image data processing apparatus according to any one of claims 1 to 13. With the image sensor
The image data processing apparatus according to any one of claims 1 to 15.
With
The image data acquisition means acquires captured image data from the imaging element, generates the low-resolution image data from the captured image data, enlarges the low-resolution image data, and acquires the high-resolution image data.
Imaging device. This is an image data processing method performed by an image data processing device.
Image data obtained with the low-resolution image data, and a high-resolution image data is image data obtained by up-sampling the low-resolution image data, and third image data corresponding to the low resolution image data and the high resolution image data Acquisition steps and
A reliability data determination step for determining reliability data representing the reliability of pixel values of the high-resolution image data, and
The correction step for correcting the high-resolution image data and
Including
In the reliability data determination step,
The first evaluation value is determined from the nearby pixels of the pixels in the low-resolution image data corresponding to the pixel positions of the target pixels of the high-resolution image data.
A second evaluation value is determined from the neighboring pixels of the pixel in the low-resolution image data corresponding to the pixel position of the target pixel of the high-resolution image data by a method different from the first evaluation value.
Based on the comparison between the first evaluation value and the second evaluation value, the reliability of the high-resolution image data for the target pixel is determined .
In the correction step, the high-resolution image data is corrected using the pixel values of the high-resolution image data based on the reliability data and the third image data.
Image data processing method. A program for causing a computer to execute each step of the image data processing method according to claim 17.

Images