JP2011113527A - Image processing apparatus and method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To allow estimation of depth of higher accuracy. <P>SOLUTION: A super-resolution processing unit 104 of an image processing apparatus 110 converts image data of objects that are acquired by multiple image acquisition devices 102 into high-resolution image data. A depth calculation unit 108 of the image processing apparatus 110 calculates the information on the depth of the objects, based on the high-resolution image data. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a technique for calculating depth information of a subject photographed by a plurality of imaging devices.

  In feature extraction in image recognition and construction of a video component database for synthesizing, producing, processing, and editing video in a flexible and efficient manner, the depth information of the subject is extracted from the captured image data. As a method for obtaining depth information of a subject, a stereo matching method in which two cameras are arranged in parallel is well known.

Here, the stereo matching method will be briefly described with reference to FIG. In FIG. 20, reference numeral 1601 denotes a subject, and reference numeral 1602 denotes a point on the subject 1601. The subject surface has a wave shape. Two cameras using the stereo matching method are the same performance. The camera realizes image formation on the projection surface with light rays passing through the optical center. In FIG. 20, 1603 and 1604 are the optical centers of the left and right cameras. Reference numerals 1612 and 1613 denote the projection surfaces of the left and right cameras. The orthogonal coordinate system with the left projection plane center as the coordinate origin is (u, v). The line connecting the optical center and the projection plane coordinate origin intersects perpendicularly to the projection plane, and the distance between them is the focal length f. An orthogonal coordinate system with the right projection plane center as the coordinate origin is (u ′, v ′). The coordinates (X, Y, Z) on the subject 1601 in the three-dimensional space are connected to the optical center of both cameras, and the contact point with the projection plane is a pixel on the left and right stereo images. Reference numerals 1606 and 1607 in FIG. 20 indicate contact points with the left and right projection planes. The coordinates of 1606 are (u _x , v _y ). The coordinates of 1607 are (u _x ′, v _y ′). Reference numerals 1611 and 1609 denote pixel arrays obtained by enlarging lines 1610 and 1608 on the left and right projection planes. A line connecting the optical centers of both cameras is called a base line, and b is a base line length 1605 that is the length of the base line. The coordinates (X, Y, Z) on the subject 1601 are based on the left camera coordinate system. The depth value of the subject 1601 is Z. Z is calculated by the stereo matching method using the following equation. Patent Document 1 discloses a technique for generating virtual viewpoint image data by converting image data captured by a plurality of imaging devices into super-resolution image data.

JP 2006-119843 A

  However, there are cases where image data acquired by a camera, a video camera, or the like cannot be expressed with a sufficient resolution due to a sensor size limitation or the like. If the number of pixels is insufficient, dense and sharp depth information cannot be obtained even if the depth estimation accuracy by the stereo matching method is high.

  The above problem will be described in detail with reference to FIGS. FIG. 21 is a diagram illustrating a correspondence relationship between an enlarged pixel array (1611 and 1609 in FIG. 20) and an imaging target. Reference numeral 1701 in FIG. 21 is a cross-sectional view of the subject 1601. Reference numeral 1611 denotes a part of one line on the projection surface of the left camera, and image data captured by the left camera is handled as reference image data at the time of depth estimation. Reference numeral 1609 denotes a part of one line on the projection plane of the right camera. The points A, B, C, and D included in the imaging target correspond to the pixels a1, b1, c1, and d1 in the reference image data and the pixels a2, b2, c2, and d2 on the right camera projection plane, respectively. ing. Pixels a1, b1, c1, and d1 are pixels arranged continuously on the reference image data. Pixels a2, b2, c2, and d2 are pixels corresponding to the pixels a1, b1, c1, and d1 on the reference image data on the projection plane of the right camera. The pixels a2, b2, c2, and d2 are arranged on one line on the projection plane of the right camera, but are not necessarily arranged continuously. 1702 shows the result of the depth value Z of points A, B, C, and D obtained using the above equation 1. The horizontal axis represents the pixel positions of points A, B, C, and D. The vertical axis represents the depth value. It can be confirmed that the change in depth between the points A, B, C, and D on the subject cannot be accurately expressed due to the limitation of the image resolution.

  FIG. 22 is a diagram showing the correspondence between the pixels 1611 and 1609 in FIG. 20 and points on the subject 1801. The difference from FIG. 21 is that the number of pixels on one line on the projection surfaces of the left and right cameras has increased. 22 is a cross-sectional view of the subject 1601. Reference numeral 1611 denotes a part of one line on the reference image data. The points A, E, B, F, C, G, and D included in the imaging target are on the projection plane of the right camera and the pixels a1, e1, b1, f1, c1, g1, and d1 on the reference image data. These correspond to the pixels a2, e2, b2, f2, c2, g2, and d2, respectively. Points E, F, and G on the subject 1801 are points located between points A, B, C, and D, respectively. Pixels a1, e1, b1, f1, c1, g1, and d1 are pixels arranged continuously on the reference image data. Pixels a2, e2, b2, f2, c2, g2, and d2 are pixels corresponding to the pixels a1, e1, b1, f1, c1, g1, and d1 on the reference image data on the projection plane of the right camera. . The pixels a2, e2, b2, f2, c2, g2, and d2 are arranged on one line of the projection plane, but are not necessarily arranged continuously. Reference numeral 1802 indicates the result of the depth value Z of the pixels A, E, B, F, C, G, and D obtained using the above equation 1. The horizontal axis represents pixel positions corresponding to points A, E, B, F, C, G, and D. The vertical axis represents the depth value. As a result of the increase in the number of pixels of the image data to be acquired, it is possible to accurately obtain the depth values of the points E, F, and G that could not be obtained by the depth value calculation from the image data of the low pixel number described in FIG. The depth change between the points A to D can be expressed.

  As described above, as a means for increasing the number of acquired depth values, the resolution of image data can be improved by miniaturizing the sensor elements of the imaging system. However, increasing the density of the sensor element directly leads to an increase in manufacturing cost. Furthermore, when the image sensor is made smaller, the amount of light that can be received by one image sensor decreases, and there is a problem that the amount of noise applied to the image data increases.

  Accordingly, an object of the present invention is to enable more accurate depth estimation.

  The image processing apparatus according to the present invention is based on a conversion unit that converts image data of a subject acquired from a plurality of imaging units into high-resolution image data, and the high-resolution image data generated by the conversion unit. And a first calculating unit that calculates depth information of the subject.

  In the present invention, subject image data acquired from a plurality of imaging means is converted into high-resolution image data, and subject depth information is calculated based on the converted high-resolution image data. Therefore, according to the present invention, it is possible to estimate the depth with higher accuracy.

1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention. It is a block diagram which shows the detail of the super-resolution processing unit of FIG. It is a figure which shows the deterioration model of the image acquisition apparatus of FIG. It is a figure which shows the relationship between HR image data and the matrix X showing HR image data. It is a figure which shows the structure of the square matrix B which performs an optical system degradation process. It is a figure which shows the structure of the matrix D reflecting the magnification of the image reduction of a downsampling process. It is a flowchart which shows the process of the super-resolution processing unit shown in FIG. It is a figure which shows the arrangement | positioning of a camera and the image data image | photographed by stereo imaging | photography. It is a figure for demonstrating the matching method of the corresponding point shown in FIG. It is a flowchart which shows a depth calculation process. It is a block diagram which shows the detail of the super-resolution processing unit of FIG. 1 in the 2nd Embodiment of this invention. It is a figure which shows the deterioration model of the image acquisition apparatus of FIG. FIG. 5 is a diagram illustrating a positional relationship between image data when a plurality of left and right image data are acquired for a stereo image. It is a flowchart which shows the process of the super-resolution processing unit of FIG. FIG. 20 is a diagram illustrating a configuration of a matrix M _k illustrated in FIG. 19. It is a figure which shows each matrix size described in Formula 3. It is a figure which shows the matrix size of X, Y, D, and B shown in Formula 1. It is a figure which shows each matrix size described in Formula 10. FIG. Shows X described Equation _{10, Y k, D, B} , a matrix size of M _k. It is a figure for demonstrating the stereo matching method. It is a figure which shows the correspondence of the pixel arrangement | sequence displayed by magnification and imaging | photography object. It is a figure which shows the correspondence of the pixel of FIG. 20, and the point on a to-be-photographed object.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments to which the invention is applied will be described in detail with reference to the accompanying drawings.

  First, a first embodiment of the present invention will be described. FIG. 1 is a block diagram showing a configuration of an image processing apparatus according to the first embodiment of the present invention. Reference numeral 110 denotes an image processing apparatus according to the present embodiment, which receives image data of a subject imaged by the image acquisition device (imaging means) 102 and outputs a depth information map to the subject. The image acquisition device 102 is a stereo camera including a plurality of cameras (camera 1 and camera 2). A plurality of image data acquired by the image acquisition device 102 is temporarily stored in the memory 103. The plurality of image data stored in the memory 103 is input to the super-resolution processing unit 104, and high-resolution image data is created. In addition, deterioration conditions of the image acquisition apparatus 102 necessary for generating high-resolution image data are input from the deterioration condition input unit 105 to the super-resolution processing unit 104. Degradation conditions handled in this embodiment include a point spread function (PSF) representing optical degradation and a downsampling process. The downsampling process is a process of converting image data so that the subject can be accommodated in a limited number of pixels. In addition, the threshold value is input to the super-resolution processing unit 104 from the threshold value input unit 106. The stereo matching unit 107 associates feature points between a plurality of high-resolution image data based on the high-resolution image data generated by the super-resolution processing unit 104, and outputs the feature points to the depth calculation unit. The depth calculation unit 108 calculates a depth information map. The obtained depth information map is output from the information output terminal 109. The depth information map is a data group in which depth values at all pixel positions of captured image data are stored, a grayscale image created using the depth values, and the like. Reference numeral 111 denotes an interface. The depth information map and the high resolution image data generated by the super resolution processing unit 104 are stored in the memory card 112 via the interface 111. The super-resolution processing unit 104 is a configuration that is an application example of the conversion unit of the present invention, and the depth calculation unit 108 is a configuration that is an application example of the first calculation unit of the present invention.

  FIG. 2 is a block diagram showing details of the super-resolution processing unit 104 of FIG. In the following, high resolution image data is abbreviated as HR image data, and low resolution image data is abbreviated as LR image data. In FIG. 2, image data obtained by the image acquisition device 102 from the image input terminal 201 is input to the super-resolution processing unit 104. The image data input from the image input terminal 201 handled in the present embodiment is 8-bit grayscale image data. The initial HR image creation unit 202 creates an initial value of HR image data using the input image data. In this embodiment, an initial value of HR image data is generated using a linear interpolation method. The HR image generation unit 205 updates the HR image data by adding a correction amount calculated by a Maximum A Posterior (MAP) estimation method described later to the HR image data. The LR image creation unit 206 degrades the HR image data according to the degradation condition given from the degradation condition input terminal 209, and generates LR image data. The deterioration condition is downsampling of the deterioration process due to the deterioration PSF of the optical system of the photographing camera and the limitation of the number of sensor pixels. The downsampling degradation condition handled in this embodiment is given by the reduction ratio (1 / M, 1 / N) at the time of downsampling. Note that M and N are natural numbers. These deterioration conditions are assumed to be data measured prior to super-resolution processing or known data. The difference calculation unit 210 performs a difference calculation between the image data input from the image input terminal 201 and the LR image data generated by the LR image generation unit 206. Here, the average value of the difference between the image data input from the image input terminal 201 and the LR image data is calculated. The end determination unit 207 determines whether the generation of the HR image data has been sufficiently performed using the average value of the differences. The threshold value used for the determination is given by the threshold value input terminal 208 and is an arbitrary value from 0 to 255. The end determination unit 207 determines whether or not the generation of the HR image data has been sufficiently performed by comparing the calculation result of the difference calculation unit 210 with the threshold value. When the calculation result of the difference calculation unit 210 is equal to or less than the threshold value, the end determination unit 207 outputs 0. On the other hand, when the calculation result of the difference calculation unit 210 is larger than the threshold value, the end determination unit 207 outputs 1. The calculation result of the difference calculation unit 210 is also input to the correction amount calculation unit 204.

  When 1 is output from the end determination unit 207, the correction calculation unit 204 calculates the correction amount and outputs it. On the other hand, when 0 is output from the end determination unit 207, the correction calculation unit 204 outputs the correction amount 0. The HR image creation unit 205 updates the HR image data based on the correction amount output from the correction amount calculation unit 204. Specifically, the HR image data is updated by adding the correction amount obtained from the correction amount calculation unit 204 to the HR image data. Details of the correction amount calculation will be described later. When the correction amount calculated by the correction amount calculation unit 204 is 0, the HR image generation unit 205 outputs the already generated HR image data from the HR image output terminal 203 as it is. On the other hand, when the correction amount calculated by the correction amount calculation unit 204 is not 0, the HR image generation unit 205 updates the HR image data that has already been generated by adding the correction amount, and processing by the LR image generation unit 206 is performed. Proceed to Thereafter, when the calculation result of the difference calculation unit 210 is equal to or less than the threshold value, the HR image data is output from the HR image output terminal 203, and the generation of the HR image data is completed. Note that the HR image creation unit 205 is an application example of the first generation unit of the present invention. The LR image creation unit 206 is a configuration that is an application example of the second generation unit. Moreover, the difference calculation part 210 is a structure used as the application example of a 2nd calculation means. The correction amount calculation unit 204 is configured as an application example of the third calculation unit of the present invention.

FIG. 3 is a diagram illustrating a deterioration model of the image acquisition device 102 in FIG. 1, and represents a deterioration process of the imaging system in the present embodiment by a linear model. In FIG. 3, reference numeral 301 denotes HR image data before the deterioration process. Reference numeral 302 denotes an optical system process which is a deterioration process by the optical system PSF. Reference numeral 303 denotes sensor processing for downsampling. The downsampling degradation condition is provided as a magnification of image reduction. Reference numeral 304 denotes image data taken by the camera after the deterioration process. Degradation of the optical PSF is assumed to be a linear filter F having a size of (2C + 1, 2C + 1). C is a natural number. The PSF-degraded image data Y _PSF handled in the present embodiment can be expressed by Equation 2 below by convolution calculation. i and j are pixel positions of the two-dimensional image data X and Y _PSF .

  For the convenience of handling in the super-resolution processing described later, the convolution operation of Equation 2 is changed to the linear operation format defined by Equation 3. A conversion method from Equation 2 to Equation 3 will be described later. According to the linear calculation format, the deterioration model of the camera can be expressed by the following Expression 3.

  X is a vector representing the original HR image data. Y is taken image data (LR image data). The matrix B is a square matrix that performs optical system deterioration processing. The matrix D is a matrix reflecting the image reduction magnification of the downsampling process. The sizes of the matrix B and the matrix D vary depending on the input image size. Let the input image size be W (number of horizontal pixels) and H (number of vertical pixels). The downsampling condition is that the image reduction magnification is 1 / M and 1 / N. In that case, each matrix size described in Equation 3 is as shown in FIG.

  The configurations of the matrix X, the matrix B, and the matrix D illustrated in FIG. 16 will be described with reference to FIGS. 4, 5, and 6. FIG. 4 shows the relationship between the HR image data and the matrix X. The horizontal and vertical sizes of the HR image data are W and H. FIG. 4A shows HR image data, and (i, j) is a pixel number of the HR image data. FIG. 4B shows a matrix X in which pixel data is arranged in one column from horizontal to vertical.

  FIG. 5 is a diagram illustrating the configuration of the matrix B. The size of the linear filter is (2N + 1, 2N + 1), and the product is summed with the linear filter for all pixels as shown in Equation 2. As shown in FIG. 5, when the pixel (i, j) is at the center of the linear filter, the value of the filter is substituted into the corresponding position of the matrix in the matrix i × j row. A value other than 0 is substituted with 0.

  FIG. 6 is a diagram illustrating the configuration of the matrix D. The function of the matrix D is a matrix for obtaining pixel values by using the reduced size and thinning out the pixel positions of the image. For the matrix value, 1 is substituted into the position where the pixel value exists, and 0 is substituted into the other positions. Here is one specific example. The image data of size 1024 × 1024 is assumed to be HR image data, and the reduction magnification according to the downsampling condition is 1/2 and 1/2. Then, the size of the image data to be shot is 512 × 512. FIG. 17 shows the matrix sizes of X, Y, D, and B shown in Equation 1 above.

  In the present embodiment, the HR image data is obtained using a MAP method (Maximum A Posterior). The MAP method is a method for estimating HR image data by minimizing an evaluation function obtained by adding probability information of HR image data to a square error. This is a super-resolution processing method that estimates HR image data as an optimization problem that maximizes the posterior probability by using certain prior information for the HR image data. In the present embodiment, HR image data is generated using the following Equation 4. The MAP method is described in IEEE Signal Processing MAGAZINE 2003/03 “Super-Resolution Image Reconstruction: A technical Overview” author Sung Cheol Park, Min Kyu Park and Moon Gi Kang.

  The first term on the right side is a constraint term that takes into account prior information that the pixel values of adjacent pixels often have similar values. The first term on the right side has the effect of smoothing the entire image data. A Laplacian filter is used as the linear filter C. α is a parameter for adjusting the degree of smoothing in the image data of the first term on the right side. Increasing the value of α is effective for obtaining image data with a high degree of smoothing. The second term on the right side is a term for calculating a difference value between the image data and the image data (LR image data) estimated through the deterioration process. σ is a standard deviation of noise of the LR image data Y acquired by the image acquisition device 102. Expression 5 is an expression that extracts the right side of Expression 4 and uses it as an evaluation function during super-resolution processing.

  FIG. 7 is a flowchart showing processing of the super-resolution processing unit 104 shown in FIG. In step S <b> 701, the initial HR image creation unit 202 inputs image data Y from the image input terminal 201. In step S <b> 702, the LR image creation unit 206 inputs a matrix B that is an optical deterioration condition of the camera from the deterioration condition input terminal 209. In step S <b> 703, the LR image creation unit 206 inputs the camera downsampling matrix D from the deterioration condition input terminal 209. In step S704, the initial HR image creation unit 202 generates an initial value of HR image data. Here, the initial value of the HR image data is enlarged by M times the horizontal size and N times the vertical size by linear interpolation. In step S <b> 705, the end determination unit 207 inputs a threshold value used when determining the achievement level of the super-resolution processing from the threshold value input terminal 208. In step S <b> 706, the LR image creation unit 206 creates LR image data in consideration of optical degradation PSF and downsampling from the HR image data created by the initial HR image creation unit 202 or the HR image creation unit 205. Here, LR image data is generated using Equation 6.

Here, HR image data generated by the i-th calculation is X _i . In step S707, the difference calculation unit 210 calculates differences for all the pixels of the LR image data obtained in step S706 and the image data input from the image input terminal 201, and the end determination unit 207 calculates the difference. The average value is compared with the threshold value. If the average value of the differences is less than or equal to the threshold value, the process proceeds to step S711. If the average value of the differences is larger than the threshold value, the process advances to step S708, and the correction amount calculation unit 204 obtains an update term for calculating the correction amount of the HR image data. The method for obtaining the update term will be described later. In step S709, the HR image creation unit 205 counts the number of loops. In step S710, the HR image creation unit 205 adds the result of the update term calculated in step S708 to the HR image data to update the HR image data. The processes in steps S706 to S710 are repeated until the average value of the differences calculated in step S707 is less than or equal to the threshold value.

  In step S710, the HR image creation unit 205 updates the HR image data in response to the evaluation result of the evaluation function. In the present embodiment, the HR image data is updated using the steepest descent method. The steepest descent method is an optimization method using the first derivative of the function to be minimized. The update term (first derivative of the function) required for the steepest descent method can be expressed by Equation 7.

  FIG. 8 is a diagram showing the arrangement of cameras and image data taken by stereo shooting. In FIG. 8, the camera optical axes of the two cameras A802 and B801 are parallel. A camera in such a state is called a parallel camera. The image data captured by the camera A 802 is 807. The image data captured by the camera B 801 is 806. The image data 807 is handled as reference image data at the time of stereo matching processing described later. Image data 806 is handled as reference image data. A point 805 in the standard image data 807 corresponds to a point 804 in the reference image data 806.

  FIG. 9 is a diagram for explaining a method for associating corresponding points shown in FIG. In the present embodiment, the corresponding points are calculated using the stereo matching method. The stereo matching method uses one image data as reference image data, and calculates a pixel position by searching a point on the same subject from the other reference image data for a pixel in the reference image data. It is. In FIG. 9, the left image is reference image data at the time of stereo matching. The right image is reference image data. In FIG. 9, reference numeral 902 denotes a block used for stereo matching calculation. The horizontal and vertical size of the block is nxn pixels. Here, the reason why the block of nxn pixels is used for detection of the target point is preferable in consideration of the correlation between the pattern characteristics of the surrounding pixels of the pixel k and the pattern characteristics of the surrounding pixels of the reference image target point k ′. This is to achieve a search for a corresponding point. In the case of parallel stereo, a corresponding point on the right image of a certain point on the reference image data exists on a straight line having the same height as a certain point on the reference image data. At all pixel positions on the corresponding straight line, the correlation in the image region surrounded by the block 902 and the block 903 is calculated, and the combination that maximizes the correlation is set as the corresponding point. The correlation value uses a correlation coefficient. The central pixel of the block of reference image data showing the highest correlation coefficient is detected as the corresponding point k ′ of the pixel k.

FIG. 10 is a flowchart showing the depth calculation process. The search for corresponding points between the reference image data and the reference image data employs the stereo matching method described above. In FIG. 10, corresponding points on the reference image data are searched for all the pixels of the standard image data from step S1001 to step S1006. In step S1002, the stereo matching unit 107 obtains a correlation coefficient in units of nxn pixel blocks in the standard image data and reference image data that are left and right stereo images. In step S1003, the stereo matching unit 107 obtains a corresponding pixel position on the reference image data that maximizes the calculated correlation coefficient. When the pixel position on the reference image data is (x _i , y _i ) and the corresponding pixel position on the reference image data is (x _r , y _r ), the parallax between both pixels is | x _i −x _r | Can express. In step S1004, the depth calculation unit 108 obtains parallax information. In step S1005, the depth calculation unit 108 calculates a depth value. The depth value Z is obtained by Expression 8 using the parallax | x _i −x _r |, the distance b between the two cameras, and the focal length f of the camera.

  In this embodiment, the super-resolution processing is performed on the image data using the MAP method. However, the resolution of the image data may be improved by other software processing. For example, edge enhancement may be performed after interpolation processing. Further, the image data handled in the present embodiment has been described as grayscale image data, but color RGB image data may be used. When color RGB image data is used, depth estimation may be performed using an arbitrary channel, or depth estimation may be performed after converting RGB information into luminance and brightness information.

  As described above, according to the first embodiment of the present invention, the depth value (depth information map) is acquired using the HR image data generated from the left and right image data captured by the stereo camera. I have to. Therefore, it is possible to obtain a depth value (depth information map) having a large amount of information spatially.

  Next, a second embodiment of the present invention will be described. In the first embodiment described above, the HR image data is generated from the left and right image data captured by the stereo camera, and then the depth value of the subject is estimated. On the other hand, in the second embodiment, one HR image data is generated from a plurality of image data acquired from each imaging device, and a depth value is acquired using the HR image data. Since the HR image data used for stereo matching is composed of a plurality of pieces of image data, the image data has a large amount of information, that is, high resolution, and depth values can be estimated with higher accuracy. Hereinafter, only a part that is different from the first embodiment and generates one piece of HR image data from a plurality of pieces of image data will be described.

FIG. 11 is a block diagram showing details of the super-resolution processing unit 104 of FIG. 1 in the second embodiment. In FIG. 11, N pieces of image data are input from an image input terminal 1101. N is an integer of 2 or more. The image data handled in this embodiment is 8-bit grayscale image data. An initial HR image creation unit 1102 creates an initial value of HR image data using the input image data. In this embodiment, an initial value of HR image data is created using a linear interpolation method. In FIG. 11, reference numeral 1105 denotes an HR image generation unit. Here, the HR image data is updated by adding the correction amount calculated by the MAP estimation method described above to the HR image data. Reference numeral 1106 denotes an LR image creation unit that degrades HR image data in accordance with a degradation condition given from a degradation condition input terminal 1109 to generate LR image data. The deterioration condition is downsampling, which is a deterioration process due to the deterioration PSF of the optical system of the photographing camera and the limitation of the number of sensor pixels. In this embodiment, the downsampling degradation condition handled is given by the reduction ratio (1 / M, 1 / N) at the time of downsampling. Note that M and N are natural numbers. Further, information indicating a positional shift between a plurality of pieces of image data is input from the deterioration condition input terminal 1109. In the present embodiment, horizontal and vertical misalignments are handled. Of the N pieces of acquired image data, the first piece of image data is used as reference image data, and the relative movement amount from the reference image data is used as positional deviation information. The center image position of the reference image data is set as the origin, and horizontal and vertical displacement amounts (Δx _k , Δy _k ) with respect to the origin are calculated. Note that k = 2, 3,..., N. In the present embodiment, the amount of positional deviation (Δx _k , Δy _k ) is calculated by inter-frame matching with reference image data.

  In the present embodiment, these deterioration conditions are data measured in advance prior to super-resolution processing or known data. The difference calculation unit 1110 calculates the difference between the image data input from the image input terminal 1101 and the LR image data generated by the LR image generation unit 1106. Then, the difference calculation unit 1110 calculates the average value of the differences by dividing the sum of the difference information between the obtained image data by the number of pixels. The end determination unit 1107 determines whether HR image data has been generated using the average value. The threshold is a certain value from 0 to 255 given by the threshold input terminal 1108. The end determination unit 1107 compares the calculation result of the difference calculation unit 1110 with a threshold value. When the calculation result of the difference calculation unit 1110 is equal to or less than the threshold value, the end determination unit 1107 outputs 0. On the other hand, when the calculation result of the difference calculation unit 1110 is larger than the threshold, the end determination unit 1107 outputs 1. In addition, the calculation result of the difference calculation unit 1110 is input to the correction amount calculation unit 1104. When 1 is output from the end determination unit 1107, the correction amount calculation unit 1104 calculates and outputs the correction amount. On the other hand, when 0 is output from the end determination unit 1107, the correction amount calculation unit 1104 outputs the correction amount 0. The HR image creation unit 1105 generates HR image data according to the correction amount output from the correction amount calculation unit 1104. Specifically, the HR image data is updated by adding the correction amount obtained by the correction amount calculation unit 1104 to the HR image data. Details of the correction amount calculation will be described later. When the correction amount calculated by the correction amount calculation unit 1104 is 0, the HR image generation unit 1105 outputs the already created HR image data as it is from the HR image output terminal 1103. On the other hand, when the correction amount of the correction amount calculation unit 1104 is not 0, the HR image generation unit 1105 updates the HR image data that has already been generated by adding the correction amount, and proceeds to processing by the LR image generation unit 1106. Thereafter, when the calculation result of the difference calculation unit 1110 becomes equal to or less than the threshold value, the HR image data is output from the HR image output terminal 1103, and the generation of the HR image data is completed.

FIG. 12 is a diagram illustrating a deterioration model of the image acquisition device 102 in FIG. 1, and represents a deterioration process of the imaging system in the present embodiment by a linear model. In FIG. 12, reference numeral 1201 denotes HR image data before the deterioration process. Reference numeral 1202 denotes a positional deviation between a plurality of pieces of image data. An optical system process 1203 is a deterioration process by the optical system PSF. Reference numeral 1204 denotes sensor processing for performing downsampling. The downsampling degradation condition is provided as a magnification of image reduction. Reference numeral 1205 denotes image data acquired by the camera. Degradation of the optical PSF is assumed to be a linear filter F having a size of (2N + 1, 2N + 1). N is a natural number. The PSF-degraded image data Y _PSF handled in the present embodiment can be expressed by Equation 9 below by convolution calculation. i and j are pixel positions of the two-dimensional image data X and Y _PSF .

  For the convenience of handling in the super-resolution processing described later, the convolution operation of Equation 9 is defined by Equation 10. Change to the linear operation format. A conversion method from Equation 9 to Equation 10 will be described later. According to the linear calculation format, the camera deterioration model can be expressed by the following equation (10).

X is a vector representing the original HR image data. Y _k is taken image data (LR image data). The matrix M _k is a square matrix that performs positional deviation correction. The matrix B is a square matrix that performs optical system deterioration processing. The matrix D is a matrix reflecting the image reduction magnification of the downsampling process. The sizes of the matrix M _k , the matrix B, and the matrix D vary depending on the input image size. Let the input image size be W (number of horizontal pixels) and H (number of vertical pixels). The downsampling condition is that the image reduction magnification is 1 / M and 1 / N. In that case, each matrix size described in Equation 10 is as shown in FIG.

X is a vector representing the original HR image data, and the horizontal and vertical sizes are W and H, respectively. Y _k is taken LR image data. The image reduction magnification of the downsampling process is 1 / M and 1 / N. The positional deviation between the image data is (Δx _k , Δy _k ). The size of the matrix depends on the image size.

Here is one specific example. The image data of size 1024 × 1024 is assumed to be HR image data, and the reduction magnification by downsampling is 1/2 and 1/2. Then, the size of the LR image data is 512 × 512. The displacement is assumed to be (Δx _k , Δy _k ). The matrix sizes of X, Y _k , D, B, and M _k described in Equation 10 are shown in FIG. FIG. 15 shows the configuration of the matrix M _k shown in FIG. In the present embodiment, the HR image data is obtained using the MAP method. In other words, in the present embodiment, HR image data is generated using the following Expression 11.

The positional deviation information with respect to the reference image data is indicated by a matrix M _k . Deterioration processing by the optical system PSF is represented by a matrix B. A matrix reflecting the image reduction magnification of the downsampling process is represented by a matrix D. The second term on the right side of Equation 11 is a term for calculating a difference value between the input image data and the LR image data estimated through the deterioration process. σ is a standard deviation of noise of the image data Y acquired by the image acquisition device 102. Expression 12 is an expression that extracts the right side of Expression 11 and uses it as an evaluation function during super-resolution processing.

FIG. 13 is a diagram illustrating a positional relationship between image data when a plurality of left and right image data are acquired for a stereo image. In the present embodiment, the positional deviation between the first and k-th image data for each of the standard image data and the reference image data is assumed to be as small as (Δx _k , Δy _k ), and the camera shake of the stereo camera or the like is assumed. A plurality of image data that are assumed to be finely controlled are used.

FIG. 14 is a flowchart showing processing of the super-resolution processing unit 1111 in FIG. In step S _< b> 1401, the initial HR image creation unit 1102 inputs image data Y _k from the image input terminal 201. Y _k represents input image data. In step S <b> 1402, the LR image creation unit 1106 inputs a matrix M _k including positional deviation information between the acquired plurality of pieces of image data from the deterioration condition input terminal 1109. In step S1403, the LR image creation unit 1106 inputs a matrix B, which is an optical deterioration condition of the camera, from the deterioration condition input terminal 1109. In step S <b> 1404, the LR image creation unit 1106 inputs the downsampling matrix D of the camera from the deterioration condition input terminal 1109. In step S1405, the HR image creation unit 205 generates an initial value of HR image data. The initial value of the HR image data is enlarged to M times the horizontal size and N times the vertical size by linear interpolation. In step S1406, the end determination unit 1107 inputs a threshold value from the threshold value input terminal 1108. In step S 1408, the LR image creation unit 1107 creates LR image data Y _k ′ according to Equation 13. Step S1408 is repeatedly executed for the number of image data input in step S1401. Accordingly, from step S1407 to step S1409, the LR image data Y _k ′ corresponding to the number of sheets is generated.

The HR image data generated by the i-th calculation is X _i , and the HR image data X _i is generated in step S1412 by repeated calculation from step S1407. In step S1413, the difference calculation unit 1110 calculates an average value for all pixels and all input image data of the difference between the LR image data obtained by the calculation in step S1408 and the image data input from the image input terminal 1101. Ask. Then, end determination unit 1107 compares the average value with a threshold value. If the average value of the differences is equal to or less than the threshold value, the process proceeds to step S1414, and the repetitive calculation is finished. If the average value of the differences is larger than the threshold value, the process advances to step S1410, and the correction amount calculation unit 1104 obtains an update term for calculating the correction amount of the HR image data. The method for obtaining the update term will be described later. In step S1411, the HR image creation unit 1105 counts the number of loops. In step S1412, the HR image creation unit 1105 adds the result of the update term calculated in step S1410 to the HR image data to update the HR image data. Steps S1407 to S1413 are repeated until the average value of the differences calculated in step S1413 is less than or equal to the threshold value.

In step S1410, the correction amount calculation unit 1104 calculates an update term for the HR image data. The HR image creation unit 1105 updates the HR image data in response to the result of the evaluation function. In the present embodiment, the HR image data is updated using the steepest descent method. The steepest descent method is an optimization method that uses only the first derivative of the function to be minimized. When the steepest descent method is applied to Expression 12, which is an evaluation function, the update term ΔX _i of the HR image data is calculated by Expression 14.

  As described above, according to the second embodiment of the present invention, one piece of HR image data is generated from a plurality of image data acquired from each imaging device, and a depth value ( Since the depth information map) is acquired, more accurate depth estimation is possible.

  As described above, in the above-described embodiment, it is possible to generate HR image data from a plurality of pieces of image data captured by a plurality of imaging devices, and to calculate denser depth information based on parallax information. In particular, according to the first embodiment, the depth value (depth information map) is acquired using the HR image data generated from the left and right image data photographed by the stereo camera, so that the amount of information is spatially large. It is possible to obtain depth information. Further, according to the second embodiment, one HR image data is generated from a plurality of image data acquired from each imaging device, and depth information is acquired using the HR image data. Accurate depth estimation is possible. By processing based on the super-resolution image and the depth map of the stereo image, an effect of blurring the background or performing 3D display can be obtained.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

  103 Memory, 104 Super-resolution processing unit, 105 Degradation condition input unit, 106 Threshold input unit, 107 Stereo matching unit, 108 Depth calculation unit, 110 Image processing device, 202 Initial HR image creation unit, 204 Correction calculation unit, 205 HR Image creation unit, 206 LR image creation unit, 207 end determination unit, 210 difference calculation unit

Claims (11)

Conversion means for converting image data of a subject acquired from a plurality of imaging means into high-resolution image data;
An image processing apparatus comprising: a first calculation unit that calculates depth information of the subject based on high-resolution image data generated by the conversion unit.   The image processing apparatus according to claim 1, wherein the conversion unit converts the image data of the subject into high-resolution image data in consideration of characteristics of the imaging unit. The converting means includes
First generation means for generating high-resolution image data from image data of the subject acquired from the imaging means;
Second generation means for generating low-resolution image data in consideration of characteristics of the imaging means from high-resolution image data generated by the first generation means;
Second calculation means for calculating a difference between the low-resolution image data generated by the second generation means and the image data acquired by the imaging means;
The high-resolution image data generated by the first generation unit when the difference calculated by the second calculation unit satisfies a predetermined condition is output. Image processing apparatus. The converting means includes
When the difference calculated by the second calculation means does not satisfy a predetermined condition, the apparatus further includes third calculation means for calculating a correction amount according to the difference,
The image processing apparatus according to claim 3, wherein the first generation unit updates high-resolution image data according to the correction amount calculated by the third calculation unit.   5. The image processing apparatus according to claim 3, wherein the predetermined condition is satisfied when a difference calculated by the second calculation unit is equal to or less than a predetermined threshold value.   6. The characteristic of the imaging means includes at least one of a characteristic of an optical system in the imaging means and a characteristic related to downsampling processing of the imaging means. The image processing apparatus according to item 1.   The image processing apparatus according to claim 6, wherein the characteristic of the optical system is a point spread function.   The image processing apparatus according to claim 1, wherein the conversion unit generates one piece of high resolution image data from one piece of image data.   The image processing apparatus according to claim 1, wherein the conversion unit generates one piece of high-resolution image data from a plurality of image data. A conversion step of converting image data of a subject acquired from a plurality of imaging means into high-resolution image data;
A calculation step of calculating depth information of the subject based on the high-resolution image data generated by the conversion step. A conversion step of converting image data of a subject acquired from a plurality of imaging means into high-resolution image data;
A program for causing a computer to execute a calculation step of calculating depth information of the subject based on the high-resolution image data generated by the conversion step.

Images