JP2010073035A - Image generation apparatus, image generation method, image generation program, and computer readable recording medium recording the program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a high-resolution image whose dynamic range is expanded without increasing the number of pickup images to be used. <P>SOLUTION: An image generation apparatus includes: a dynamic range expansion/super resolution processing part 35 which acquires a plurality of pickup image data picked up at image pickup positions set by an image pickup position determination device 10 and generates high-resolution image data; and the image pickup position determination device 10 including an image pickup position calculation part 12 which, when virtually setting reference image pickup positions in which reference pixel regions each of which has a shape corresponding to one pixel of the pickup image data are arrayed like a matrix, sets the image pickup positions of the pickup image data so that a center region having a shape corresponding to one pixel of the high-resolution image data which is positioned in the center of the shape corresponding to one pixel of the pickup image data may be included in the reference pixel region, and in pickup image data picked up under one image pickup condition out of a plurality of sorts of image pickup conditions, sets a first image pickup position for uniforming arranging the center regions. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention provides an image generation device that generates one high-resolution image with an increased resolution from a plurality of low-resolution images obtained by imaging the same imaging target under any of a plurality of types of imaging conditions while varying the imaging position. Etc.

  The resolution of an image captured by an imaging device such as an area sensor depends on the density of the image sensor that the imaging device has. Users of imaging devices are demanding imaging devices that can capture images with higher resolution, and camera manufacturers have released high-resolution cameras with high imaging element density one after another in order to meet user requirements for resolution. Yes.

  However, there is a limit to increasing the density of the image sensor, and there is a problem that the manufacturing cost of the imaging device increases due to the increase in density. Therefore, a plurality of times of imaging are performed on the same imaging target while shifting the imaging position little by little, and a plurality of captured image data obtained by imaging are synthesized, and the resolution is higher than that of the captured image data. Conventionally, a high-resolution technique for generating resolution image data has been used.

  Note that “shifting the imaging position” means that the imaging surface of the imaging device (the surface configured by arranging the imaging elements) and the imaging target surface to be imaged remain parallel to each other in the parallel direction. This refers to changing the relative position between the imaging surface and the imaging surface. Thereby, each of the picked-up image data reflects different regions to be picked up. That is, the high resolution image data is generated based on a plurality of captured image data in which the imaging regions are shifted.

  Image resolution enhancement techniques are generally divided into two types: image shift processing and super-resolution processing. In the image shift process, imaging is performed a plurality of times while gradually shifting the imaging position for the same imaging target. Then, the correspondence between the positions of the pixels included in each of the plurality of captured image data and the pixels of the high resolution image data is obtained, and the luminance value of each pixel of the captured image data is mapped to each pixel of the high resolution image data. Thus, high-resolution image data reflecting a plurality of captured image data is generated.

  In the image shift process, since it is necessary to map all the pixels of the captured image data to the pixels of the high resolution image data, it is necessary to fix the imaging position and the number of captured images of the captured image data. Therefore, the image shift process has an advantage that high-resolution image data can be generated at a low calculation cost, but has a problem that it is difficult to dramatically improve the resolution.

  On the other hand, in the super-resolution processing, first, similarly to the image shift processing, imaging is performed a plurality of times while gradually shifting the imaging position for the same imaging target. In the super-resolution processing, high-resolution image data is estimated, and the assumed high-resolution image data is reduced so that the difference between the estimated high-resolution image and the plurality of captured image data obtained by the plurality of imaging operations is reduced. Update the resolution image data.

  Such a super-resolution process is called a reconfigurable super-resolution process. As reconstruction-type super-resolution processing, for example, ML (Maximum-likelihood) method, MAP (Maximum A Posterior) method, POCS (Projection On to Convex Set) method and the like are generally known. 1 and Patent Document 1.

  In reconstruction super-resolution processing, the number of low-resolution image data (captured image data) to be used is not limited, so that the resolution can be dramatically improved. On the other hand, in the reconfigurable super-resolution processing, since iterative calculation is used for updating high-resolution image data, the calculation cost is generally higher than that of the image shift processing.

  Hereinafter, the conventional super-resolution processing will be described in more detail. The super-resolution processing (reconstruction type) is formulated as an optimization problem of the evaluation function defined for the generated high resolution image. In other words, the super-resolution processing is an optimal evaluation function (error term) based on the square error between “estimated captured image data (high-resolution image data)” and “observed captured image data (captured image data)”. It will be reduced to the problem. Note that since the super-resolution processing is an optimization calculation with a very large number of unknowns, for example, an iterative calculation method such as the steepest descent method is often used to optimize the evaluation function.

  When the characteristics of the high resolution image data are known, an evaluation function (constraint term) based on the characteristics of the high resolution image data may be added to the evaluation function (error term). For example, when it is known in advance that the edge component of the high resolution image data is small, an evaluation function (constraint term) based on image data obtained by multiplying the high resolution image data by a Laplacian filter may be added. As a result, the fact that the edge component of the high-resolution image data is small can be reflected in the super-resolution processing, so that the accuracy of increasing the resolution can be improved.

Here, the flow of processing in the conventional super-resolution processing method will be described with reference to FIG. FIG. 17 is a flowchart showing an example of a conventional super-resolution processing method. Here, it is assumed that one piece of high-resolution image data is generated from m pieces of picked-up image data obtained by picking up the same image pickup object while shifting the image pickup position little by little (m is a positive number). ). Note that m is determined according to how many times the resolution of the captured image data is increased. Specifically, in order to increase the resolution of the captured image data by n times, for example, there may be N ² captured image data.

Note that even when the number of captured image data is less than N ^2, it is possible to increase the resolution N times, but in this case, the resolution is higher than when N ² captured image data is used. The accuracy of resolution is lowered. Conversely, by performing a resolution of N times by using more captured image data from the ^two N, it is also possible to improve the resolution of accuracy as compared with the case of using the captured image data of the ^two N is there.

  Here, the imaging method of the imaging target when acquiring the captured image data will be described with reference to FIG. FIG. 18 is a schematic diagram illustrating an imaging method of captured image data used for super-resolution processing. In the figure, for simplicity, an example of generating one piece of high-resolution image data using nine pieces of picked-up image data, that is, an example in which m = 9 and the picked-up image data is three times higher in resolution. Show.

  18A to 18I show a reference imaging region T1 indicating an imaging region of one imaging device at a certain imaging position, and imaging of the imaging device when the imaging position is moved from the reference imaging region T1. The positional relationship with the area | regions P11-P91 is shown. That is, in FIG. 18, the rectangular sizes of P11 to P91 and T1 indicate the imaging region of one imaging element.

  Further, in the center of the imaging regions P11 to P91, rectangles P13 to P93 having a size corresponding to one pixel of the high resolution image data are illustrated. This indicates that the luminance value data acquired in the imaging regions P11 to P91 is reflected on the positions indicated by the rectangles P13 to P93 on the high resolution image data. In the figure, since it is assumed that the resolution is increased three times, the length of one side of the rectangles P13 to P93 is set to one third of the length of one side of the imaging regions P11 to P91.

  In the case of increasing the resolution of the captured image data to three times, as shown in the figure, the width of the reference imaging area T1 is 1/3 of the reference imaging area T, that is, one pixel of the captured image data is captured. It is only necessary to perform imaging once at each imaging position while moving the imaging position so that the imaging area moves with a width of 1/3 of the imaging area to be performed.

  As a result, nine rectangles P13 to P93 are arranged in each region obtained by dividing the reference imaging region T into a 3 × 3 matrix as illustrated. That is, the luminance value data captured in the imaging regions P11 to P91 is reflected in each region obtained by dividing the reference imaging region T, which is an imaging region for one imaging element, into a 3 × 3 matrix, so that the number of pixels is nine. It is possible to generate high-resolution image data having double, that is, three times the resolution.

  Note that in order to generate high-resolution image data that accurately reflects the shape and color of the imaging target, it is necessary to capture each captured image data under the same imaging conditions. Here, the imaging condition refers to at least one of exposure time (shutter speed), illumination intensity, imaging sensitivity, filter to be used, and iris.

Here, the description returns to the flowchart. In the conventional super-resolution processing method, m pieces of picked-up image data ym from the first piece to the m-th piece obtained as described _above are acquired (S101). Incidentally, y _m is a representation of the m pieces of image data to m-th first picture above as a vector is expressed by the following equation (1).

  Note that the vector representation of the image data refers to a representation of the image data components as a single row vector arranged in a line as in the above equation (1). For example, when the size of the elements constituting the image data is W × H, the image data is expressed as a vector of 1 row × (W × H) columns.

  Here, as described above, the super-resolution processing is an evaluation function based on a square error between “estimated captured image data (high-resolution image data)” and “observed captured image data (captured image data)”. This results in an optimization problem of (error term). That is, in order to perform the super-resolution processing, a plurality of captured image data and high-resolution image data estimated from the plurality of captured image data are necessary.

However, at the stage of obtaining the captured image data y _m, it can not acquire the high-resolution image data to be estimated from the captured image data y _m. Therefore, in the next step, initial high-resolution image data h is acquired (S102). The initial high-resolution image data h is image data having a higher resolution than the captured image data (the number of pixels constituting the image is large). The initial high resolution image data h is a representation of the high resolution image data as a vector.

  An initial value may be set in advance for the initial high-resolution image data h. When the resolution of the captured image data is increased to 3 times, the size of each pixel of the high-resolution image data having a higher resolution than the captured image data is 1/3 times the size of each pixel of the captured image data. Become.

  Note that the initial high-resolution image data is updated so that the difference from the captured image data is reduced by the super-resolution processing, so the initial value of the high-resolution image data, that is, the luminance value of each pixel of the initial high-resolution image data. May be any value. For example, a gray image, black image, or white image with a uniform entire surface can be used as the initial high resolution image data. Further, any one selected from the captured image data may be used as the initial high resolution image data.

Here, in the initial high-resolution image data h and captured image data y _m, vary in size and number of pixels in each pixel, also have different imaging position of the imaging subject. Therefore, it is not possible to calculate directly the difference between the high resolution image data h and captured image data y _m. Therefore, when acquiring the captured image data y _m and the high resolution image data h at S101 and S102, the high-resolution image data h to the lower resolution with aligning the captured image data y _m, m pieces of the pseudo low resolution image Data B _m h is generated (S103).

Incidentally, B _m is a matrix of low resolution with a high-resolution image data h to align the captured image data y _m. B _m is a matrix represented by the product of M _{1 to} M _m and C, as shown in the following formula (2). Then, the matrix M ₁ ~M _m, after a high-resolution image data h are aligned at the imaging position of the captured image data y _m, sampling the luminance value of the pixel corresponding to each pixel position of the captured image data y _m processing Is a matrix. C is a matrix representing a convolution (a process of convolutioning and blurring an image) using a point spread function (PSF) that causes blur in the observation model (camera model of the imaging apparatus).

Next, the difference between the pseudo low resolution image data B _m h generated in S103 and the captured image data y _m is calculated, and the difference image data X _m is calculated (S104). This difference image data _Xm is an error between “estimated captured image data (high resolution image data)” and “observed captured image data (captured image data)”.

Then, by substituting the evaluation function difference image data X _m calculated above, obtaining the evaluation value E that indicates the high resolution accuracy (degree to which the high resolution image data and the captured image data are different) (S105 ). Various known functions can be applied as the evaluation function. For example, the evaluation value E can be obtained by using the evaluation function shown in the following mathematical formula (3).

The above mathematical formula (3) is an evaluation function used in a super-resolution processing method called a MAP method. Equation (3) is an equation composed of an error term and a constraint term, and B _m h−y _{m on} the right side of Equation (3), that is, difference image data X _m is an error term. A term including the characters w, L, and h on the right side of Expression (3) is a constraint term. Note that L is a matrix representing prior probability information of the high-resolution image data h, and w is a parameter representing the strength of the constraint term. By incorporating such a constraint term into the evaluation function, more accurate super-resolution processing can be performed by reflecting the prior probability information of the high-resolution image data.

Value obtained by substituting the difference image data X _m in this formula (3) is an evaluation value E. The value of the error term as the high-resolution image data h is accurately reflect the captured image data y _m decreases, the evaluation value E is also reduced accordingly. Therefore, it can be said that the super-resolution processing is processing for searching for high-resolution image data h that minimizes the evaluation value E.

  Next, the calculated evaluation value E is compared with a preset threshold value (S106). When the evaluation value E is smaller than the threshold value (YES in S106), the high resolution image data h used for calculating the evaluation value E is finally obtained as a result of the super-resolution processing. The super-resolution processing is finished as the high-resolution image data h.

  The threshold value can be changed as appropriate according to the required accuracy of resolution enhancement. For example, when higher resolution and higher resolution are required, the threshold value may be set small, and higher resolution accuracy is not required, but when a rapid processing speed is required, the threshold value is set. Should be set larger. That is, the threshold value is a value for determining the accuracy of the super-resolution processing, and can be set as appropriate according to various conditions required for the super-resolution processing.

  On the other hand, when the value of the evaluation value E is equal to or greater than the threshold value (NO in S106), the high-resolution image data h is updated by repeatedly performing the calculation so that the value of the evaluation value E becomes small (S107). For example, the high resolution image data h can be updated by using the following mathematical formula (4).

Formula (4) is a formula used in the steepest descent method. According to equation (4), made from the high resolution image data h _k, by subtracting a value obtained by multiplying the weight β determined in advance in the difference image data, as the high resolution image data h _{k + 1} new is generated ing. By repeating this calculation, high-resolution image data h having a smaller evaluation value E is generated. The iterative calculation is not limited to the steepest descent method, and a known iterative calculation method can also be applied.

  The high-resolution image data h updated in this manner is subjected to pseudo-resolution reduction in S103 again, and is used for calculating the evaluation value E in S104 and S105. In S106, the evaluation value is compared with the threshold value again. That is, in the super-resolution process, the processes of S103 to S107 are repeatedly performed until the evaluation value E is smaller than the threshold value. Then, the high-resolution image data h obtained when the evaluation value E is smaller than the threshold value is used as the final high-resolution image data h obtained as a result of the super-resolution processing, and the super-resolution processing ends. As a result, high-resolution image data h with high resolution at a desired magnification and desired accuracy can be obtained.

The above is the conventional super-resolution processing technique. Through such super-resolution processing and image shift processing, high-resolution image data can be generated from a plurality of low-resolution image data, thereby generating fine image data that exceeds the resolution of the imaging device. Is possible.
"Super-resolution processing of images", Shigeki Sugimoto, Masatoshi Okutomi, Journal of the Robotics Society of Japan Vol.23, No.3, pp.305-309,2005 JP 2006-127241 A (published May 11, 2006) "Analysis of multi-aperture color images", Naoki Asada, Takashi Matsuyama, Takatoshi Mochizuki, IPSJ Journal Vol.32, No.10, pp.1338-1348, Oct.1991 Japanese Patent Laid-Open No. 02-174470 (published July 5, 1990) JP 2007-49228 A (published February 22, 2007)

  By the way, the image sensor has a narrower range of light (dynamic range) that can be captured at one time than the human eye. For this reason, when an imaging target with a large contrast between light and dark is imaged, in the captured image data obtained by imaging, there may be white overtones in which the gradation disappears in a bright part and blackouts that become black uniformly in a dark part. . A portion where overexposure or underexposure occurs is not data representing the color or shape of the imaging target.

  Therefore, if the multiple low-resolution image data (captured image data) used for the super-resolution processing or the image shift processing includes those with overexposure or underexposure, The color is not accurately reflected in the high-resolution image data, so that the accuracy of the high-resolution processing is reduced. Therefore, when imaging an imaging target having a large difference in brightness, it is necessary to take some measures to prevent a reduction in the accuracy of the high resolution processing.

  Here, conventionally, there is known a technique for expanding a substantial dynamic range of an image sensor, that is, a technique for expanding a dynamic range by performing image processing on a plurality of captured image data obtained by imaging under different imaging conditions. It has been. For example, Non-Patent Document 2, Patent Document 2, and Patent Document 3 describe expanding the dynamic range by performing a combining process using a plurality of images having different exposure times.

  According to the method described in the above-mentioned document, an imaging target that is a target for expanding the dynamic range is imaged with two types of exposure time, that is, short exposure and long exposure. Then, for each image captured with two types of exposure time, improper exposure areas such as overexposure pixels and blackout pixels are detected, and these areas are replaced with short exposure images or long exposure images. Yes.

  However, in the conventional method, the pixels to be replaced need to have the same imaging position. That is, in the above conventional method, it is necessary to use a set of images in which the imaging positions coincide at the pixel level. This is because, when the above-described conventional method is used with image data whose imaging position is shifted, the place where the pixel is replaced becomes discontinuous with the place where the replacement is not performed.

  For this reason, it has not been considered to apply the conventional dynamic range expansion method in super-resolution processing in which imaging is performed while shifting the imaging position. That is, the above-described conventional dynamic range expansion method cannot be employed as a measure for preventing a reduction in accuracy of the high resolution processing when imaging an imaging target having a large contrast.

  The present invention has been made in view of the above problems, and a first object of the present invention is to realize an image generation apparatus and the like that can generate a high-resolution image with an expanded dynamic range.

  In order to expand the dynamic range, it is necessary to perform imaging under a plurality of imaging conditions. When imaging is performed under a plurality of imaging conditions, the number of captured image data used is the number of types of imaging conditions. There is a problem that the imaging time is doubled due to the increase.

  Here, in order to reduce the imaging time, it is conceivable to reduce the number of captured image data used. However, if the number of used images is simply reduced, the information of the captured image data is lost by that amount, resulting in high resolution. The quality of the high-resolution image data obtained by the conversion processing may be impaired.

  The present invention has been made in view of the above problems, and a second object of the present invention is to generate high-resolution image data with an expanded dynamic range without increasing the number of captured image data used. It is to realize an image generation apparatus and the like that can be used.

  In order to solve the above problems, an image generation apparatus according to the present invention obtains a resolution from a plurality of low-resolution images obtained by imaging the same imaging target under one of a plurality of types of imaging conditions with different imaging positions. An image generation apparatus that generates one raised high-resolution image, the imaging position determination unit that sets the imaging position of the plurality of low-resolution images, and the image that is captured at the imaging position set by the imaging position determination unit A high-resolution image generation unit configured to acquire a plurality of low-resolution images and generate the high-resolution image, and the imaging position determination unit includes a reference pixel region having a shape corresponding to one pixel of the low-resolution image as a matrix When the reference imaging positions arranged in a shape are virtually set, a shape corresponding to one pixel of the high-resolution image is located at the center of the shape corresponding to one pixel of the low-resolution image. For the low-resolution image that is set under the imaging positions of the plurality of low-resolution images so that the center region to be included in the reference pixel region and that is captured under at least one imaging condition among the plurality of types, First imaging position setting means is provided for setting a first imaging position, which is an imaging position where the central area is arranged at a uniform or substantially uniform position with respect to the entire reference pixel area arranged in a matrix. It is characterized by being.

  Further, in order to solve the above-described problem, the image generation method according to the present invention provides a resolution from a plurality of low-resolution images obtained by imaging the same imaging target under any one of a plurality of types of imaging conditions with different imaging positions. An image generation method of an image generation apparatus that generates a single high-resolution image with a raised image, an imaging position setting step for setting imaging positions of the plurality of low-resolution images, and an imaging set in the imaging position setting step A high-resolution image generation step of acquiring the plurality of low-resolution images captured at positions and generating the high-resolution image, wherein the imaging position setting step has a shape corresponding to one pixel of the low-resolution image. When the reference imaging position in which the reference pixel areas having the matrix are arranged in a matrix is virtually set, the center is located at the center of the shape corresponding to one pixel of the low-resolution image, The imaging positions of the plurality of low resolution images are set so that a central area having a shape corresponding to one pixel of the resolution image is included in the reference pixel area, and at least one of the plurality of types is captured. For a low-resolution image captured under conditions, the first imaging position, which is an imaging position in which the central area is arranged at a uniform or almost uniform position, is set with respect to the entire reference pixel area arranged in a matrix. The first imaging position setting step is included.

  According to the above configuration, one high-resolution image with an increased resolution is generated from a low-resolution image captured under any of a plurality of types of imaging conditions. That is, a low resolution image based on a plurality of types of imaging conditions is reflected in the high resolution image generated by the above configuration. Thereby, compared with the case where the low resolution image imaged only with one type of imaging condition is used, the dynamic range of a high resolution image can be expanded.

  Note that when a high resolution image is generated from a low resolution image captured under any of a plurality of types of imaging conditions, normalization processing may be performed to compensate for the difference in imaging conditions. That is, normalization processing for correcting the luminance value of each low-resolution image so that it can be considered that all the low-resolution images have been imaged under the selected imaging condition by selecting any one of a plurality of types of imaging conditions. By performing the above, it is possible to generate one high-resolution image with an expanded dynamic range from low-resolution images captured under different imaging conditions.

  Further, according to the above configuration, for the low resolution image captured under at least one imaging condition, the central region is arranged at a uniform or substantially uniform position with respect to the entire reference pixel region arranged in a matrix. A first imaging position that is an imaging position is set.

  Here, the present invention is premised on a technique for generating one high-resolution image with an increased resolution from a plurality of low-resolution images obtained by imaging the same imaging target with different imaging positions.

  In this technique, when a reference imaging position (T1 in FIG. 18) where a reference pixel area having a shape corresponding to one pixel of a low resolution image is virtually set, it corresponds to one pixel of the low resolution image. The imaging position is set so that a central area (P13 to P93 in FIG. 18) corresponding to one pixel of the high-resolution image located at the center of the area to be performed (P11 to P91 in FIG. 18) is included in the reference pixel area. Is set.

  By setting the imaging position in this way, the reference pixel region is imaged a plurality of times from different positions by one pixel of the imaging element that generates luminance value data of one pixel of the low resolution image. By reflecting this luminance value data in the luminance value data of the central area corresponding to one pixel of the high resolution image, the number of luminance value data in the reference pixel area can be increased, that is, the resolution can be improved.

  In the example of FIG. 18, only one region corresponding to one pixel of the low resolution image and one reference pixel region are illustrated. However, in an actual imaging device, the imaging elements are arranged in a matrix, An area corresponding to one pixel of the low resolution image and a reference pixel area are also arranged in a matrix.

  Here, when capturing a plurality of low-resolution images, if the imaging conditions are changed for each imaging position, the central area corresponding to the low-resolution images captured under the same imaging conditions is a reference pixel area arranged in a matrix May be crowded within.

  For example, in the example of FIG. 18, when (a), (c), (e), (g), and (i) are imaged under the first imaging condition and the rest are imaged under the second imaging condition. The imaging conditions of P13, P33, P53, P73, and P93 are all the first imaging conditions, and the imaging conditions of P23, P43, P63, and P83 are all the second imaging conditions. Since the reference pixel regions including the central region of such an arrangement are arranged in a matrix, the central region of the first imaging condition is also arranged in the three directions on the left, upper left, and upper side of P13. Similarly, for P33, P73, and P93, the imaging condition of the center region adjacent in the three directions is the first imaging condition. That is, in this case, a dense region in which four central regions of the first imaging condition are gathered is formed in the reference pixel regions arranged in a matrix.

  When such a dense area is formed, for example, when halation occurs in the low-resolution image captured under the first imaging condition, there is a possibility that the luminance value data of the dense area is lost collectively. When luminance value data is missing in a wide area, even if the missing part is interpolated, the luminance value data used for interpolation will be located away from the missing part, so the resolution of the generated high-resolution image will be increased. The accuracy is significantly reduced.

  In order to solve such a problem, for example, a plurality of sets of low resolution image sets are generated by changing the imaging conditions at the same imaging position and imaging is performed a plurality of times, and each low resolution image set is synthesized. It is conceivable to use a captured image set. Note that each set of captured image sets is captured under the same imaging condition, and one high-resolution image can be generated by one captured image set.

  As described above, when a plurality of low resolution image sets are used, even if the low resolution image set includes a low resolution image in which halation has occurred, the brightness value data of the low resolution image is Therefore, the resolution can be replaced with the low-resolution image data included in the low-resolution image set.

  However, when using a plurality of sets of low-resolution images, there is a problem that the time required for imaging is doubled by the number of sets to be used.

  Therefore, according to the above configuration, for a low resolution image captured under at least one imaging condition, the central region is arranged at a uniform or substantially uniform position with respect to the entire reference pixel region arranged in a matrix. The first imaging position that is the imaging position to be set is set.

  Thereby, even if, for example, halation occurs in any of a plurality of imaging conditions, the luminance value data lost due to the luminance value data of the low-resolution image captured at the first imaging position where the central region is uniformly arranged Can be interpolated. That is, when the luminance value data is interpolated from the uniformly arranged central region, the luminance value data at a position close to the missing portion is used for interpolation of the missing portion. The decrease can be suppressed.

  That is, according to the above configuration, it is possible to generate a high-resolution image with an expanded dynamic range without increasing the number of captured images and without reducing the resolution accuracy.

  In addition, the first imaging position setting unit calculates a distance between the center regions included in one of the reference pixel regions arranged in a matrix, and includes a center region included in the reference pixel region; It is preferable to calculate the distance from each center area included in the reference pixel area adjacent to the reference pixel area, and set the first imaging position so that the minimum value of the calculated distance is maximized.

  According to the above configuration, it is possible to obtain an imaging position where the central region is arranged at a uniform or substantially uniform position with respect to the entire reference pixel region arranged in a matrix.

  In addition, the image generation apparatus corresponds to each of the center regions when the image pickup positions other than the first image pickup position are arranged in a matrix in the reference pixel region with a number of center regions equal to the total number of image pickup times. It is preferable to include second imaging position setting means for selecting and setting from the second imaging position.

  As described above, each pixel of the high resolution image is obtained based on each pixel of the low resolution image. Since the pixels of the high resolution image are arranged in a matrix, the central region is preferably arranged in a matrix.

  Therefore, according to the above configuration, first, the second imaging positions corresponding to the respective central regions when the number of the central regions equal to the total number of times of imaging are arranged uniformly or substantially uniformly are set. In the example of FIG. 18, all the low resolution images are captured at the second imaging position.

  Here, when all the low-resolution images are imaged at the second imaging position, there is a problem in that the central areas corresponding to the same imaging condition are dense as described above. Therefore, according to the above configuration, an imaging position other than the first imaging position is selected and set from the second imaging position.

  Thereby, a part of the central area can be arranged in a matrix, and another part of the central area can be distributed over the entire reference pixel area arranged in a matrix. it can. Therefore, it is possible to generate a high-resolution image with an expanded dynamic range without increasing the number of images to be taken and without reducing the resolution enhancement accuracy, and it is possible to improve the accuracy of the high-resolution image.

  Further, the image generation device selects an imaging position that is the shortest distance from the first imaging position from the imaging positions that are not selected by the second imaging position setting unit among the second imaging positions, and selects the selected imaging position. It is preferable to include weighted imaging position setting means for correcting the first imaging position by weighted averaging of the imaging position and the first imaging position.

  According to the above configuration, the first imaging position is corrected to a position corresponding to the weight between the first imaging position and the second imaging position. Here, as described above, the first imaging position is an imaging position for disposing the center area with respect to the entire reference pixel area arranged in a matrix, and the second imaging position is the center. This is an imaging position for arranging the regions in a matrix.

  And according to said structure, a 1st imaging position can be correct | amended near a 2nd imaging position by setting a weight. Thereby, if necessary, it is possible to set an imaging position that places importance on dispersion of the central area over the entire reference pixel area arranged in a matrix, and the central area is arranged in a matrix. It is also possible to set an imaging position that emphasizes this. Since both the first and second imaging positions have merits and demerits, it is more preferable that the weights can be changed as necessary.

  The imaging condition is preferably at least one of exposure time, aperture, optical filter, and imaging sensitivity.

  By using a plurality of low-resolution images with different imaging conditions as described above, the dynamic range of the high-resolution image can be expanded.

  Moreover, it is preferable that the imaging condition is set according to at least one of brightness, reflectance, and intensity of illumination irradiated on the imaging target.

  As described above, the imaging is performed under a plurality of imaging conditions in order to expand the dynamic range of the high resolution image. Therefore, it is preferable that the imaging conditions are set so that the dynamic range can be efficiently expanded. However, if the intensity of illumination irradiated to the imaging target or the imaging target changes, a suitable imaging condition is set accordingly. Will change.

  Therefore, according to the above configuration, an imaging condition set in accordance with at least one of the brightness and reflectance of the imaging target and the intensity of illumination applied to the imaging target is used. Thereby, a dynamic range can be expanded efficiently.

  For example, the higher the lightness, reflectance, and illumination intensity of the imaged object, the shorter the exposure time, thereby preventing halation in low-resolution images and efficiently expanding the dynamic range. can do.

  In addition, since the luminance value of each pixel of the low-resolution image captured at the first imaging position is used for interpolation of luminance value data lost due to halation or the like, the low-resolution image captured at the first imaging position includes It is preferable to assign an imaging condition in which halation or the like is unlikely to occur compared to other imaging conditions.

  Further, in order to solve the above-described problem, an image generation apparatus different from the above of the present invention has a plurality of low-capacity images obtained by imaging the same imaging target under one of a plurality of types of imaging conditions with different imaging positions. An image generation apparatus that generates one high-resolution image with an increased resolution from a resolution image, wherein a reference imaging position in which reference pixel regions having a shape corresponding to one pixel of the low-resolution image are arranged in a matrix is obtained. When set virtually, the plurality of low-resolution images are located at the center of the shape corresponding to one pixel of the low-resolution image, and the central region having the shape corresponding to one pixel of the high-resolution image, A low-resolution image that is captured so as to be included in the reference pixel region and that is captured under one of the plurality of types of image capturing conditions is the entire reference pixel region arranged in a matrix. In contrast, it is characterized in that which is captured as the central region is arranged in a uniform or substantially uniform position.

  According to the above configuration, the low-resolution image based on a plurality of types of imaging conditions is reflected in the high-resolution image, so that the high-resolution image is compared with the case where the low-resolution image captured using only one type of imaging condition is used. The dynamic range can be expanded.

  Further, according to the above configuration, even when halation or the like occurs in a low-resolution image captured under any imaging condition, the low-resolution captured at the first imaging position where the central region is uniformly arranged Since the missing luminance value data can be interpolated by the luminance value data of the image, it is possible to suppress a decrease in the resolution accuracy of the high resolution image.

  That is, according to the above configuration, it is possible to generate a high-resolution image with an expanded dynamic range without increasing the number of low-resolution images to be used and without reducing the resolution accuracy.

  The image generation apparatus may be realized by a computer. In this case, a control program for causing the image generation apparatus to be realized by a computer by causing the computer to operate as each unit of the image generation apparatus, and A computer-readable recording medium on which it is recorded also falls within the scope of the present invention.

  As described above, the image generation apparatus according to the present invention includes the imaging position determination unit that sets the imaging positions of a plurality of low resolution images, and the plurality of low resolutions that are captured at the imaging positions set by the imaging position determination unit. A high-resolution image generation unit configured to acquire an image and generate the high-resolution image. The imaging position determination unit includes reference pixel regions having a shape corresponding to one pixel of the low-resolution image arranged in a matrix. When the reference image pickup position is virtually set, a center region having a shape corresponding to one pixel of the high resolution image located at the center of the shape corresponding to one pixel of the low resolution image is the reference pixel. The imaging positions of the plurality of low-resolution images are set so as to be included in the region, and the matrix for the low-resolution images captured under at least one imaging condition among the plurality of types is set. 1st imaging position setting means which sets the 1st imaging position which is the imaging position where the above-mentioned center field is arranged in the uniform or almost uniform position to the whole above-mentioned reference pixel area arranged in a shape It is a configuration.

  As described above, the image generation method according to the present invention includes the imaging position setting step for setting the imaging positions of a plurality of low-resolution images, and the plurality of images captured at the imaging positions set in the imaging position setting step. A high-resolution image generation step of acquiring a low-resolution image and generating the high-resolution image, wherein the imaging position setting step includes a matrix of reference pixel regions having a shape corresponding to one pixel of the low-resolution image When the arranged reference imaging positions are set virtually, a central region having a shape corresponding to one pixel of the high-resolution image located at the center of the shape corresponding to one pixel of the low-resolution image is A low-resolution image that is set under the imaging positions of the plurality of low-resolution images so as to be included in the reference pixel region and that is captured under at least one imaging condition among the plurality of types. With respect to the first imaging position setting step, a first imaging position setting step for setting a first imaging position that is an imaging position where the central area is arranged at a uniform or substantially uniform position with respect to the entire reference pixel area arranged in a matrix. It is the structure containing.

  In addition, as described above, the image generation apparatus different from the above according to the present invention has a plurality of low resolutions in which the same imaging target is imaged under one of a plurality of types of imaging conditions with different imaging positions. An image generation apparatus that generates one high-resolution image with an increased resolution from an image, wherein a reference imaging position in which reference pixel areas having a shape corresponding to one pixel of a low-resolution image are arranged in a matrix is virtually The plurality of low-resolution images are located in the center of the shape corresponding to one pixel of the low-resolution image, and the central region having the shape corresponding to one pixel of the high-resolution image is the reference region A low-resolution image that is imaged so as to be included in the pixel region and is imaged under one of the plurality of types of imaging conditions is based on the entire reference pixel region arranged in a matrix. Those captured as serial central region is disposed in a uniform or substantially uniform position.

  Therefore, there is an effect that it is possible to generate a high-resolution image with an expanded dynamic range without increasing the number of captured images and without reducing the resolution enhancement accuracy.

  An embodiment of the present invention will be described below with reference to FIGS.

[Configuration of image processing system]
The configuration of the image processing system (image generation apparatus) 1 according to the present embodiment will be described with reference to FIG. FIG. 2 is a block diagram showing a main configuration of the image processing system 1. As illustrated, the image processing system 1 includes an imaging position determination device (imaging position determination means) 10, an imaging device 20, and a control device 30.

  The imaging position determination device 10 determines an imaging position when the imaging device 20 captures an imaging target. The imaging device 20 is a device that captures an image of an imaging target in accordance with an instruction from the control device 30. The control device 30 controls the operation of the imaging device 20, and also acquires image data (the imaging target acquired by the imaging device 20) ( Hereinafter, super-resolution processing is performed on the captured image data) to generate high-resolution image data.

  In the present invention, the imaging position determination apparatus 10 determines the imaging position when the imaging apparatus 20 performs imaging of the imaging target, thereby increasing high-resolution image data with an expanded dynamic range without increasing the number of imaging. The main feature is in the point of generation. Hereinafter, details of the imaging position determination device 10, the imaging device 20, and the control device 30 which are the components of the image processing system 1 will be described.

  First, the imaging position determination device 10 will be described. The imaging position determination apparatus 10 includes a storage unit 11 and an imaging position calculation unit 12 as illustrated.

  The storage unit 11 stores imaging information in which imaging conditions of the imaging device 20 are associated with the number of captured image data captured under the imaging conditions. That is, in the image processing system 1, imaging is performed a predetermined number of times under a plurality of imaging conditions, and high-resolution image data is generated using the captured image data obtained by the imaging. In addition to the imaging information, the storage unit 11 stores combination-specific imaging positions that indicate imaging positions under each imaging condition. Further, a combination-specific weight for weighting the combination-specific imaging position is stored. Details of these pieces of information will be described later.

  Below, the example which uses exposure time (shutter speed) as an imaging condition is demonstrated. However, the imaging conditions are not limited to this example, and data indicating imaging sensitivity, an optical filter to be used, an iris (aperture), and data combining these can also be used.

  In addition, it is preferable that these imaging conditions are set according to at least one of the brightness of the imaging target, the reflectance, and the illumination intensity at the time of imaging. That is, when the brightness, reflectance, and illumination intensity at the time of imaging are high, halation is likely to occur in the captured image data. In such a case, it is preferable to set the exposure time short. . Similarly, when the brightness, reflectance, and illumination intensity during imaging are low, it is preferable to set the exposure time longer.

  The imaging position calculation unit 12 determines an imaging position at which the imaging device 20 captures an imaging target. The imaging position calculation unit 12 refers to the imaging information stored in the storage unit 11 and determines the imaging position of the captured image data for each imaging condition. Then, the imaging position calculation unit 12 associates the imaging conditions with the imaging positions in the imaging conditions and stores them in the imaging condition storage unit 32. Details of the processing performed by the imaging position calculation unit 12 will be described later.

  Next, the imaging device 20 will be described. The imaging device 20 includes a lens 21, an imaging element 22, and an actuator 23 as illustrated. That is, in the imaging device 20, the light reflected from the imaging target is optically imaged on the imaging element 22 in the imaging device 20 via the lens 21. The imaging element 22 spatially discretizes and samples the image of the imaging target that has been optically imaged, and converts it into an image signal, whereby the imaging target image is acquired as image data. ing.

  As the imaging element 22, for example, a charge-coupled device (CCD), a complementary mental-oxide semiconductor (CMOS), or the like can be used. Here, it is assumed that the imaging device 20 is an area sensor that performs imaging with a plurality of imaging elements 22 arranged in a matrix.

  The actuator 23 changes the relative position between the imaging target and the image sensor 22. The actuator 23 is fixed to the inner wall of the housing of the imaging device 20, and the imaging element 22 is supported by the actuator 23. That is, the actuator 23 moves the imaging element 22 in a two-dimensional manner in a plane perpendicular to the optical axis connecting the imaging target and the lens 21 (a plane parallel to the imaging target surface of the imaging target). And the relative position of the image sensor 22 are changed.

  That is, when the actuator 23 changes the relative position between the imaging target and the imaging element 22, the imaging position at which the imaging device 20 captures the imaging target (position on the imaging target in the imaging region) changes. Yes. For this reason, the imaging position can be expressed as coordinates on a two-dimensional plane parallel to the imaging target surface to be imaged. Here, it is assumed that the actuator controller 36 moves the image sensor to a position specified by the designated two-dimensional coordinates by designating the two-dimensional coordinates. The actuator 23 may move the image sensor 22 not only in two dimensions but also in three dimensions including rotation.

  As the actuator 23, for example, a piezo actuator or a stepping motor can be used. The actuator 23 is not particularly limited as long as it can change the relative position between the imaging target and the image sensor 22, but here, a piezoelectric actuator is used.

  Next, the control device 30 will be described. According to the super-resolution processing performed by the control device 30 according to the present embodiment, high-resolution image data with high accuracy can be generated even when overexposed or underexposure occurs in the captured image data. . Further, according to the super-resolution processing executed by the control device 30, high-resolution image data having a wider dynamic range than that of the conventional super-resolution processing is generated.

  As illustrated, the control device 30 includes an imaging control unit 31, an imaging condition storage unit 32, a low-resolution image storage unit 33, a high-resolution image storage unit 34, and a dynamic range expansion super-resolution processing unit (hereinafter referred to as DR expansion super-resolution). 35) (referred to as a resolution processing unit), and the functions of the control device 30 as described above are realized by these configurations.

  The imaging control unit 31 includes an actuator control unit 36 and an imaging timing control unit 37 as illustrated. The actuator control unit 36 controls the imaging position of the imaging device 20 by moving the actuator 23. Specifically, the actuator control unit 36 controls the timing for starting and stopping the movement of the actuator 23, the moving speed, the moving distance, and the like. Under such control of the actuator control unit 36, the imaging device 20 can perform imaging at the imaging position as described with reference to FIG.

  The imaging timing control unit 37 controls the timing of starting and stopping imaging of the imaging device 20. Specifically, the imaging timing control unit 37 includes information indicating the current imaging position of the imaging device 20 sent from the actuator control unit 36, imaging conditions stored in the imaging condition storage unit 32, and By sending a control signal to the imaging device 20 based on the imaging position corresponding to the imaging condition, the imaging time at each imaging position, that is, the exposure time is controlled.

  In the image processing system 1, when imaging is performed at a certain imaging position according to an instruction from the imaging timing control unit 37, the imaging device 20 can identify the captured image data from the captured image data obtained by imaging. An identifier (for example, an image number) is attached and stored in the low-resolution image storage unit 33.

  By repeating this, at each imaging position determined by the imaging position determination device 10, an imaging target is imaged under an imaging condition corresponding to the imaging position. The captured image data obtained by the imaging is attached with an identifier and stored in the low-resolution image storage unit 33. Since the imaging condition storage unit 32 stores the imaging condition and the imaging position in association with each other, the imaging position and the imaging condition of each captured image data can be specified by using the identifier.

  In the following description, the number of captured image data used to generate one piece of high-resolution image data is m (m is a positive number), and a set of m pieces of images corresponding to one piece of high-resolution image data. The image data will be described as a captured image set.

Although details will be described later, in the image processing system 1, the dynamic range of the high-resolution image data generated by the super-resolution processing can be expanded by using a plurality of captured image sets having different exposure times. Here, the number of captured image sets is n (n is a positive number). In this case, m × n pieces of captured image data y _mn are stored in the low resolution image storage unit 33.

Although details will be described later, in the image processing system 1, high-resolution image data generated by super-resolution processing by using one set of captured image data composed of a plurality of captured image data having different exposure times. The dynamic range of can also be expanded. In this case, the low resolution image storage unit 33, so that m pieces of captured image data y _m is stored.

The high resolution image storage unit 34 stores the high resolution image data h generated by performing the super-resolution processing using the captured image data y _m or y _mn stored in the low resolution image storage unit 33. As in the conventional super-resolution process, since it is necessary to use initial high-resolution image data in order to perform the super-resolution process, the high-resolution image storage unit 34 is preliminarily used before the super-resolution process. The initial high resolution image data is stored in.

  The initial high-resolution image data is not particularly limited as long as it is an image having a resolution desired to be generated by the super-resolution processing. For example, when the resolution of the captured image data is tripled, a black image, a white image, or a gray image having a uniform entire surface where the size of each pixel is 1/3 of the captured image data is used as the initial high resolution image data. Can be used.

  The DR enlargement super-resolution processing unit 35 performs super-resolution processing for generating high-resolution image data by increasing the resolution of the captured image data stored in the low-resolution image storage unit 33. Specifically, the DR expansion super-resolution processing unit 35 is based on the captured image data stored in the low-resolution image storage unit 33 and the high-resolution image data 42 stored in the high-resolution image storage unit 34. New high-resolution image data is generated, and the generated high-resolution image data is stored in the high-resolution image storage unit 34. Details of the super-resolution processing executed by the DR expansion super-resolution processing unit 35 will be described later.

[Example of using multiple captured image sets]
In the image processing system 1, the imaging position determination device 10 appropriately sets the imaging position of the captured image data, thereby increasing the dynamic range without increasing the number of captured image data used for the super-resolution processing. Resolution image data can be generated.

  Here, first, a high-resolution image whose dynamic range is expanded by increasing the number of captured image data used for super-resolution processing (using a plurality of captured image sets), which is a premise of the above configuration. An example of generating data will be described.

As described above, high-resolution image data having N times the resolution of the captured image data can be generated by using N ² captured image data captured at different imaging positions. Note that the N ² sheets of captured image data are all captured under the same imaging conditions. That is, each captured image data included in one set of captured image sets is captured under the same imaging conditions, and each captured image data included in different captured image sets is captured under different imaging conditions. It will be what was done.

The DR expansion super-resolution processing unit 35 generates one set of captured image sets by generating one set of high-resolution image data using a plurality of sets of captured image sets obtained by imaging under different imaging conditions. Compared to the case of using it, the dynamic range of the high-resolution image data can be expanded. In this case, high-resolution image data is generated using m × n pieces of captured image data y _mn stored in the low-resolution image storage unit 33.

[Detailed configuration of DR expansion super-resolution processing unit]
Next, a detailed configuration of the DR expansion super-resolution processing unit 35 will be described with reference to FIG. FIG. 3 is a block diagram illustrating a main configuration of the DR expansion super-resolution processing unit 35. As shown in the figure, the DR expansion super-resolution processing unit 35 includes a normalization processing unit 43, an image synthesis unit 44, a difference image generation unit 45, a pseudo low-resolution image generation unit 46, a region extraction unit 47, and an image update unit 48. I have.

  When the captured image set includes images captured under different imaging conditions, the normalization processing unit 43 converts all captured image data included in the captured image set to the different imaging conditions. The luminance value of each pixel of each captured image data is corrected and normalized so that it can be processed as being captured under one imaging condition. By the normalization processing performed by the normalization processing unit 43, it is possible to perform super-resolution processing using a captured image set including captured image data obtained by imaging under different imaging conditions.

Specifically, the normalization processing unit 43 _converts the m × n pieces of captured image data y _mn stored in the low resolution image storage unit 33 into the m × n sheets stored in the imaging condition storage unit 32. Normalization processing is performed on the basis of the imaging conditions used for imaging the captured image data y _mn .

  For the normalization process performed by the normalization processing unit 43, a technique of correcting the luminance value of each pixel of the image data captured under the other image capturing conditions can be employed with one image capturing condition as a reference.

  For example, consider the case where there are captured image data A, B, and C, and the exposure times of the captured image data are a, b, and c (a <b <c), respectively. In this case, with the exposure time a as a reference, the brightness values of the captured image data B and C can be expressed by, for example, the following mathematical formula (5).

  That is, the captured image data B with the exposure time b is b / a times as long as the exposure time is b / a times that of the captured image data A with the exposure time a. It is expected to be. Similarly, the captured image data C with the exposure time c is c / a times as long as the exposure time is c / a times that of the captured image data A with the exposure time a. It is expected that

  Therefore, when the captured image data A with an exposure time of a is used as a reference, the brightness value of the captured image data with an exposure time of b is corrected to a / b times, and the brightness of the captured image data with an exposure time of c seconds is corrected. By correcting the value by a / c times, all of the luminance values of the captured image data A to C can be made equal. The reference exposure time can be selected from any of the captured image data A to C.

  Note that the normalization method is not limited to the above example. For example, normalization may be performed by obtaining an average value of the luminance values of the captured image data and performing correction so that the luminance values of the respective captured image data are approximately the same based on the obtained average value. Also, in addition to the exposure time, when the imaging conditions such as the imaging device to be used, the illumination intensity, the imaging sensitivity, the filter to be used, and the iris are different, all the captured image data used for the super-resolution processing is The luminance value of each pixel of each captured image data is corrected (normalized) so that it can be processed as being captured under one of the different imaging conditions.

  The image composition unit 44 synthesizes a plurality of captured image sets with different imaging conditions normalized by the normalization processing unit 43 to generate composite captured image data. Specifically, the image composition unit 44 synthesizes the m captured image data included in each captured image set with the corresponding image capturing positions to generate composite captured image data. Thereby, one set of captured image sets composed of m pieces of composite captured image data can be obtained.

  The composite captured image data reflects the captured image data obtained by imaging under a plurality of different imaging conditions, so that a high resolution image generated by performing super-resolution processing using the composite captured image data The dynamic range of data can be expanded. A method for generating composite captured image data will be described later.

  The difference image generation unit 45 calculates the difference between the captured image data and the pseudo low-resolution image data as difference image data, and sends the calculated difference image data to the image update unit 48. When there are a plurality of captured image sets (n ≧ 2), the difference image generation unit 45 calculates the difference between the combined captured image data generated by the image synthesis unit 44 and the pseudo low-resolution image data as difference image data. To do. The pseudo low resolution image data generated by the pseudo low resolution image generation unit 46 is used.

  The pseudo low resolution image generation unit 46 reads the high resolution image data stored in the high resolution image storage unit 34 and reduces the resolution so that the resolution of the read high resolution image data is the same as that of the captured image data. . Further, the pseudo low resolution image generation unit 46 performs conversion by PSF on the high resolution image data, and generates a plurality (m pieces) of pseudo low resolution image data aligned with each of the captured image data. The pseudo low-resolution image data is sent to the difference image generation unit 45 and used for generation of the difference image data.

  The region extraction unit 47 detects an inappropriate exposure region that is inappropriate for the super-resolution processing in the captured image data stored in the low-resolution image storage unit 33. The improper exposure region is a region having a luminance value outside the effective dynamic range of the imaging device 20, and is a region where blackout or whiteout is highly likely to occur. When captured image data including such a region is used for the super-resolution processing, there arises a problem that the accuracy of the super-resolution processing is reduced and the processing time required for the repetitive calculation is increased.

  Specifically, the area extraction unit 47 extracts, for each of m × n pieces of captured image data, a first area whose luminance value is outside a predetermined normal range, and each captured image data of the extracted area Remember the top position. Here, in the captured image data, an area other than the extracted first area is referred to as a second area. That is, when the region extraction unit 47 extracts the first region, each captured image data is divided into a first region having a luminance value outside the normal range and a second region having a luminance value within the normal range. .

  Note that the normal range is a range of luminance values defined by an upper limit value and a lower limit value of luminance values that can be regarded as appropriate exposure. An area that shows a luminance value that exceeds the upper limit is an area where halation such as overexposure is considered to occur, and an area that shows a luminance value that is lower than the lower limit is considered to be in a blackened state. It is an area that is Therefore, the second region having the luminance value within the normal range is a region having the luminance value obtained by imaging within the effective dynamic range of the imaging device 20.

  The region extraction may be performed in units of pixels, or may be performed in units of blocks in which a plurality of pixels are one block. For example, the captured image data is divided into a plurality of blocks by setting an area of x × y pixels (x and y are positive numbers) as one block, and each block is first based on an average value of luminance values of pixels included in each block. You may make it judge whether it is classified into an area | region.

  The storage method in which the region extraction unit 47 stores the extracted region is not limited as long as the first region or the second region is indicated for all the pixels of each captured image data. By storing in this way, it is possible to speed up subsequent processing.

That is, the region extraction unit 47 sets a memory space Z _mn having the same size as that of m × n pieces of captured image data. Then, 1 is set in the pixel position on the memory space _Zmn corresponding to the pixel determined to be the first region in the captured image data, and the memory corresponding to the pixel determined to be the second region. 255 is set in the pixel position on the space _Zmn . As a result, in the memory space Z _mn , a value of 1 is stored at the position of the overexposed or blackened pixel, and a value of 255 is stored at the position of the pixel within the dynamic range.

  Here, since it is assumed that the luminance value is 256 gradations from 1 to 255, the value of 255 is applied to the pixel position determined to be the second area, and the first area is determined. A value of 1 is applied to each pixel position. The purpose of this process is to make the contribution of the pixels determined to be the first area to the high-resolution image data smaller than the pixels determined to be the second area. Thus, the luminance value to be set can be changed as appropriate.

Here, the data generated by the region extraction unit 47 as described above is referred to as extraction region data _Zmn . The extracted area data Z _mn is used for weighting of the difference image data by the image update unit 48 and is used for image synthesis by the image synthesis unit 44.

In other words, the image composition unit 44 refers to the extracted region data _Zmn when integrating the captured image data having the same imaging position and different exposure time into one image data, and is included in the composite captured image data. Integration is performed so that one area is reduced. When a plurality of sets of captured image sets are used, when the image composition unit 44 generates composite captured image data, the region extraction unit 47 uses the extracted region data Z _mn for each pixel of the composite captured image data. updating the extracted region data Z _m indicating whether belonging to the second region belongs to the first region.

Specifically, the image composition unit 44 refers to the extraction region data _Zmn to determine whether each pixel of the captured image data having the same imaging position and different exposure time belongs to the first region or the second region. judge. When a pixel belonging to the first area and a pixel belonging to the second area are detected in the pixel at the same position, the image composition unit 44 uses the brightness value of the pixel at the position in the composite captured image data as the brightness value of the pixel at the position. The luminance values of the pixels belonging to the two areas are adopted. The image synthesizing unit 44 updates the extracted area data Z _m pixel values of the position as the second region.

  When a plurality of pixels belonging to the second region are detected in the pixels at the same position, for example, the luminance value of the pixel having the longest exposure time may be employed. Further, when only pixels belonging to the first region are detected in the pixels at the same position, the luminance value of any one of the pixels is employed. Therefore, for example, with the captured image set having the longest exposure time as a reference, among the pixels included in the captured image set, the pixels belonging to the first region are pixels at the same position in the other captured image sets, and the second What is necessary is just to replace with the luminance value of the pixel which belongs to an area | region.

Thus, m × n post-normalized captured image data is integrated into m composite captured image data, and the extracted region data Z _mn generated based on m × n post-normalized captured image data There is updated to extract region data Z _m corresponding to m pieces of synthetic image data. Similarly to the extraction area data Z _m In extracted area data Z _mn, the position of the pixel in the first region is a value of 1 is stored, the position of the pixel of the second area is stored the value of 255.

  The composite captured image data generated by such composition processing reflects the captured image data obtained by capturing images at a plurality of exposure times, so the dynamic range is expanded compared to the captured image data. . Therefore, by performing the super-resolution processing using the composite captured image data, it is possible to generate high-resolution image data having a wide dynamic range compared to the case of performing the super-resolution processing using the captured image data. .

  Furthermore, in the composite captured image data generated by such composition processing, if possible, the brightness value of the pixel in the first region is the brightness of the pixel in the second region where the position is the same in the other captured image data. Has been replaced by a value. Therefore, by performing super-resolution processing using the composite captured image data, the ratio of data useful for super-resolution processing is higher than when super-resolution processing is performed using captured image data. The accuracy of the super-resolution processing can be improved, and the calculation time of the super-resolution processing can be shortened.

Image update section 48, based on the extracted region data Z _m the region extraction unit 47 is generated, on the difference image data difference image generation unit 45 is calculated, weighting corresponding to the first image region and a second image area The weighted difference image data is generated, and the high-resolution image data is updated using the weighted difference image data. The weighting by the image updating unit 48 reduces the influence of the first region having a luminance value outside the normal range on the super-resolution processing, so that overexposure or blackout occurs in the captured image data. Even in such a case, highly accurate super-resolution processing can be performed. Then, the image update unit 48 stores the updated high resolution image data in the high resolution image storage unit 34.

The weighting is not particularly limited as long as it is performed so that the contribution of the first image area to the super-resolution process is smaller than the contribution of the second image area. Here, in the extraction area data Z _m, the position of the pixels of the first area stores a value of 1, the position of the pixel of the second area so that the value of 255 is stored. Therefore, as a weight value obtained by multiplying the 1/255 to extract region data Z _m, multiplied by the difference image data.

  Thereby, in the difference image data, the luminance value of the pixel in the region which is the first region in the composite image data is 1/255. On the other hand, in the difference image data, the luminance value of the pixel in the region that is the second region in the composite image data is not corrected. Therefore, according to the above configuration, the contribution of the pixels in the first area in the composite image data is sufficiently lower than the contribution of the pixels in the second area in the composite image data. can do.

Here, an example has been described for weighting by multiplying the extracted area data Z _m in the difference image data, she is also without using the extracted region data Z _m and extracted area data Z _mn to perform weighting. For example, the image update unit 48 refers to the normalized captured image data or the luminance value of each pixel position of the captured image data, and the luminance value of each pixel of the difference image data is based on the pixels in the first region. It may be determined whether it is calculated or calculated based on the pixels in the second region, and a weight may be assigned based on the determination result.

  Further, here, an example has been described in which weighting is performed for two regions, the first region and the second region, but it may be divided into three or more regions. For example, when the amount of incident light and the output are not linear in the effective dynamic range of the imaging device 20, it is preferable to divide into three or more regions and assign weights according to each region.

  As a specific example, when the luminance value is 256 gradations from 0 to 255, the weight in the range of the luminance value 0 to 2 is set to 0, the range of the luminance value 3 to 7 and the luminance value is The weight in the range of 248 to 252 may be ¼ of the weight of the second region, and the weight in the range of the luminance value from 253 to 255 may be zero. In this way, high-accuracy and high-precision reflection of captured image data is performed by performing weighting stepwise so that the region whose luminance value is closer to the effective dynamic range has a greater contribution to super-resolution processing. Resolution image data can be generated.

[Flow of dynamic range expansion super-resolution processing in the super-resolution processor]
Next, the super-resolution processing executed in the DR expansion super-resolution processing unit 35 will be described with reference to FIG. FIG. 4 is a flowchart illustrating an example of super-resolution processing executed in the DR expansion super-resolution processing unit 35. In the following description, it is assumed that m × n pieces of captured image data y _mn are captured by the imaging device 20 and stored in the imaging condition storage unit 32. Here, for the sake of simplicity, an example in which only the exposure time is changed as the imaging condition will be described. However, imaging conditions other than the exposure time may be changed. Furthermore, although an example in which super-resolution processing is performed using a plurality of sets of captured image sets will be described here, super-resolution processing can be performed if there is at least one captured image set.

First, the region extraction unit 47 and the normalization processing unit 43, acquires the stored m × n pieces of captured image data y _mn, from the imaging condition storage section 32 of the m × n pieces of captured image data y _mn The imaging conditions used for imaging are read (S201).

Area extracting unit 47 acquires the captured image data y _mn, from each of the m × n pieces of captured image data y _mn, based on the luminance value distribution of the captured image data y _mn, overexposure area or underexposure region like The improper exposure area is identified and extracted as the first area (S202). Then, the region extraction unit 47 generates extraction region data Z _mn indicating whether each pixel of the m × n normalized captured image data belongs to the first region or the second region.

On the other hand, the normalization processing unit 43 acquires the exposure time of each captured image data y _mn from the acquired imaging conditions, and normalizes the captured image data y _mn based on the acquired exposure time (S203). Then, the normalization processing unit 43 sends the captured image data y _mn generated by normalization to the image composition unit 44.

When the image composition unit 44 receives the captured image data y _mn , in each captured image set, the image composition unit 44 _{combines the} captured image data obtained by capturing images at the same image capturing position to obtain one set of _m captured composite image data y _m . generated, and sends the synthesized image data _{y m} of m sheets thus generated in the difference image generation unit 45 (S204). The image synthesizing unit 44 instructs the region extraction unit 47, the extraction area data Z _mn corresponding to m × n pieces of captured image data y _mn, corresponding to m pieces of synthetic image data y _m extraction and updates the region data _{Z m.} When the number of imaging sets is 1, the process of S204 can be omitted.

  Here, the image processing system 1 performs super-resolution processing using the composite captured image data, but may perform image shift processing using the composite captured image data. When image shift processing is performed using composite captured image data, high-resolution image data having a wide dynamic range can be generated compared to when image shift processing is performed using captured image data. Note that the image shift process is performed by mapping the luminance value of each pixel of the composite captured image data to each pixel of the high resolution image data.

After the image synthesizing unit 44 has completed the generation of the synthesized image data y _m, or if the imaging number of sets is 1, the pseudo low-resolution image generating unit 46 is stored in the high-resolution image storage unit 34 Initial high-resolution image data h is acquired (S205).

  Subsequently, the pseudo low resolution image generation unit 46 aligns the acquired initial high resolution image data h with each of the m pieces of captured image data, and considers factors such as optical degradation factors and downsampling. Then, the resolution is reduced, and pseudo low-resolution image data is generated (S206).

Specifically, the pseudo low-resolution image generation unit 46 multiplies the initial high-resolution image data h by B _m expressed by Equation (2) to generate pseudo low-resolution image data B _m h. Although the initial high resolution image data h is one, the pseudo low resolution image data B _m h is aligned with each of the captured image data, so the number of the pseudo low resolution image data B _m h is m. It becomes a sheet. Then, the pseudo low resolution image generation unit 46 sends the generated pseudo low resolution image data B _m h to the difference image generation unit 45.

When the differential image generation unit 45 receives the pseudo low-resolution image data B _m h, the difference image generation unit 45 subtracts the composite captured image data y _m generated by the image synthesis unit 44 from the received pseudo low-resolution image data B _m h to obtain m images. It calculates the differential image data _{X m} (S207). Then, the difference image generation unit 45 sends the differential image data X _m of m Like the calculated the weighting image update section 48. When there is only one set of captured image data, the difference image generation unit 45 subtracts the captured image data y _m from the pseudo low-resolution image data B _m h to calculate m pieces of difference image data X _m . To do.

Image update section 48 receives the differential image data _{X m,} for weighting based on the extracted region data _{Z m} the region extraction unit 47 is generated (S208). Specifically, the image updating unit 48, by multiplying the .alpha.z _m the differential image data X _m, to generate a weighted difference image data X _m .alpha.z _m obtained by correcting the luminance value of each pixel of the difference image data X _m. Note that α is a weight to be multiplied by the extraction region data Z _m , and the value of α can be appropriately set as necessary.

For example, here, in the extraction area data Z _m, the luminance values of the pixels of the first area and 1, since the luminance values of the pixels of the second area is assumed to be 255, the value of alpha 1 / It may be set to 255. As a result, the luminance values of the pixels in the first region in the weighted difference image data X _m αZ _m are all corrected to values of 1 or less, while the luminance values of the pixels in the second region are the same as the difference image data X _m. . Therefore, the contribution of the luminance value of the pixels belonging to the first area can be made lower than that of the pixels in the second area, and thereby the accuracy of the super-resolution processing can be improved.

Further, the image update unit 48 substitutes the weighted difference image data X _m αZ _m generated by weighting into the evaluation function, and calculates the evaluation value E (S209). As the evaluation function, for example, the mathematical formula (3) can be used as in the conventional super-resolution processing. Then, the image update unit 48 compares the calculated evaluation value E with a preset threshold value (S210). Note that, similarly to the conventional super-resolution processing, the threshold value can be appropriately changed according to the required accuracy of resolution enhancement.

  If the evaluation value E is smaller than the threshold value (YES in S210), the image update unit 48 ends the super-resolution process. At this time, the high-resolution image data h stored in the high-resolution image storage unit 34 is the high-resolution image data that is the result of the super-resolution processing.

  On the other hand, when the value of the evaluation value E is equal to or greater than the threshold value (NO in S210), the image update unit 48 uses the high resolution used for calculating the evaluation value E so that the evaluation value E becomes smaller. The image data h is updated (S211). The image update unit 48 can update the high-resolution image data h by using the above formula (4), for example, similarly to the conventional super-resolution processing. The image update unit 48 stores the updated high resolution image data h in the high resolution image storage unit 34.

  Thereafter, the processing from S206 to S210 is repeated again. Then, the DR expansion super-resolution processing unit 35 repeats the processes from S206 to S210 until the evaluation value E becomes smaller than the threshold value. As a result, it is possible to obtain high-resolution image data h having a desired accuracy and having a higher resolution at a desired magnification.

  In addition, said flowchart shows an example of the super-resolution process of this invention, and this invention is not limited to this example. For example, when all the captured image data used for the super-resolution process are captured under the same imaging condition, the normalization process in S203 may be omitted.

Further, as described above, when the pseudo low-resolution image data B _m h is generated from the high-resolution image data h, a PSF convolution process is performed. By the PSF convolution process, the luminance value of the pixel whose luminance value is corrected to a sufficiently small value by weighting is interpolated and updated by the influence of the luminance value of the pixels located around the pixel. Therefore, the pixel values of the pixels in the first region where overexposure and underexposure occur also approach the values reflecting the shape and color of the imaging target by repeating the processes from S206 to S210.

  That is, when the captured image data includes only the first set of m images and the first area is included in the captured image data, the data effective for super-resolution image processing is reduced. The high-resolution image data h is not necessarily not updated at all.

  In addition, each of the captured image data included in the captured image set used for the super-resolution processing may be captured under different imaging conditions. Of course, even when the number of captured image data is only one set m, the imaging conditions of the captured image data may be different.

[Problems and solutions for using multiple captured image sets]
As described above, by using a plurality of captured image sets, the dynamic range of high-resolution image data can be expanded. However, when imaging is performed with a plurality of exposure times, there is a problem that the imaging time is doubled by the number of types of exposure times.

  For example, in the above-described super-resolution processing, it is necessary to capture nine times in order to obtain high-resolution image data with three times the resolution. In this case, if the exposure time is set to 100 milliseconds, imaging is completed in 900 milliseconds. However, when the exposure time is set to three types of 100 milliseconds, 200 milliseconds, and 300 milliseconds, it takes a minimum of 5400 milliseconds to complete the imaging.

  In order to solve this problem, in this embodiment, the imaging position of the captured image data is determined using the imaging position determination device 10. Thereby, it is possible to reduce the number of times of imaging while performing imaging with a plurality of exposure times. Thereby, it is possible to generate high-resolution image data with an expanded dynamic range while suppressing an increase in imaging time. Below, the detail of the imaging position determination apparatus 10 is demonstrated.

[Details of imaging position determination device]
FIG. 1 is a block diagram illustrating a main configuration of the imaging position determination apparatus 10. As illustrated, the storage unit 11 includes an imaging information storage unit 13, a combination-specific imaging position storage unit 14, and a combination-specific weight storage unit 15.

  The imaging information storage unit 13 stores in advance imaging information in which imaging conditions for capturing captured image data and the number of images to be captured under the imaging conditions are associated with each other.

  Here, a specific example of the imaging information stored in the imaging information storage unit 13 will be described with reference to FIG. FIG. 5 is a data table showing an example of imaging information stored in the imaging information storage unit 13. As shown in the figure, the imaging information includes “captured image group number” indicating an identifier of the imaging information specified by the captured image group, “imaging condition / exposure time” indicating the imaging condition, and imaging performed under the imaging condition. It includes items of “number of captured images” indicating the number of images, “image number” indicating an identifier of captured image data captured under the imaging conditions, and “priority” indicating priority of the imaging conditions with respect to other imaging conditions.

  Here, since it is assumed that only the exposure time is changed as the imaging condition, the imaging information includes information indicating “exposure time”. However, when the imaging condition other than the exposure time is changed. The imaging conditions are stored in the imaging information storage unit 13 as imaging information.

  The imaging information in FIG. 5 includes two records of “captured image group G1” and “captured image group G2”. In the “captured image group G1” and “captured image group G2” records, “1” and “2” are set as priorities, respectively.

  The priority is used when determining the imaging position. That is, the imaging position determination device 10 determines the imaging position of the captured image group in descending order of priority. In addition, the imaging position determination device 10 causes pixels derived from a group of captured images with high priority to be evenly arranged on the high resolution image data. In the present embodiment, the priority means that 1 is the highest priority, and the priority of 2 or more is assumed to decrease in order as the number increases.

  Here, the main purpose is to eliminate the influence of overexposure due to halation, so the priority of the captured image group (captured image group with a short exposure time) captured under an imaging condition in which overexposure hardly occurs is increased. Yes. This is because, in high-resolution image data, the pixels derived from the captured image data in which overexposure occurs are appropriately arranged by uniformly arranging the pixels derived from the captured image data in which overexposure does not occur. This is because interpolation is possible. The priority can be appropriately set as necessary.

  The combination-specific imaging position storage unit 14 stores the imaging positions determined by the combination-specific imaging position calculation unit (first imaging position setting unit) 17 and the condition-specific imaging position selection unit (second imaging position setting unit) 18. Is. The combination-specific imaging position storage unit 14 may be a working area that is responsible for storing a certain period until the imaging position is determined, or may be one that permanently stores the imaging position.

  The combination-specific weight storage unit 15 stores weights used to correct the imaging position determined by the weighted imaging position determination unit 19 to a more appropriate position. Details of the weight will be described later.

  As illustrated, the imaging position calculation unit 12 includes a combination-specific imaging position calculation unit 17, a condition-specific imaging position selection unit 18, and a weighted imaging position determination unit (weighted imaging position setting unit) 19. .

  The combination-specific imaging position calculation unit 17 determines the imaging position of each captured image group stored in the imaging information storage unit 13. Then, the condition-specific imaging position selection unit 18 determines an imaging position corresponding to each imaging condition based on the imaging position calculated by the combination-specific imaging position calculation unit 17.

  For example, when imaging is performed at nine imaging positions, when determining four imaging positions at which imaging is performed under the first imaging condition and five imaging positions at which imaging is performed under the second imaging condition, combinational imaging is performed. The position calculation unit 17 and the condition-specific imaging position selection unit 18 determine the imaging position in the following procedure.

  First, the combination-specific imaging position calculation unit 17 selects a captured image group with the highest priority from the captured image group stored in the imaging information storage unit 13, and determines the imaging position of the selected captured image group. . For example, in the example of FIG. 5, since the priority of the captured image group G1 is the highest, the combined-capture imaging position calculation unit 17 determines the imaging positions of the four captured images included in the captured image group G1.

  Although details will be described later, at this time, the imaging position calculation unit 17 for each combination determines the imaging positions so that the imaging positions are dispersed as evenly as possible. The main feature point of the imaging position determination device 10 is in this imaging position determination method.

  Subsequently, the combination-specific imaging position calculation unit 17 selects a captured image group having the next highest priority from among the captured image groups stored in the imaging information storage unit 13, and sets the imaging position of the selected captured image group. Determine candidates. For example, in the example of FIG. 5, the combination-specific imaging position calculation unit 17 determines the imaging positions of the nine captured images included in the captured image group G2 having the second highest priority.

  As described above, when the combination-specific imaging position calculation unit 17 determines the imaging position for all captured image groups stored in the imaging information storage unit 13, the condition-specific imaging position selection unit 18 selects the combination-specific imaging position calculation unit. The five imaging positions excluding the four imaging positions located at the shortest distance from each of the four imaging positions determined by the combination-specific imaging position calculation unit 17 from the nine imaging positions determined by 17 under the second imaging condition. It is determined as an imaging position for imaging. As a result, an imaging position where imaging is performed under the first imaging condition and an imaging position where imaging is performed under the second imaging condition are determined.

  The weighted imaging position determination unit 19 corrects the imaging position determined by the combination-specific imaging position calculation unit 17. For example, in the example of FIG. 5, the weighted imaging position determination unit 19 selects the four imaging positions determined by the combination imaging position calculation unit 17 from the four imaging positions and the nine imaging positions. Correction is made to a position corresponding to a preset weight on a line segment connecting the four imaging positions located at the shortest distance from each of the imaging positions (imaging positions not selected as the five imaging positions). This weight is stored in advance in the weight storage unit 15 for each combination. The details of the method for determining the imaging position based on this weight will be described later.

[Concept of dispersion of imaging positions]
In order to expand the dynamic range, it is necessary to use captured image data captured under different imaging conditions (exposure conditions). However, since the time required for imaging increases as the number of captured image data used increases, it is preferable not to increase the number of captured image data used.

  Here, for example, by performing imaging at an imaging position where high-resolution image data as shown in FIG. 6 is generated, the dynamic range can be expanded without increasing the number of used captured image data.

  FIG. 6 is a diagram illustrating an example of an arrangement of pixels derived from captured image data captured under different imaging conditions in high resolution image data. FIG. 6A is used to generate high resolution image data. FIG. 11B shows a state in which no halation or the like has occurred in the captured image data, and FIG. 5B shows that in the captured image data used to generate the high-resolution image data, halation has occurred in the captured image data having a long exposure time. It shows the state.

  In FIG. 6, super-resolution processing is performed using four captured image data captured at the exposure time a and five captured image data captured at the exposure time b, and is three times the captured image data. An example of generating high-resolution image data having a resolution of (a <b) is shown. As described based on FIG. 18, each pixel of the high-resolution image data is generated based on each pixel of the captured image data. In FIG. 6, the pixels derived from the captured image data captured at the exposure time “a” and the pixels derived from the captured image data captured at the exposure time “b” are shown in different colors.

  Further, as described with reference to FIG. 18, when the resolution is increased to 3 times, the image pickup position is moved in eight directions at 1/3 pitch of one pixel of the picked-up image data, and once at each image pickup position. Images are taken one by one, and this pitch is the width of one pixel of the high resolution image data.

  That is, in order to generate the high-resolution image data as shown in FIG. 6, after the pixel T2 is imaged with the exposure time b, the imaging position is moved to the right in FIG. 6 by 1/3 pitch of one pixel of the imaged image data. Imaging is performed at the exposure time a, and further, the imaging position is moved to the right in the figure by 1/3 pitch of one pixel of the captured image data, and imaging is performed at the exposure time b.

  In this way, each pixel of the high-resolution image data corresponds to each captured image data that is the source of the pixel, and the position of each pixel of the high-resolution image data is each captured image data that is the source of the pixel. It corresponds to the imaging position.

  Here, in the example of FIG. 6, there are locations where four pixels derived from the captured image data captured at the exposure time b are gathered. Even if there is such a part, each captured image data does not include the first area (the area where overexposure, blackout occurs, etc.), and the second area (the shape, color, etc. of the imaging target is reflected normally) In particular, there is no problem when it includes only the (region).

  Here, if any of the captured image data used for generating the high-resolution image data includes the first region, information is lost in the pixels of the high-resolution image data corresponding to the captured image data. .

  For example, since the captured image data obtained by imaging with the exposure time b is b> a, the captured image data obtained by imaging with the exposure time a does not include an overexposed region. Even in such a case, the captured image data obtained by imaging with the exposure time b may include an overexposed region.

  In particular, depending on how the imaging conditions are set, a large amount of pixels in the first region may occur in the captured image data captured under one imaging condition. For example, as in the example of FIG. 6, when one exposure time is long, overexposure may occur in a wide area of captured image data captured with a long exposure time.

  In FIG. 6B, the captured image data captured at the exposure time a is all in the second area, and the captured image data acquired at the exposure time b is all the first area (out of focus). The high resolution image data in the case is shown. In the same figure, the area | region which attached | subjected the code | symbol "j (Jay)" represents the area | region which has become overexposed. This is because all the captured image data corresponding to the area is the first area (out-of-field), and thus the information of the captured image data does not contribute to the generation of the pixels of the high-resolution image data. It is shown that.

  In addition, it is possible to estimate the pixels of the overexposed portion of the high-resolution image data by using the portion of the second region other than the first region (overexposed) of the captured image data. However, in this case, it is based on a portion different from the portion of the captured image data that should originally be a reference for estimation, that is, a pixel classified in the second region at a position distant from the pixel classified in the first region. Estimation will be performed.

  This is because the image information effective for generating high-resolution image data is missing in the overexposed and underexposed areas in the captured image data, and such areas do not contribute to the super-resolution processing at all. is there. Therefore, in such a case, the accuracy of estimation is reduced, and as a result, the quality of high-resolution image data is reduced.

  It can be said that such a decrease in estimation accuracy occurs when the captured image data includes the first region and the pixels of the effective captured image data become coarse and dense. In particular, as shown in FIG. 6B, when one of the exposure times is long and overexposure occurs in a wide area of the captured image data captured with a long exposure time, the pixels of the effective captured image data are coarsely and densely. It is remarkable that occurs.

  Therefore, even if a pixel with captured image data is classified as the first region, it is necessary to determine the imaging position so that the pixel classified as the second region serving as a reference for estimation is not extremely separated instead. There is. In other words, in the high-resolution image data, it is necessary to determine the imaging position when imaging is performed under each imaging condition so that pixels derived from the captured image data captured under the same imaging condition are not concentrated. An imaging position determination method considering the degree is necessary. In addition, a more suitable imaging position can be obtained by correcting the imaging position determined by such a method by weighting processing.

[Method to distribute imaging positions]
Hereinafter, an imaging position determination method executed by the imaging position determination apparatus 10 in consideration of weighting and the degree of dispersion of the imaging positions will be described. First, a method for distributing the imaging positions under each imaging condition will be described with reference to FIGS. In order to distribute the imaging positions, it is important to grasp the positional relationship between the imaging positions.

First, a method for grasping the positional relationship between the imaging positions will be described with reference to FIG. FIG. 7 is an explanatory diagram illustrating a method for grasping the positional relationship between the imaging positions. In FIG. 7, the positional relationship of four imaging positions is shown as an example. In FIG. 7, K ^L indicates one imaging region (reference pixel region) serving as a reference, and V ^L _{0 to} V ^L ₃ indicate four imaging positions (imaging regions) that are targets for grasping the positional relationship. ).

Here, the imaging position for acquiring the captured image data that is the basis of the high-resolution image data is within a range that does not exceed the width of one imaging element, and more specifically, at the center of the imaging area at each imaging position. When an area corresponding to one pixel of the resolution image data is arranged, the area is set within a range included in the reference pixel area. For example, also in the example of FIG. 18, P13 to P93 are all included in T1. Therefore, V L 0 ~V L 3 also falls within the framework of the K L.

In the context of the K L, in order to disperse the imaging position, V L 0 ~V L 3 mutual distance (V L both arrows are labeled 0 -V L 1, V L 0 -V L 2 , VL 0 -V L 3 , V L 1 -V L 2 , V L 1 -V L 3, and V L 2 -V L 3 ), and the imaging position where the minimum value is maximized may be determined, considering the positional relationship between the frame of the K L alone is sometimes an imaging position in the frame of the K L, and the imaging position of the imaging elements adjacent to the K L is close .

  For example, in the high-resolution image data of FIG. 6, four short-time exposure pixels and five long-time exposure pixels are dispersedly arranged in nine 3 × 3 pixels corresponding to one image sensor. . However, in the example of FIG. 6, since the imaging positions of the adjacent imaging elements are not taken into consideration, there are places where the imaging positions are not uniformly distributed as viewed from the entire high-resolution image data.

  In view of this, the imaging position determination apparatus 10 determines the imaging positions so that the imaging positions are evenly distributed as viewed from the whole of the high-resolution image data in consideration of the imaging positions of adjacent imaging elements. Thereby, it is possible to prevent pixels derived from captured image data captured under the same imaging conditions from being collected in the high-resolution image data, and to generate high-resolution image data with high accuracy without increasing the number of captured image data. be able to.

In FIG. 7 (b), it shows the imaging position ^V _R 0 ^~V _{R 3} of the image pickup device ^{K R} which is adjacent to the ^{K L.} As shown, in view of the image pickup position V ^R ₀ ~V ^R ₃ adjacent the image pickup device K ^R, so that the imaging position to determine the distance between the V ^L ₁ is added four (two arrows V ^L ₁ -V ^R ₀ , V ^L ₁ -V ^R ₁ , V ^L ₁ -V ^R _2, and V ^L ₁ -V ^R ₃ are attached), and V ^L ₀ , V ^L ₂ , V since same is true for ^L _3, it is necessary to calculate the distance of 16 points in total.

The imaging elements adjacent to the right of the image pickup element K ^L, since not only the image pickup device K ^R, the even imaging elements adjacent in a direction other than the rightward direction of the image pickup element K ^L, determine the positional relationship between the imaging position It will be.

Although the imaging device around eight directions across the imaging element K ^L can be considered cases of eight adjacent, due to the symmetry, to examine the positional relationship may only when the four halves. Since same as the examining and checking the positional relationship between the imaging position of the imaging element K ^R which is adjacent to the right of the image pickup device K ^L, the positional relationship between the imaging position of the imaging element adjacent to the left of the image pickup element K ^L That is why.

As described above, the imaging position of the frame of the imaging element K ^L maximizes the minimum value of the distance between the imaging position in the frame, also the imaging position in the frame of the image pickup device K ^L, the imaging device K ^L Is determined based on the condition that the minimum value of the distance to each imaging position within the frame of the image sensor adjacent to the image sensor is maximized.

  Next, a method for specifically obtaining these distances and determining the imaging position will be described below with reference to FIGS. In the following description, as shown in FIG. 6, super-resolution processing is performed using a total of nine captured image data obtained by capturing four images with an exposure time a and five images with an exposure time b. It is about the case of performing.

First, a method for obtaining the distance between the imaging positions in the frame of the imaging device will be described with reference to FIG. FIG. 8 is an explanatory diagram showing the four imaging positions included in the frame of the imaging element in two-dimensional coordinates. Here, a description will be given by introducing a two-dimensional coordinate system in which V ^B ₀ is the origin on the xy coordinate plane. Then, in the two-dimensional coordinate system, an area occupied by the image pickup device K ^B, was translated into position relative to the imaging position V ^B ₀ is a region R ^B. Further, in the figure, the rightward direction in the left-right direction is the positive direction of the x-axis, and the leftward direction in the horizontal direction is the positive direction of the y-axis. Further, here, the image pickup element K ^B for simplicity is the length of one side is a square h.

Here, the two-dimensional coordinates of the imaging positions V ^B ₀ , V ^B ₁ , V ^B ₂ , and V ^B ₃ are respectively (0, 0), (x1, y1), (x2, y2), and (x3, y3). ), The mutual distances of the imaging positions V ^B ₀ , V ^B ₁ , V ^B ₂ , V ^B ₃ can be expressed by the following equations (6-1) to (6-6).

Incidentally, B in the above equation indicates that the distance of the imaging device K ^B, p and q indicates that the distance between the imaging position V ^{B _p,} V ^B _q.

Next, with reference to FIG. 9, the imaging position in the frame of the image sensor K ^B, the method of obtaining the distance between the imaging position of the frame of the imaging elements adjacent to the image pickup device K ^B will be described. FIG. 9 is an explanatory diagram showing the positional relationship between the image sensor and the image sensor adjacent to the image sensor in two-dimensional coordinates.

As described above, the distance between the imaging positions to be obtained is the distance between the imaging positions of the imaging elements adjacent to the left, lower left, upper left, and lower directions with respect to a certain imaging element. In the figure, it is assumed the image pickup element K ¹ to the left of the image sensor K ^B, the image pickup element K ² in the lower ^left, the imaging device K ^{3 below,} the imaging device K ³ below are located, region R ^1, region R ^2, the region ^{R 3} is an area corresponding to the image pickup device ^K 1 ~K ^3.

As illustrated in FIG. 9A, the region R ¹ includes the imaging positions V ¹ ₀ , V ¹ ₁ , V ¹ ₂ , and V ¹ ₃ of the imaging device K ¹ . The imaging positions V ¹ ₀ , V ¹ ₁ , V ¹ ₂ , V ¹ ₃ move the imaging positions V ^B ₀ , V ^B ₁ , V ^B ₂ , V ^B ₃ by h in the negative direction of the x axis, respectively. since those with two-dimensional coordinates of the imaging positions _{^{V 1 0, V 1 1,}} V 1 2, V 1 3 , respectively, (- h, 0), (x1-h, y1), (x2-h , Y2) and (x3-h, y3).

Following the example described above, the imaging position ^V _B 0 included in the region _{^{R B, V B 1, V}} B 2, V and _{B 3,} the image pickup position ^V ₁ 0 included in the region _{^{R 1, V 1 1, V}} 1 The distance between ₂ and V ¹ ₃ is expressed as d ¹ _pq . Regarding the meaning of the subscript of d ¹ _pq , 1 indicates that it is the distance for the imaging positions included in the region R ^B and the region R ^1, and p and q are the imaging positions V ^B _p , V ¹ _q It indicates that the distance is between.

Here, connecting the imaging position ^{_{V B 0, V B 1,}} V B 2, V B 3 of area ^{R B,} and a region image pickup position ^V ₁ 0 of ^{_{R 1, V 1 1, V}} 1 2, V 1 3 Although there are 16 line segments in total, it is not necessary to obtain distances for all 16 line segments. For the following reasons, it is sufficient to obtain distances for nine of the line segments. That is, when p = q, d ¹ _pq = h. Further, when q = 0, d ^B _p0 <d ¹ _p0 . Therefore, such d ¹ _p0 need not be considered. Then, the distances to be obtained are only those expressed by the following nine formulas (7-1) to (7-9). Thus, by narrowing down the objects to be calculated in advance, the amount of calculation performed by the imaging position calculation unit 12 to determine the imaging position can be reduced.

Similarly, the distance between the imaging positions between the region R ² adjacent to the lower left region R ^B and the region R ^B shown in (b) of FIG. 9, region R ^B and the region R shown in (c) of FIG. 9 The distance between the imaging positions between the area R ³ adjacent to ^B and the distance between the imaging positions between the area R ² and the area R ³ shown in (d) of FIG. 9 can be obtained. .

  By performing such an operation, 6+ (4 × 9) = 42 distances can be obtained. In order to obtain an imaging position where the minimum value of 42 types of distances is maximized, nonlinear optimization processing may be performed on the evaluation formula represented by the following formula (8). Since Equation (8) is a continuous function having k imaging positions, that is, 2k-dimensional vector variables, the solution can be obtained by a numerical solution.

  As described above, it is possible to obtain imaging positions dispersed as viewed from the entire high-resolution image data. Although an example in which four imaging positions included in the frame of one imaging element are shown is shown here, an arbitrary number of imaging positions can be determined by a similar method. For example, in the example of FIG. 6, when five imaging positions to be imaged at the exposure time b are determined, the total number of distances to be obtained is 228. In this case, the same calculation method can be applied.

The method for distributing the imaging positions described above can be generally expressed as follows. That is, when k times of imaging are performed within the frame of one image sensor, the distance d ^B _mn between the imaging positions within the frame of the image sensor of interest is expressed as (x0, y0) = Assuming that (0, 0), (x1, y1),..., (Xk, yk), the distance between the imaging positions within the frame of the imaging element can be expressed by the following formula (9).

  In the above formula (9), m ≠ n, and m, n = {1,..., K}.

Further, a mutual distance d ⁱ _mn between the imaging position within the frame of the noted imaging element and each imaging position included in the frame of the imaging element adjacent to the imaging element is expressed by the following formula (10). it can.

Then, by solving the evaluation formula for these distances d ^B _mn and d ⁱ _mn is expressed by Equation (8), it is possible to determine the distributed imaging position.

  As described above, the imaging position determination device 10 virtually captures captured image data when the reference imaging position in which the reference pixel areas having a shape corresponding to one pixel of the captured image data are arranged in a matrix is set. The imaging position is set so that a central region having a shape corresponding to one pixel of the high-resolution image data located at the center of the shape corresponding to one pixel is included in the reference pixel region. In addition, for captured image data captured under at least one imaging condition, an imaging position is set such that the central region is arranged at a uniform or substantially uniform position with respect to the entire reference pixel region arranged in a matrix. To do.

  As a result, on the high resolution image data, the pixels generated based on the captured image data captured under the same imaging conditions are evenly arranged, so that halation or the like occurs in the captured image data captured under any of the imaging conditions. Even in this case, since interpolation processing can be performed on neighboring pixels, it is possible to prevent a reduction in accuracy of high-resolution image data.

[Flow of imaging position determination processing]
Next, an operation when the imaging position determination device 10 determines the imaging position in consideration of weighting and dispersion of the imaging positions will be described with reference to the flowchart shown in FIG. FIG. 10 is a flowchart illustrating an example of an imaging position determination process that is a process in which the imaging position determination device 10 determines an imaging position.

  Here, a case will be described in which super-resolution processing is performed by using a total of nine captured image data obtained by capturing four images with an exposure time a and five images with an exposure time b. That is, in the following, it is assumed that the record illustrated in FIG. 5 is stored in the imaging information storage unit 13.

  First, when the imaging position determination process is started in the imaging position determination device 10, the combination-specific imaging position calculation unit 17 of the imaging position calculation unit 12 determines the priority from the imaging information stored in the imaging information storage unit 13. The captured image group having the highest value is identified, and the number of captured images of the identified captured image group is read (S301). In the example of FIG. 5, “4”, which is the number of captured images of the captured image group G1 having the highest priority, is read.

  Subsequently, the combination-specific imaging position calculation unit 17 calculates an imaging position at which the imaging positions when imaging the number of images read out are substantially equal (S302). The method for calculating the imaging position is as described above in [Method for distributing imaging positions]. Here, since four imaging positions are determined within the frame of each imaging element, the imaging positions as shown in FIG. 11 are determined.

FIG. 11 is a diagram illustrating the imaging positions determined so that the imaging positions at which four of the nine imagings are performed are equal when nine imagings are performed. In the figure, K is shown an image pickup element, P _a shows the imaging position in the pixels of the high resolution image data. Therefore, the length of one side of P1 is 1/3 of the length of one side of the image sensor K. As shown in the figure, P1 is located at the center of each rectangle obtained by dividing the image sensor K into 2 × 2 quadrants, and is uniformly arranged on the entire surface of the high resolution image data.

Then, the combination-specific imaging position calculation unit 17 stores the calculated four imaging positions in the combination-specific imaging position storage unit 14. As described above, in the example of FIG. 5, the priority of the captured image group having a short exposure time is set high. That is, the number of captured images read in S301 indicates the number of captured images with the exposure time a. Hereinafter, the four imaging positions corresponding to the exposure time a calculated in S302, referred to as image pickup position group P _a.

  Next, the combination-specific imaging position calculation unit 17 identifies a captured image group having a lower priority from the imaging information stored in the imaging information storage unit 13 and reads the number of captured images of the identified captured image group. In the example of FIG. 5, “9”, which is the number of captured images of the captured image group G2, is read out. Subsequently, the combination-specific imaging position calculation unit 17 calculates an imaging position at which the imaging positions when imaging the number of images read out are substantially equal (S303). Here, since nine imaging positions are determined within the frame of each imaging element, the imaging positions as shown in FIG. 12 are determined.

FIG. 12 is a diagram illustrating the imaging positions determined so that the imaging positions at which nine imagings are performed are equal. In the figure, K represents an image sensor, and P _ab represents a pixel of high-resolution image data corresponding to the determined imaging position. As shown in the figure, P _ab is a position obtained by dividing the image sensor K by 9 of 3 × 3, and is uniformly arranged on the entire surface of the high resolution image data.

The combination-by-combination imaging position calculation unit 17 stores the calculated nine imaging positions in the combination-by-combination imaging position storage unit 14. As described above, here, it is assumed that nine pieces of captured image data are used for the super-resolution processing. That is, the number “9” of images picked up in S301 indicates the number of images picked up at the exposure times a and b. Hereinafter, the nine imaging positions corresponding to the exposure times a and b calculated in S303 are referred to as an imaging position group _Pab .

In S302, since the imaging positions of the four captured image data to be imaged at the exposure time a have been determined, the condition-specific imaging position selection unit 18 then selects the five captured image data to be imaged at the exposure time b. An imaging position is determined (S304). Specifically, the imaging position selection unit 18 by condition sets the five imaging positions among the nine imaging positions out of the nine imaging positions stored in the combination imaging position storage unit 14. The imaging position is selected so that Hereinafter, five imaging positions corresponding to the exposure time b selected in S304, referred to as image pickup position group P _b.

For example, it is possible to determine the five imaging position of the imaging element group P _b as FIG. FIG. 13 shows the arrangement of pixels of high-resolution image data corresponding to five imaging positions selected from nine imaging positions selected from the imaging positions determined so that the imaging positions for performing nine imagings are equal. FIG.

In the figure, K is shown an image pickup element, P _b represents the pixels of the high resolution image data corresponding to the selected imaging position. As shown in the drawing, the condition-specific imaging position selection unit 18 has five imaging positions from the imaging positions corresponding to each of the nine central areas when nine central areas are arranged in a matrix in the reference pixel area, that is, The imaging position group _Pb is selected and set.

As shown, by condition imaging position selecting unit 18 thus configured P _b is the center of the rectangle obtained by 9 divide the image pickup element K 3 × 3, upper left, lower left, upper right, and located in the lower right ing. Further, in the drawing, pixels of high-resolution image data corresponding to four imaging positions that are not selected as five imaging positions among the nine imaging positions are indicated by P _a ′. An imaging position group corresponding to the pixel P _a ′ is referred to as an imaging position group P _a ′.

In the example of FIG. 10, _'an example is shown for determining the and imaging position group P _b, the imaging position group P _a' imaging position group P _a after determining the imaging position group P _a and the image pickup position group P _b may be determined first, or the processing for determining these imaging position groups may be performed in parallel.

As described above, the combination by the image pickup position calculating unit 17 determines the imaging position group P _a of four imaging positions corresponding to the exposure time a, Conditional imaging position selection section 18 corresponds to the exposure time b 5 When an imaging position group P _b consisting of four imaging positions and an imaging position group P _a ′ consisting of four imaging positions are determined and stored in the combination-specific imaging position storage unit 14, the weighted imaging position determination unit 19 is notified accordingly. Is done.

Weighted imaging position determining unit 19 receives the notification, the imaging position group P _a stored in the combination by the image pickup position storage unit 14, reads the P a _'. Then, the weighted imaging position determination unit 19 determines the four imaging positions obtained by the weighted average of the imaging position groups P _a and P _a ′ as final imaging positions corresponding to the exposure time a (S305).

Specifically, the weighted imaging position determination unit 19 determines each of the four imaging positions (imaging position group P _a ) corresponding to the exposure time a obtained in S302 and the imaging positions included in the imaging position group P _a ′. the distance between the respective (e.g., the distance between the two-dimensional coordinates indicating the center of the image pickup position) seeking, for each imaging position included in the captured position group P _a, the image pickup position group P _a closest position and the captured position Specify the imaging position included in '.

That is, in this and the imaging position is included in the captured position group P _a, at the imaging position included in the imaging position group P _{a ',} a pair of imaging position recently become neighbor is identified. In this example, the imaging position groups P _a and P _a ′ are each composed of four imaging positions, so that four pairs are identified. Note that for one imaging position, when the imaging position of the same distance there are multiple, select one imaging position from the imaging position of the same distance, and each imaging position included in the imaging position group P _a, A pair in which each imaging position included in the imaging position group P _a ′ is on a one-to-one basis is specified.

Weighting imaging position determining unit 19, for a pair of image pickup positions to be nearest specified above, by performing weighting using the weights stored in combination by weight storage unit 15, the imaging position group P _a Each imaging position is corrected, and four final imaging positions corresponding to the exposure time a are determined.

In other words, the weighted imaging position determining unit 19 includes an imaging position of the imaging position group P _a, in the imaging position of the imaging position group P _{a ',} and the imaging position and recent the neighbor imaging position of the imaging position group P _a The final imaging position is determined on the line segment connecting the two. The position on this line segment is determined by the weight.

The imaging position group P _a, so are determined in consideration of dispersion of the imaging position in the adjacent image pickup device, when determining the imaging position group P _a as an imaging position corresponding to the exposure time a, In the high-resolution image data, pixels derived from the captured image data captured at the exposure time a are uniformly distributed (see FIG. 11).

In other words, the imaging of the imaging position group P _a when applied to an exposure as halation in the captured image data imaged has occurred overexposed pixels occurs at time b, a short exposure time a than the exposure time b Since pixels derived from the captured image data are uniformly distributed, overexposed pixels can be interpolated with neighboring pixels in the high-resolution image data.

On the other hand, the imaging position group P _a is deviated from the imaging position group P _a ′, which is a general imaging position as described with reference to FIG. _{When a} is determined as the imaging position corresponding to the exposure time a, the accuracy of the high-resolution image data may be lower than when the imaging position group P _a ′ is determined as the imaging position corresponding to the exposure time a. Conceivable.

Therefore, weighting the imaging position determining unit 19, the interpolation of overexposure or underexposure pixels, in order to appropriately set the balance between the accuracy of high-resolution image data, by performing weighted imaging position P _a and P _a ' The final imaging position is determined between That is, when importance is attached interpolation overexposure or underexposure pixels, the weight of the imaging position group P _a is set larger. On the other hand, when the need for interpolation of overexposed or underexposed pixels is low, the weight of the imaging position group P _a ′ is set large.

For example, the final imaging position after weighting is as shown in FIG. FIG. 14 is a diagram illustrating an example of the finally determined imaging position. P _a in the figure shows the pixels of the high resolution image data corresponding to the imaging position to the imaging exposure time a, P _b in the figure, the high-resolution image data corresponding to the imaging position to the imaging exposure time b A pixel is shown.

In the example of FIG. 14 shows an example that emphasizes interpolation overexposed pixels, the weight of the imaging position group P _a is 1, the weight of the imaging position group P _{a 'is} set to 0. In this case, as illustrated, P _{a 'is} the imaging position group is not considered, is determined as the imaging position imaging position group P _a is directly corresponding to the exposure time a.

Of course, the setting of the weight is not limited to the above example. For example, the weight of the imaging position group _{P a} 0.5, the weight of the imaging position group _{P a} 'may be set to 0.5. In this case, the middle point of the imaging position group P _a and the imaging position group P _{a 'is} determined as the image pickup position corresponding to the exposure time a. In other words, the weights only need to be set so that the sum of the weights of the respective imaging position groups to be weighted becomes 1.

In the present embodiment, 1 the weight of the imaging position group P _a, it is assumed that fixing the weight of the imaging position group P _{a '0,} it is preferably configured to weight can be changed. As a result, for example, after confirming the appearance of the generated high-resolution image data (edge appearance, blurring, etc.), the optimum weight is determined by changing the weight according to the confirmation result. It is also possible to generate high-resolution image data.

  Since all the imaging positions are determined by the above processing, the weighted imaging position determining unit 19 determines the determined imaging position and the imaging condition that can be placed at the imaging position (in this example, information indicating whether the exposure time is a or b). Are associated with each other and stored in the imaging condition storage unit 32. Then, when the imaging control unit 31 drives the imaging device 20 based on the imaging conditions stored in the imaging condition storage unit 32, imaging is performed at the imaging position determined by the imaging position determination device 10, and captured image data Is acquired.

  The imaging condition can be a data structure as shown in FIG. 15, for example. FIG. 15 is a diagram illustrating an example of a data structure of imaging conditions. As illustrated, the imaging conditions include items of an image number that is an identifier of the captured image data, an imaging condition (in this example, an exposure time), an imaging X coordinate that indicates an imaging position, and an imaging Y coordinate.

  In the example of FIG. 15, specific numerical values are shown as the imaging conditions. This numerical value may be set by the imaging position determination device 10 or may be set by another device (for example, the control device 30). The numerical value is preferably configured so that the user of the image processing system 1 can input, set, and change the numerical value.

  In the image processing system 1, the imaging control unit 31 drives the imaging device 20 using such data, so that imaging is performed at an imaging position as illustrated in FIG. 14 and captured image data is acquired. Then, the obtained captured image data is subjected to the above-described dynamic range expansion super-resolution processing, thereby generating high-resolution image data with an expanded dynamic range.

[About super-resolution processing]
In the above-described dynamic range expansion super-resolution processing, an example in which a plurality of captured image sets are used has been described. However, the imaging position determination device 10 determines the imaging positions of one set of captured image sets. Therefore, the processing in the case where the super-resolution processing is performed using the captured image data captured at the imaging position determined by the imaging position determination device 10 is partially different from the above-described processing. Here, a process in the case where the super-resolution processing is performed using the captured image data captured at the imaging position determined by the imaging position determination device 10 will be described with reference to FIG.

  When the imaging position determination device 10 stores the imaging conditions in the imaging condition storage unit 32 and the captured image data captured based on the imaging conditions is stored in the low resolution image storage unit 33, the normalization processing unit 43 Based on the imaging conditions stored in the imaging condition storage unit 32, the captured image data stored in the low resolution image storage unit 33 is normalized. Since the imaging position determination apparatus 10 determines an imaging position used for imaging one set of captured image data necessary for generating one piece of high-resolution image data, here, between the one set of captured image data. Normalization is performed.

  After normalization, the region extraction unit 47 may perform region extraction to generate high-resolution image data, or the high-resolution image data may be generated in the same procedure as in general super-resolution processing. It is also possible to generate. In this case, the pseudo low-resolution image generation unit 46 performs pseudo-low-resolution image data obtained by reducing the resolution of the high-resolution image data stored in the high-resolution image storage unit 34, and the captured image data after the normalization. The difference image data is generated by the difference image generation unit 45. Then, the high resolution image data can be generated by substituting the difference image data into the evaluation function to obtain the evaluation value and repeating the process of updating the high resolution image data so that the evaluation value becomes small.

[About three or more imaging conditions]
In the above description, the example in which the imaging position is determined when there are two imaging conditions has been described. However, the imaging position determination device 10 can determine the imaging position even when there are three or more imaging conditions. Here, an example in which the imaging position is determined when there are three imaging conditions will be described with reference to FIG. FIG. 16 is a flowchart illustrating an example of a process flow when the imaging position determination apparatus 10 determines the imaging position when there are three imaging conditions.

  Here, it is assumed that the captured image information storage unit 13 stores “captured image group G11”, “captured image group G12”, and “captured image group G13” as captured image information. Further, it is assumed that “A” images are set in the “number of imaged images” of the “imaged image group G11”, and “1” is set in the “priority”. Then, it is assumed that “A + B + C” is set in the “number of captured images” in the “captured image group G12”, “3” is set in the “priority”, and the “number of captured images” in the “captured image group G13” is set. Is “C” and “2” is set for “priority”.

  The captured image group G11 is captured at the exposure time a, the captured image group G12 is captured at the exposure time b, and the captured image group G13 is captured at the exposure time c. The relationship between the exposure times is a <b <c.

  As described above, when the number of imaging conditions is three, the dynamic range can be further expanded as compared with the case where the number of imaging conditions is two. That is, when a part of the captured image data captured at the exposure time b is overexposed, the brightness of the overexposed pixel is obtained by using the captured image data captured at the exposure time a with a shorter exposure time. Values can be interpolated. Similarly, when a part of the imaged image data imaged at the exposure time b is blacked out, the image of the blackout pixel is obtained by using the imaged image data imaged at the exposure time a having a shorter exposure time. Luminance values can be interpolated.

  First, when the imaging position determination process is started in the imaging position determination device 10, the combination-specific imaging position calculation unit 17 of the imaging position calculation unit 12 determines the priority from the imaging information stored in the imaging information storage unit 13. The captured image group having the highest value is identified, and the number of captured images of the identified captured image group is read. In this example, “A”, which is the number of captured images of the captured image group G11 having the highest priority, is read.

Then, the combination-specific imaging position calculation unit 17 obtains imaging positions at which the imaging positions are dispersed when the above-described readout number of images is captured (S401). The method for calculating the imaging position is as described above in [Method for distributing imaging positions]. In the following description, it referred to A number of imaging position determined in step S401 and the imaging position group P _a. The combination-specific imaging position calculation unit 17 stores the obtained imaging position in the combination-specific imaging position storage unit 14.

  Next, the combination-specific imaging position calculation unit 17 identifies a captured image group having the second highest priority from the imaging information stored in the imaging information storage unit 13, and reads the number of captured images of the identified captured image group. . In this example, “C”, which is the number of captured images of the captured image group G13, is read out.

Then, the combination-specific imaging position calculation unit 17 calculates the imaging positions at which the imaging positions are dispersed when the above-described read number of images is captured (S402). The method for calculating the imaging position is as described above in [Method for distributing imaging positions]. In the following description, the C imaging positions obtained in S402 are referred to as an imaging position group _Pc . The combination-specific imaging position calculation unit 17 stores the obtained imaging position in the combination-specific imaging position storage unit 14.

  Further, the combination-specific imaging position calculation unit 17 identifies the captured image group having the lowest priority from the imaging information stored in the imaging information storage unit 13 and reads the number of captured images of the identified captured image group. In this example, “A + B + C”, which is the number of captured images of the captured image group G12, is read.

Then, the combination-specific imaging position calculation unit 17 calculates the imaging positions at which the imaging positions are dispersed when performing the imaging of the read number of images (S403). The method for calculating the imaging position is as described above in [Method for distributing imaging positions]. In the following description, the A + B + C imaging positions obtained in S403 are referred to as an imaging position group _Pabc . The combination-specific imaging position calculation unit 17 stores the obtained imaging position in the combination-specific imaging position storage unit 14.

When the combination-specific imaging position calculation unit 17 obtains the imaging position group corresponding to the captured image groups G11 to G13 as described above, the condition-specific imaging position selection unit 18 acquires the imaging position group from the imaging position group P _abc. each imaging position and distance contained in P _a selects the a-number of the imaging position close this and imaging position group P _{a '.} Further, the imaging position selection unit 18 by condition sets the remaining B + C imaging positions as the imaging position group P _bc (S404).

Subsequently, the condition-specific imaging position selection unit 18 selects C imaging positions that are close to each imaging position included in the imaging position group P _c from the obtained imaging position group P _bc , and selects these imaging positions. Let it be a group P _c ′. The condition-specific imaging position selecting unit 18, the rest of the B + C-number of the imaging position and the imaging position group _{P b} (S404). This is not for the imaging position group P _b is performed weighting is determined as the imaging position for imaging in this state exposure time b.

On the other hand, the final imaging position is determined by weighting with respect to the imaging position where imaging is performed at the exposure time a or c. That is, the combination-specific imaging position calculation unit 17 and the condition-specific imaging position selection unit 18 determine the imaging positions of the captured image groups P _a , P _c , P _a ′, and P _c ′, and then weight the imaging positions. This is transmitted to the determination unit 19.

Weighting imaging position determining unit 19 which receives the transmission calculates the A-number of the final imaging position for imaging by exposure time a by performing weighted out with captured images P _a and P _{a '.} Further, the weighted imaging position determination unit 19 performs weighting with the captured image groups P _c and P _c ′ and calculates C final imaging positions to be imaged at the exposure time c (S406).

  With the above processing, A image pickup positions to be imaged at the exposure time a, B image pickup positions to be imaged at the exposure time b, and C image pickup positions to be imaged at the exposure time c are obtained. The position is stored in the imaging condition storage unit 32, and the imaging position determination process ends.

As described above, even when there are three types of imaging conditions, as in the case where there are two types of imaging conditions, first, the imaging positions for imaging under the imaging conditions with high priority (the imaging position group P _a in FIG. 16 imaging position groups P _a , P _c ) are obtained by the method described in the above [Method for distributing imaging positions].

The imaging position for imaging by the low priority imaging conditions (imaging position group P _b in FIGS. 10 and ₁₆₎ is a normal imaging position (see FIG. 12, FIG. 13). Also, among the normal imaging position, not to have been determined as an imaging position (imaging position group P _a 'of FIG. 10, the imaging position group P _a in FIG. 16', P _c '), the [image pickup position image pickup position obtained by the method described in a distributed approach to] (imaging position group P _a in FIG. _10, the imaging position group P _a in FIG. _16, P _c) pairs out to be formed. Then, the final imaging position is determined by performing weighting between the formed pairs.

In the example of FIG. 10, to determine the imaging position group P _b from the imaging position group P _ab, and the imaging position group P _{a 'the} imaging position has not been determined, in the example of FIG. 16, the imaging position group those from the P _abc the position corresponding to the imaging position group P _a is determined as the imaging position group P _{a ',} further imaging position as a position corresponding to the imaging position group P _c from the imaging position group P _bc It is determined as a group P _c ′. Then, it is the imaging position group P _b an imaging position that is not selected.

  That is, in the example of FIG. 10 and the example of FIG. 16, the determination process of the imaging position corresponding to the imaging condition with low priority is reversed. In this process, among general imaging positions as shown in FIGS. 18, 12, and 13, an imaging position corresponding to an imaging condition with a low priority, an imaging position corresponding to an imaging condition with a high priority, Any process may be adopted as long as it is selected.

[Application to other high resolution processing]
In the above, the example in which the super-resolution processing is performed using the captured image data captured at the imaging position determined by the imaging position determination device 10 has been described. However, the captured image data has a higher resolution than the super-resolution processing. It can also be used in the method.

  For example, it can be applied to a technique for increasing the resolution by interpolation. When the resolution of the captured image data is increased by interpolation, an interpolation process for generating each pixel data of the high resolution image data based on the pixel data of the surrounding captured image data is performed.

  Here, when the surrounding captured image data includes a region where valid image information such as a whiteout region is missing, the pixel data of the high-resolution image data is calculated from the pixels of the captured pixel data at a distant position. Is done. In such a case, the accuracy of the interpolation process decreases.

  On the other hand, when using the captured image data obtained by determining the imaging positions so that the imaging positions are dispersed using the imaging position determination method described above, the pixels to be interpolated are substantially uniform. Therefore, it is possible to prevent a decrease in the accuracy of the interpolation process. In this case, since captured image data obtained by performing imaging with a plurality of exposure times is used, interpolation processing is performed after the luminance value of the captured image data is normalized.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope shown in the claims. That is, embodiments obtained by combining technical means appropriately modified within the scope of the claims are also included in the technical scope of the present invention.

[Example of software configuration]
Finally, each block of the imaging position determination device 10, particularly the combination-specific imaging position calculation unit 17, the condition-specific imaging position selection unit 18, and the weighted imaging position determination unit 19 may be configured by hardware logic. Thus, it may be realized by software using a CPU.

  That is, the imaging position determination device 10 includes a central processing unit (CPU) that executes instructions of a control program that realizes each function, a read only memory (ROM) that stores the program, and a random access memory (RAM) that expands the program. ), A storage device (recording medium) such as a memory for storing the program and various data. An object of the present invention is a recording medium in which a program code (execution format program, intermediate code program, source program) of a control program of the imaging position determination apparatus 10 which is software that realizes the above-described functions is recorded in a computer-readable manner Can also be achieved by reading the program code recorded on the recording medium and executing it by the computer (or CPU or MPU).

  Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy (registered trademark) disks / hard disks, and disks including optical disks such as CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

  Further, the imaging position determination device 10 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Also, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  According to the present invention, the dynamic range of high-resolution image data can be efficiently expanded without increasing the number of times of imaging. Therefore, it can be widely applied to image processing systems that perform high resolution processing.

1, showing an embodiment of the present invention, is a block diagram illustrating a main configuration of an imaging position determination device. FIG. It is a block diagram which shows the principal part structure of the image processing system containing the said imaging position determination apparatus. It is a block diagram which shows the principal part structure of DR expansion super-resolution process part contained in the said image processing system. It is a flowchart which shows an example of the super-resolution process performed in the said DR expansion super-resolution process part. It is a data table which shows an example of the imaging information which the imaging information storage part of the said imaging position determination apparatus has memorize | stored. FIG. 6 is a diagram illustrating an example of an arrangement of pixels derived from captured image data captured under different imaging conditions in high-resolution image data. FIG. (A) illustrates captured image data used to generate high-resolution image data. FIG. 5B shows a state in which halation is occurring in captured image data having a long exposure time among captured image data used for generating high-resolution image data. ing. It is explanatory drawing which shows the method of grasping | ascertaining the positional relationship of an imaging position. It is explanatory drawing which represented the four imaging positions contained in the frame of an image sensor with the two-dimensional coordinate. It is explanatory drawing which represented the positional relationship of an image sensor and the image sensor adjacent to the said image sensor with the two-dimensional coordinate. It is a flowchart which shows an example of the imaging position determination process which is a process in which the said imaging position determination apparatus determines an imaging position. It is a figure which shows the imaging position determined so that the imaging position which performs 4 times imaging among 9 times might be equal, when performing 9 times imaging. It is a figure which shows the imaging position determined so that the imaging position which performs 9 times of imagings may become equal. It is a figure which shows arrangement | positioning of the pixel of the high-resolution image data corresponding to five imaging positions selected from nine imaging positions selected from the imaging position determined so that the imaging position which performs 9 times of imagings becomes equal. It is a figure which shows an example of the imaging position finally determined by the said imaging position determination apparatus. It is a figure which shows an example of the data structure of the imaging condition produced | generated by the said imaging position determination apparatus. In the said imaging position determination apparatus, it is a flowchart which shows an example of the flow of a process in the case of determining an imaging position when there are three imaging conditions. It is a flowchart which shows an example of the conventional super-resolution processing method. It is a schematic diagram which shows the imaging method of the captured image data used for a super-resolution process.

Explanation of symbols

1 Image processing system (image generator)
10. Imaging position determination device (imaging position determination means)
17 Combination-specific imaging position determination unit (first imaging position setting means)
18 Conditional imaging position selection unit (second imaging position setting means)
19 Weighted imaging position determination unit (weighted imaging position setting means)
35 Dynamic range expansion super-resolution processor (high-resolution image generation means)

Claims (10)

An image generation device that generates a single high-resolution image with an increased resolution from a plurality of low-resolution images that are captured under one of a plurality of types of imaging conditions while varying the imaging position of the same imaging target,
Imaging position determining means for setting imaging positions of the plurality of low resolution images;
High-resolution image generation means for acquiring the plurality of low-resolution images captured at the imaging position set by the imaging position determination means and generating the high-resolution image,
The imaging position determining means is
When a reference imaging position where reference pixel regions having a shape corresponding to one pixel of the low resolution image are arranged in a matrix is virtually set,
The plurality of low resolutions so that a central region having a shape corresponding to one pixel of the high resolution image located at the center of the shape corresponding to one pixel of the low resolution image is included in the reference pixel region. For a low resolution image that is set with an image pickup position and is picked up with at least one of the above-mentioned types of image pickup conditions, the central region is uniform with respect to the entire reference pixel region arranged in a matrix. Alternatively, an image generation apparatus comprising first imaging position setting means for setting a first imaging position that is an imaging position arranged at a substantially uniform position.   The first imaging position setting means calculates a distance between the center areas included in one of the reference pixel areas arranged in a matrix, and includes a center area included in the reference pixel area, and the reference The distance from each center area included in a reference pixel area adjacent to the pixel area is calculated, and the first imaging position is set so that the minimum value of the calculated distance is maximized. The image generating apparatus described.   An imaging position other than the first imaging position is selected from the second imaging positions corresponding to each of the central areas when the same number of central areas as the total number of imaging are arranged in a matrix in the reference pixel area. The image generating apparatus according to claim 1, further comprising a second imaging position setting unit that sets the second imaging position.   Of the second imaging positions, an imaging position that is the shortest distance from the first imaging position is selected from the imaging positions not selected by the second imaging position setting means, and the selected imaging position and the first imaging position are selected. The image generating apparatus according to claim 3, further comprising weighted imaging position setting means that corrects the first imaging position by weighted averaging the position.   5. The image generation apparatus according to claim 1, wherein the imaging condition is at least one of an exposure time, an aperture, an optical filter, and imaging sensitivity.   6. The image pickup condition according to claim 1, wherein the image pickup condition is set according to at least one of brightness, reflectance, and intensity of illumination irradiated on the image pickup target. The image generation apparatus according to item. An image generation device that generates a single high-resolution image with an increased resolution from a plurality of low-resolution images that are captured under one of a plurality of types of imaging conditions while varying the imaging position of the same imaging target,
When a reference imaging position where reference pixel regions having a shape corresponding to one pixel of the low resolution image are arranged in a matrix is virtually set,
The plurality of low-resolution images includes a central region having a shape corresponding to one pixel of the high-resolution image located in the center of the shape corresponding to one pixel of the low-resolution image in the reference pixel region. As well as being imaged,
The low-resolution image captured under one of the plurality of types of imaging conditions is such that the central region is arranged at a uniform or substantially uniform position with respect to the entire reference pixel region arranged in a matrix. An image generation apparatus characterized by being captured. An image generation method for an image generation apparatus that generates a single high-resolution image with an increased resolution from a plurality of low-resolution images obtained by imaging the same imaging target under any of a plurality of types of imaging conditions while varying the imaging position. Because
An imaging position setting step for setting imaging positions of the plurality of low-resolution images;
A high-resolution image generation step of acquiring the plurality of low-resolution images captured at the imaging position set in the imaging position setting step and generating the high-resolution image,
The imaging position setting step includes
When a reference imaging position where reference pixel regions having a shape corresponding to one pixel of the low resolution image are arranged in a matrix is virtually set,
The plurality of low resolutions so that a central region having a shape corresponding to one pixel of the high resolution image located at the center of the shape corresponding to one pixel of the low resolution image is included in the reference pixel region. For a low resolution image that is set with an image pickup position and is picked up with at least one of the above-mentioned types of image pickup conditions, the central region is uniform with respect to the entire reference pixel region arranged in a matrix. Alternatively, an image generation method comprising a first imaging position setting step of setting a first imaging position that is an imaging position arranged at a substantially uniform position.   An image generation program for operating the image generation apparatus according to any one of claims 1 to 6, wherein the image generation program causes a computer to function as each unit.   A computer-readable recording medium on which the image generation program according to claim 9 is recorded.

Images