JP4355744B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus that generates one high resolution image data from a plurality of low resolution image data.

  The resolution of an image captured by an imaging device such as an area sensor depends on the density of the image sensor that the imaging device has. Users of imaging devices are demanding imaging devices that can capture images with higher resolution, and camera manufacturers have released high-resolution cameras with high imaging element density one after another in order to meet user requirements for resolution. Yes.

  However, there is a limit to increasing the density of the image sensor, and there is a problem that the manufacturing cost of the imaging device increases due to the increase in density. Therefore, a plurality of times of imaging are performed on the same imaging target while shifting the imaging position little by little, and a plurality of captured image data is synthesized, and high-resolution image data having a higher resolution than the captured image data Conventionally, a high-resolution technique for generating is used.

  Image resolution enhancement techniques are generally divided into two types: image shift processing and super-resolution processing. In the image shift process, imaging is performed a plurality of times while gradually shifting the imaging position for the same imaging target. Then, the correspondence between the positions of the pixels included in each of the plurality of captured image data and the pixels of the high resolution image data is obtained, and the luminance value of each pixel of the captured image data is mapped to each pixel of the high resolution image data. Thus, high-resolution image data reflecting a plurality of captured image data is generated.

  In the image shift process, since it is necessary to map all the pixels of the captured image data to the pixels of the high resolution image data, it is necessary to fix the imaging position and the number of captured images of the captured image data. Therefore, the image shift process has an advantage that high-resolution image data can be generated at a low calculation cost, but has a problem that it is difficult to dramatically improve the resolution.

  On the other hand, in the super-resolution processing, first, similarly to the image shift processing, imaging is performed a plurality of times while gradually shifting the imaging position for the same imaging target. In the super-resolution processing, high-resolution image data is estimated, and the assumed high-resolution image data is reduced so that the difference between the estimated high-resolution image and the plurality of captured image data obtained by the plurality of imaging operations is reduced. Update the resolution image data.

  Such a super-resolution process is called a reconfigurable super-resolution process. As reconstruction-type super-resolution processing, for example, ML (Maximum-likelihood) method, MAP (Maximum A Posterior) method, POCS (Projection On to Convex Set) method and the like are generally known. 1 and Patent Document 1.

  In reconstruction super-resolution processing, the number of low-resolution image data (captured image data) to be used is not limited, so that the resolution can be dramatically improved. On the other hand, in the reconfigurable super-resolution processing, since iterative calculation is used for updating high-resolution image data, the calculation cost is generally higher than that of the image shift processing.

  Hereinafter, the super-resolution processing will be described in more detail. The super-resolution processing (reconstruction type) is formulated as an optimization problem of the evaluation function defined for the generated high resolution image. In other words, the super-resolution processing is an optimal evaluation function (error term) based on the square error between “estimated captured image data (high-resolution image data)” and “observed captured image data (captured image data)”. It will be reduced to the problem. Note that since the super-resolution processing is an optimization calculation with a very large number of unknowns, for example, an iterative calculation method such as the steepest descent method is often used to optimize the evaluation function.

  When the characteristics of the high resolution image data are known, an evaluation function (constraint term) based on the characteristics of the high resolution image data may be added to the evaluation function (error term). For example, when it is known in advance that the edge component of the high resolution image data is small, an evaluation function (constraint term) based on image data obtained by multiplying the high resolution image data by a Laplacian filter may be added. As a result, the fact that the edge component of the high-resolution image data is small can be reflected in the super-resolution processing, so that the accuracy of increasing the resolution can be improved.

Here, the flow of processing in the conventional super-resolution processing method will be described with reference to FIG. FIG. 22 is a flowchart showing an example of a conventional super-resolution processing method. Here, it is assumed that one piece of high-resolution image data is generated from m pieces of picked-up image data obtained by picking up the same image pickup object while shifting the image pickup position little by little (m is a positive number). Note that m is determined according to how many times the resolution of the captured image data is increased. Specifically, in order to increase the resolution of the captured image data by x times, at least x ² captured image data are required.

  An imaging method of an imaging target will be described based on FIG. FIG. 23 is a diagram illustrating an example of an imaging method of captured image data used for super-resolution processing. In the figure, for simplicity, an example of generating one piece of high-resolution image data using nine pieces of picked-up image data, that is, an example in which m = 9 and the picked-up image data is three times higher in resolution. Show. Each captured image data is assumed to be composed of 25 pixels of 5 pixels in the vertical direction and 5 pixels in the horizontal direction as shown in the figure.

  In order to increase the resolution of the captured image data by three times, the image capturing is performed once at each image capturing position while moving the image capturing position by 1/3 pitch of one pixel of the captured image data. As a result, nine pieces of captured image data whose image capturing positions are shifted by 1/3 pitch of one pixel of the captured image data as illustrated are acquired. In the figure, the imaging positions of each captured image data are indicated by numerals 1 to 9. That is, the number given to each pixel indicates the imaging position. For example, each pixel of the captured image data described in the upper left of the figure is numbered 1, and this is because each pixel of the captured image data described in the upper left of the figure is captured. It shows that the image was taken at position 1.

  Note that in order to generate high-resolution image data that accurately reflects the shape and color of the imaging target, it is necessary to capture each captured image data under the same imaging conditions. Here, the imaging condition refers to at least one of exposure time (shutter speed), illumination intensity, imaging sensitivity, filter to be used, and iris. In the example of the figure, such an imaging condition is the same at each imaging position, and in the figure, this imaging condition is indicated by a. That is, the alphabetic character attached to each pixel indicates the imaging condition.

Here, the description returns to the flowchart. In the conventional super-resolution processing method, m pieces of picked-up image data ym from the first piece to the m-th piece obtained as described _above are acquired (S101). Incidentally, y _m is a representation of the m pieces of image data to m-th first picture above as a vector is expressed by the following equation (1).

  Note that the vector representation of the image data refers to a representation of the image data components as a single row vector arranged in a line as in the above equation (1). For example, when the size of the elements constituting the image data is W × H, the image data is expressed as a vector of 1 row × (W × H) columns.

  Here, as described above, the super-resolution processing is an evaluation function based on a square error between “estimated captured image data (high-resolution image data)” and “observed captured image data (captured image data)”. This results in an optimization problem of (error term). That is, in order to perform the super-resolution processing, a plurality of captured image data and high-resolution image data estimated from the plurality of captured image data are necessary.

However, at the stage of obtaining the captured image data y _m, it can not acquire the high-resolution image data to be estimated from the captured image data y _m. Therefore, in the next step, initial high-resolution image data h is acquired (S102). The initial high-resolution image data h is image data having a higher resolution than the captured image data (the number of pixels constituting the image is large). Note that h represents high-resolution image data as a vector.

  Here, the initial high-resolution image data will be described with reference to FIG. FIG. 24 is a diagram illustrating an example of initial high-resolution image data. In addition, in the same figure, similarly to the captured image data in the example of FIG. 23, an example is shown in which captured image data composed of 25 pixels of 5 pixels in the vertical direction and 5 pixels in the horizontal direction is tripled in resolution.

  When the captured image data composed of 25 pixels of 5 × 5 pixels is three times higher in resolution, high-resolution image data having a total of 225 pixels of 5 × 3 pixels and 5 × 3 pixels is obtained. Will be generated. Therefore, data representing the image data of 5 × 3 pixels in the vertical direction and 5 × 3 pixels in the horizontal direction as the vector is prepared in advance as the initial high resolution image data h. When the resolution of the captured image data is increased to 3 times, the size of each pixel of the high-resolution image data is 1/3 times the size of each pixel of the captured image data.

  Note that the initial high-resolution image data is updated so that the difference from the captured image data is reduced by the super-resolution processing, so the initial value of the high-resolution image data, that is, the luminance value of each pixel of the initial high-resolution image data. May be any value. For example, a gray image, black image, or white image with a uniform entire surface can be used as the initial high resolution image data. Further, any one selected from the captured image data may be used as the initial high resolution image data.

Here, as can be seen from the comparison between FIGS. 23 and 24, in the initial high-resolution image data h and captured image data y _m, vary in size and number of pixels in each pixel, also different imaging position of the imaging target since it is, it can not be directly calculated the difference between the high resolution image data h and captured image data y _m. Therefore, when acquiring the captured image data y _m and the high resolution image data h at S101 and S102, the high-resolution image data h to the lower resolution with aligning the captured image data y _m, m pieces of the pseudo low resolution image Data B _m h is generated (S103).

Incidentally, B _m is a matrix of low resolution with a high-resolution image data h to align the captured image data y _m. B _m is a matrix represented by the product of M _{1 to} M _m and C, as shown in the following formula (2). Then, the matrix M ₁ ~M _m, after a high-resolution image data h are aligned at the imaging position of the captured image data y _m, sampling the luminance value of the pixel corresponding to each pixel position of the captured image data y _m processing Is a matrix. C is a matrix representing a convolution (a process of convolutioning and blurring an image) using a point spread function (PSF) that causes blur in the observation model (camera model of the imaging apparatus).

  The pseudo low resolution image data is, for example, as shown in FIG. FIG. 25 is a diagram illustrating an example of pseudo low-resolution image data. In the figure, an example of pseudo low resolution image data generated by aligning the high resolution image data of FIG. 24 with the captured image data of FIG. 23 is shown. As shown in the figure, the pseudo low-resolution image data is composed of nine pieces of image data corresponding to each of the captured image data imaging positions 1 to 9.

Next, the difference between the pseudo low resolution image data B _m h generated in S103 and the captured image data y _m is calculated, and the difference image data X _m is calculated (S104). This difference image data _Xm is an error between “estimated captured image data (high resolution image data)” and “observed captured image data (captured image data)”.

Then, by substituting the evaluation function difference image data X _m calculated above, obtaining the evaluation value E that indicates the high resolution accuracy (degree to which the high resolution image data and the captured image data are different) (S105 ). Various known functions can be applied as the evaluation function. For example, the evaluation value E can be obtained by using the evaluation function shown in the following mathematical formula (3).

The above mathematical formula (3) is an evaluation function used in a super-resolution processing method called a MAP method. Equation (3) is an equation composed of an error term and a constraint term, and B _m h−y _{m on} the right side of Equation (3), that is, difference image data X _m is an error term. A term including the characters w, L, and h on the right side of Expression (3) is a constraint term. Note that L is a matrix representing prior probability information of the high-resolution image data h, and w is a parameter representing the strength of the constraint term. By incorporating such a constraint term into the evaluation function, more accurate super-resolution processing can be performed by reflecting the prior probability information of the high-resolution image data.

Value obtained by substituting the difference image data X _m in this formula (3) is an evaluation value E. The value of the error term as the high-resolution image data h is accurately reflect the captured image data y _m decreases, the evaluation value E is also reduced accordingly. Therefore, it can be said that the super-resolution processing is processing for searching for high-resolution image data h that minimizes the evaluation value E.

  Next, the calculated evaluation value E is compared with a preset threshold value (S106). When the evaluation value E is smaller than the threshold value (YES in S106), the high resolution image data h used for calculating the evaluation value E is finally obtained as a result of the super-resolution processing. The super-resolution processing is finished as the high-resolution image data h.

  The threshold value can be changed as appropriate according to the required accuracy of resolution enhancement. For example, when higher resolution and higher resolution are required, the threshold value may be set small, and higher resolution accuracy is not required, but when a rapid processing speed is required, the threshold value is set. Should be set larger. That is, the threshold value is a value for determining the accuracy of the super-resolution processing, and can be set as appropriate according to various conditions required for the super-resolution processing.

  On the other hand, when the value of the evaluation value E is equal to or greater than the threshold value (NO in S106), the high-resolution image data h is updated by repeatedly performing the calculation so that the value of the evaluation value E becomes small (S107). For example, the high resolution image data h can be updated by using the following mathematical formula (4).

Formula (4) is a formula used in the steepest descent method. According to equation (4), made from the high resolution image data h _k, by subtracting a value obtained by multiplying the weight β determined in advance in the difference image data, as the high resolution image data h _{k + 1} new is generated ing. By repeating this calculation, high-resolution image data h having a smaller evaluation value E is generated. The iterative calculation is not limited to the steepest descent method, and a known iterative calculation method can be applied.

  The high-resolution image data h updated in this manner is subjected to pseudo-resolution reduction in S103 again, and is used for calculating the evaluation value E in S104 and S105. In S106, the evaluation value is compared with the threshold value again. That is, in the super-resolution process, the processes of S103 to S107 are repeatedly performed until the evaluation value E is smaller than the threshold value. Then, the high-resolution image data h obtained when the evaluation value E is smaller than the threshold value is used as the final high-resolution image data h obtained as a result of the super-resolution processing, and the super-resolution processing ends. As a result, high-resolution image data h with high resolution at a desired magnification and desired accuracy can be obtained.

As described above, high-resolution image data can be generated from a plurality of low-resolution image data by super-resolution processing or image shift processing, thereby generating fine image data exceeding the resolution of the imaging device. It becomes possible.
Shigeki Sugimoto, Masatoshi Okutomi, “Super Resolution Processing of Images”, Journal of the Robotics Society of Japan, 2005, Vol.23, No.3, p.305-309 JP 2006-127241 A (published May 18, 2006)

  By the way, the imaging device has a narrower light range (dynamic range) that can be captured at a time than the human eye. For this reason, when an imaging target having a wide dynamic range is imaged, overexposed or underexposure may occur in the captured image data. A portion where overexposure or underexposure occurs is not data representing the color or shape of the imaging target.

  Therefore, if the multiple low-resolution image data (captured image data) used for the super-resolution processing or the image shift processing includes those with overexposure or underexposure, The color is not accurately reflected in the high-resolution image data, which causes a problem that the accuracy of the high-resolution processing is lowered.

  The present invention has been made in view of the above-described problems, and an object thereof is to cause overexposure and underexposure in a plurality of low-resolution image data (captured image data) used for super-resolution processing. An object is to realize an image processing apparatus or the like that can generate high-precision high-resolution image data in which the shape and color of an imaging target are accurately reflected even if the image is included.

  In order to solve the above problems, an image processing apparatus according to the present invention generates high-resolution image data obtained by increasing the resolution of an imaging target based on a number of low-resolution captured image data corresponding to the resolution-enhancing magnification. An image processing apparatus, wherein the high-resolution image data is aligned with one of the plurality of captured image data, converted to the same resolution as the captured image data, and the captured image data is captured The pseudo low resolution image data is generated by multiplying the point spread function obtained from the camera model, and the difference between the pseudo low resolution image data and the captured image data is processed for each of the plurality of captured image data. Based on the difference image data generated by the difference image generator and the difference image data generated by the difference image generator. An evaluation value indicating an error between the image data and the plurality of captured image data is calculated, and when the evaluation value is equal to or greater than a predetermined threshold, the high-resolution image data is updated so that the evaluation value becomes small And an appropriate exposure pixel determining unit that determines whether or not the luminance value is within a predetermined normal range for each pixel of the plurality of captured image data, and In one difference image data corresponding to one of the picked-up image data, a target pixel that exists at a position corresponding to a pixel determined by the appropriate exposure pixel determining unit not to be within a normal range in one of the plurality of picked-up image data The weighted difference image is obtained by performing the process of weighting the target pixel so that the contribution to the evaluation value of each of the plurality of difference image data is reduced. Anda generating unit, the image updating means is characterized by updating the high-resolution image data using a plurality of difference image data which the weighted difference image generation means went weighting.

  In addition, in order to solve the above-described problem, the image processing apparatus control method according to the present invention provides a high-resolution image-capturing target based on the number of low-resolution captured image data corresponding to the resolution-enhancing magnification. A method of controlling an image processing device that generates image data, wherein the high-resolution image data is aligned with one of the plurality of captured image data, converted to the same resolution as the captured image data, and A process of generating pseudo low-resolution image data by multiplying a point spread function obtained from the camera model of the imaging device that captured the captured image data, and taking a difference between the pseudo low-resolution image data and the captured image data. A difference image generation step for generating a plurality of difference image data, and the plurality of differences generated in the difference image generation step. Based on the image data, an evaluation value indicating an error between the high-resolution image data and the plurality of captured image data is calculated, and the evaluation value decreases when the evaluation value is equal to or greater than a predetermined threshold value. And an image update step for updating the high-resolution image data, and for each pixel of the plurality of captured image data, a proper exposure pixel determination for determining whether or not the luminance value is within a predetermined normal range Corresponding to a pixel that is determined not to be within a normal range in one of the plurality of captured image data in the appropriate exposure pixel determination step in one difference image data corresponding to one of the plurality of captured image data A process of weighting the target pixel so that the contribution to the evaluation value of the target pixel existing at the position to be reduced is reduced. A weighted difference image generation step performed on each of the image data, and the image update step updates the high-resolution image data using the plurality of difference image data weighted in the weighted difference image generation step. It is characterized by that.

  According to the above configuration, it is determined whether or not each pixel is within a predetermined normal range for each of a plurality of captured image data used to generate one piece of high-resolution image data. The normal range is a luminance value range (dynamic range) in which the pixel can be regarded as being properly exposed.

  Moreover, according to said structure, the contribution which the target pixel which exists in the position corresponding to the pixel judged not to be in a normal range in picked-up image data among the pixels of difference image data gives to an evaluation value becomes small. Weighting is performed as follows. Then, the high-resolution image data is updated using the weighted difference image data.

  Since the difference image data is data indicating the difference between the captured image data and the pseudo low resolution image data, the target pixel reflects the luminance value of the pixel determined not to be within the normal range in the captured image data. Has been. In other words, the luminance value of the target pixel is not data that accurately indicates the shape and color of the imaging target because the pixel value of the pixel of inappropriate exposure is reflected. Therefore, when high-resolution image data is updated using difference image data including such data, low-precision high-resolution image data that does not accurately reflect the shape and color of the imaging target is generated. End up.

  Therefore, according to the above configuration, weighting is performed so that the target pixel existing at the position corresponding to the pixel determined not to be within the normal range has a small contribution to the evaluation value. Thereby, the influence which the data originating in the improper exposure pixel has on the update of the high resolution image data can be reduced.

  Therefore, according to the above configuration, the shape and color of the imaging target are accurately reflected even when the captured image data includes improperly exposed pixels in which whiteout or blackout occurs. It is possible to generate high-resolution image data with high accuracy.

  In the image processing device, the plurality of captured image data includes captured image data captured under different imaging conditions, and the appropriate exposure pixel determination unit determines that the luminance value is a predetermined normal range. Each of the imaging images so that it can be considered that all of the set of captured image data is captured under one imaging condition selected from the different imaging conditions. It is preferable to include captured image correction means for correcting the luminance value of each pixel of the image data.

  In the conventional super-resolution processing, a plurality of captured image data used to generate a single piece of high-resolution image data was imaged under the same imaging conditions, but by performing correction as described above, High-resolution image data can be generated using captured image data captured under different imaging conditions. The imaging conditions are at least one of exposure time (shutter speed), imaging device to be used, illumination intensity, imaging sensitivity, filter to be used, and iris.

  Here, according to the above configuration, when the high resolution image data is updated, pseudo low resolution image data is generated from the updated high resolution image data. At this time, in the high resolution image data, the luminance value data of each pixel included in the region corresponding to one pixel of the pseudo low resolution image data is integrated as the luminance value data of one pixel of the pseudo low resolution image data.

  Therefore, in the high-resolution image data, when pixels derived from captured image data captured under different imaging conditions are included in an area corresponding to one pixel of the pseudo low-resolution image data, Luminance values will be integrated. That is, each pixel of the pseudo low-resolution image data reflects the luminance value of the pixel derived from the captured image data captured under different imaging conditions, so that the dynamic range of the generated high-resolution image data Will also expand.

  In addition, in order to solve the above-described problem, the image processing apparatus control method according to the present invention provides a high-resolution image-capturing target based on the number of low-resolution captured image data corresponding to the resolution-enhancing magnification. An image processing apparatus for generating image data, wherein the high-resolution image data is generated based on a plurality of sets of captured image sets, each of which includes a number of captured image data corresponding to the resolution enhancement magnification. The captured image data included in each captured image set is captured under the same imaging conditions, and the captured image data included in different captured image sets are captured under different imaging conditions. Appropriate exposure pixel determination for determining whether the luminance value is within a predetermined normal range for each pixel of each captured image data included in the plurality of captured image sets And the appropriate exposure pixel determining means determines whether or not the luminance value is within a predetermined normal range, and then the plurality of sets of imaging under one imaging condition selected from the different imaging conditions. From the plurality of sets of captured image sets, the captured image correction means for correcting the luminance value of each pixel of each captured image data so that all of the captured image data included in the image set can be regarded as captured. For each pixel of the captured image data included in the selected set, the luminance value of the replacement target pixel determined by the appropriate exposure pixel determination unit not to be within the normal range is set as the replacement target pixel in another captured image set. If there is a replacement candidate pixel that is determined to be within the normal range by the appropriate exposure pixel determination unit, the luminance value of the replacement target pixel is determined as the replacement candidate. Image synthesizing means for generating a set of composite captured image data by replacing with the luminance value of the pixel, and the high-resolution image data for one of the composite captured image data included in the set of composite captured image data And pseudo low-resolution image data generated by multiplying by the point spread function obtained from the camera model of the imaging device that captured the plurality of captured image data, The difference image data is obtained by taking the difference from the combined captured image data and generating the differential image data for each of the combined captured image data included in the set of combined captured image data. A high-resolution image data and the set of combinations based on the set of difference image data generated by the difference image generation means Image update means for calculating an evaluation value indicating an error from the captured image data, and for updating the high-resolution image data so that the evaluation value becomes smaller when the evaluation value is equal to or greater than a predetermined threshold value; The appropriate exposure pixel determination means further determines whether or not the luminance value is within a predetermined normal range for each pixel of the composite captured image data included in the set of composite captured image data; With respect to the evaluation value of the pixel of the difference image data existing at the position corresponding to the pixel of the composite captured image data determined by the appropriate exposure pixel determination means not being within the normal range in the set of composite captured image data. Weighting difference image generation means for weighting each pixel of a set of difference image data generated by the difference image generation means so that the contribution is reduced, and the image update Stage is characterized by updating the high-resolution image data using a plurality of difference image data which the weighted difference image generation means went weighting.

  According to the above configuration, a plurality of captured image sets with different imaging conditions are combined to generate combined captured image data. Further, in the composite captured image data, each captured image data included in one captured image set selected from the plurality of captured image sets includes a replacement target pixel (inappropriate exposure pixel), and the replacement target If there is a replacement candidate pixel (a pixel included in captured image data of another captured image set and an appropriate exposure pixel at a position corresponding to the replacement target pixel), the luminance value of the replacement target pixel Are generated by replacing the luminance values of the replacement candidate pixels.

  That is, the set of composite captured image data includes the captured images included in the captured image set in which the inappropriate exposure pixels included in the captured image data included in the selected captured image set are not selected. That is, the captured image set is generated by being replaced with appropriate exposure pixels included in the data.

  In addition, according to said structure, it can be considered that all the picked-up image data contained in the said several picked-up image set were imaged on one imaging condition selected from the different imaging conditions. Since the luminance value of each pixel of each captured image data is corrected, high-resolution image data can be generated using the composite captured image data generated by performing the above replacement. By correcting the brightness value and replacing the brightness value, the dynamic range of the composite captured image data is expanded as compared with the captured image sets.

  Therefore, in the set of composite captured image data, since the ratio of pixels in which the shape and color of the imaging target are accurately reflected is increased as compared with each of the captured image sets, the set of composite captured image data is By using this to generate high-resolution image data, high-precision high-resolution image data can be generated.

  Note that each of the captured image data included in one captured image set selected from the plurality of captured image sets includes a replacement target pixel (inappropriate exposure pixel), and the replacement candidate pixel exists. If not, it can be considered.

  Therefore, according to the above configuration, it is further determined whether or not the luminance value is within a predetermined normal range for the pixels of the generated set of composite captured image data. Based on the determination result, the difference image data is weighted.

  Therefore, according to the above configuration, each captured image data included in one captured image set selected from a plurality of captured image sets includes a replacement target pixel (inappropriate exposure pixel). Even when the replacement candidate pixel does not exist, it is possible to reduce the influence of improper exposure pixels and generate high-precision image data with high accuracy.

  Further, if the image processing system includes the image processing device and the imaging device, the imaging device captures a plurality of captured image data, and the image processing device captures the captured image data. Can be used to generate high-resolution image data, and even if the captured image data includes improperly exposed pixels with overexposure or underexposure, the shape and color of the imaging target It is possible to generate high-precision high-resolution image data in which is accurately reflected.

  The image processing apparatus and the imaging control apparatus may be realized by a computer. In this case, the image processing apparatus and the imaging control apparatus are operated by causing the computer to operate as each unit of the image processing apparatus and the imaging control apparatus. A control program for realizing the control device on a computer and a computer-readable recording medium on which the control program is recorded also fall within the scope of the present invention.

  As described above, the image processing apparatus according to the present invention aligns high-resolution image data with respect to one of a plurality of captured image data, converts the high-resolution image data to the same resolution as the captured image data, and captures the captured image data. A process of generating pseudo low-resolution image data by multiplying a point spread function obtained from the camera model of the imaging device that captured the image, and obtaining a difference between the pseudo low-resolution image data and the captured image data A difference image generation unit configured to generate a plurality of difference image data, and the high resolution image data and the plurality of difference images generated by the difference image generation unit. An evaluation value indicating an error from the captured image data is calculated, and when the evaluation value is equal to or greater than a predetermined threshold value, the high-resolution image is reduced so that the evaluation value becomes small. And an image update means for updating data, and for each pixel of the plurality of captured image data, appropriate exposure pixel determination means for determining whether the luminance value is within a predetermined normal range, In one difference image data corresponding to one of the plurality of captured image data, the appropriate exposure pixel determining unit exists at a position corresponding to a pixel determined to be not within the normal range in one of the plurality of captured image data. Weighting difference image generation means for performing weighting on the target pixel so that the contribution of the target pixel to the evaluation value is small for each of the plurality of difference image data. The means updates the high-resolution image data using the plurality of difference image data weighted by the weighted difference image generation means. It is formed.

  In addition, as described above, the control method of the image processing apparatus according to the present invention aligns the high-resolution image data with respect to one of the plurality of captured image data, and converts the high-resolution image data to the same resolution as the captured image data. The pseudo low resolution image data is generated by multiplying the point spread function obtained from the camera model of the imaging device that captured the captured image data, and the difference between the pseudo low resolution image data and the captured image data is calculated. A difference image generation step for generating a plurality of difference image data performed on each of the plurality of captured image data, and the high resolution based on the plurality of difference image data generated in the difference image generation step An evaluation value indicating an error between the image data and the plurality of captured image data is calculated, and the evaluation value is small when the evaluation value is equal to or greater than a predetermined threshold. And an image update step for updating the high-resolution image data so as to determine whether or not the brightness value of each pixel of the plurality of captured image data is within a predetermined normal range. In one of the difference image data corresponding to one of the plurality of captured image data in the determination step, the pixel determined to be not within the normal range in one of the plurality of captured image data in the appropriate exposure pixel determination step A weighted difference image generation step for weighting each of the plurality of difference image data further includes a process of weighting the target pixel so that the contribution of the target pixel existing at the corresponding position to the evaluation value is reduced. In the image update step, the plurality of difference image data weighted in the weighted difference image generation step. It is configured to update the high-resolution image data using.

  Therefore, even if the captured image data contains overexposure or underexposure, highly accurate high-resolution image data that accurately reflects the shape and color of the imaging target is generated. There is an effect that can be done.

[Configuration of Image Processing System 1]
An embodiment of the present invention will be described below with reference to FIGS. First, the configuration of the image processing system 1 of the present embodiment will be described with reference to FIG. FIG. 2 is a block diagram showing a main configuration of the image processing system 1. As illustrated, the image processing system 1 includes an imaging device 2 and a control device 3.

  The imaging device 2 is a device that performs imaging in accordance with an instruction from the control device 3, and includes a lens 11, an imaging element 12, and an actuator 13 as illustrated. That is, in the imaging device 2, the light reflected on the imaging target is optically imaged on the imaging element 12 in the imaging device 2 via the lens 11. Then, the imaging element 12 spatially discretizes and samples the image of the imaging target that has been optically formed, and converts it into an image signal, whereby the imaging target image is acquired as image data. ing.

  As the image sensor 12, for example, a charge-coupled device (CCD), a complementary mental-oxide semiconductor (CMOS), or the like can be used. Here, it is assumed that the imaging device 2 is an area sensor that performs imaging with a plurality of imaging elements 12 arranged in a matrix.

  The actuator 13 changes the relative position between the imaging target and the image sensor 12. The actuator 13 is fixed to the inner wall of the housing of the imaging device 2, and the imaging element 12 is supported by the actuator 13. That is, the actuator 13 changes the relative position between the imaging target and the imaging element 12 by moving the imaging element 12 two-dimensionally on a plane perpendicular to the optical axis connecting the imaging target and the lens 11. It has become.

  That is, when the actuator 13 changes the relative position between the imaging target and the imaging element 12, the imaging position (position on the imaging target in the imaging region) at which the imaging device 2 captures the imaging target changes. Yes. The actuator 13 may move the image sensor 12 not only in two dimensions but also in three dimensions including rotation.

  As the actuator 13, for example, a piezo actuator or a stepping motor can be used. The actuator 13 is not particularly limited as long as it can change the relative position between the imaging target and the imaging element 12, but here, a piezoelectric actuator is used.

  The control device 3 controls the operation of the imaging device 2 and performs super-resolution processing on the image data of the imaging target acquired by the imaging device 2 (hereinafter referred to as captured image data) to generate high-resolution image data. To do. According to the super-resolution processing executed by the control device 3, high-resolution image data with high accuracy can be generated even when the captured image data has overexposure or underexposure. Further, according to the super-resolution processing executed by the control device 3, high-resolution image data having a wider dynamic range than that of the conventional super-resolution processing is generated.

  As illustrated, the control device 3 includes an imaging control unit 21, an imaging condition storage unit 22, a low resolution image storage unit 23, a high resolution image storage unit 24, and a super-resolution processing unit 25. Thus, the function of the control device 3 as described above is realized.

  The imaging control unit 21 includes an actuator control unit 26 and an imaging timing control unit 27 as illustrated. The actuator control unit 26 controls the imaging position of the imaging device 2 by moving the actuator 13. Specifically, the actuator control unit 26 controls the timing of starting and stopping the movement of the actuator 13, the moving speed, the moving distance, and the like.

  The imaging timing control unit 27 controls imaging start and stop timings of the imaging apparatus 2. Specifically, the imaging timing control unit 27 is based on the information sent from the actuator control unit 26 indicating the current imaging position of the imaging device 2 and the imaging conditions stored in the imaging condition storage unit 22. By sending a control signal to the image pickup apparatus 2, the image pickup time at each image pickup position, that is, the exposure time is controlled.

  Here, the captured image data captured by the control of the imaging controller 21 will be described with reference to FIG. FIG. 3 is a diagram illustrating the positional relationship between the image sensor 12 and the pixels of the high resolution image data. In the figure, the letter “a” is written, each of the rectangles arranged in a matrix form indicates a pixel of the high-resolution image data, and a bold-line rectangle A indicates one of the image sensors 12. That is, the area included in the rectangle A is represented as one pixel in the captured image data.

  As shown in FIGS. 4A to 4D, the length of one side of the rectangle A is twice that of the pixels of the high resolution image data. That is, FIG. 3 shows the positional relationship between the imaging position (pixel position of the captured image data) and the pixel of the high resolution image data when the resolution of the captured image data is doubled.

  As shown in the figure, the imaging position is moved by the width of one pixel of the high resolution image data. The brightness value data of the imaging target acquired at each imaging position is the pixel located at the upper left among the pixels of the high-resolution image data included in the imaging position (in the illustrated example, the color is different from that of the other pixels). Is reflected in the high-resolution image data. By capturing an image in this way, a set of captured image data necessary for estimating the luminance value of each pixel of the high-resolution image data can be acquired.

  In the image processing system 1, when imaging is performed at a certain imaging position according to an instruction from the imaging timing control unit 27, the imaging apparatus 2 captures the captured image data, the imaging position at the time of imaging, and imaging conditions (for example, The exposure time is stored in the low-resolution image storage unit 23 in association with each other. The actuator control unit 26 shifts the imaging position by the pixel width of the high-resolution image data, and imaging is performed again according to an instruction from the imaging timing control unit 27.

  By repeating this, an entire image of the imaging target is captured. Then, the captured image data generated by the imaging is stored in the low-resolution image storage unit 23 as captured image data 28 associated with the imaging position and imaging conditions. In the following description, the number of captured image data necessary for generating one piece of high resolution image data is m (m is a positive number), and a set of m pieces of captured image data corresponding to one piece of high resolution image data. Is described as a captured image set.

  Although details will be described later, in the image processing system 1, the dynamic range of the high-resolution image data generated by the super-resolution processing can be expanded by using a plurality of captured image sets having different exposure times. Here, the number of captured image sets is n (n is a positive number). Therefore, in the image processing system 1, m × n pieces of captured image data are used to perform the super-resolution processing.

  The high resolution image storage unit 24 stores high resolution image data 29 generated by performing super-resolution processing using m × n pieces of captured image data stored in the low resolution image storage unit 23. Similar to the above-described conventional super-resolution processing, since it is necessary to use initial high-resolution image data in order to perform the super-resolution processing, the high-resolution image storage unit 24 is preliminarily used before the super-resolution processing. The initial high resolution image data is stored in.

  The initial high-resolution image data is not particularly limited as long as it is an image having a resolution desired to be generated by the super-resolution processing. For example, when the resolution of the captured image data is doubled, a black image, a white image, or a gray image in which the size of each pixel is 1/2 of the captured image data is uniform as the initial high-resolution image data. Can be used.

  In addition, an interpolation process may be performed on an arbitrary piece selected from the captured image data, and the number of pixels increased may be used as the initial high resolution image data. As described above, when the high-resolution image data is generated from the captured image data, the captured image data is reflected in the high-resolution image data before the super-resolution processing is performed. As a result, it is possible to reduce the number of repetitive calculations in the super-resolution processing, and it is possible to generate high-precision image data with high accuracy in a short time.

  As the above interpolation processing method, for example, in one piece of picked-up image data selected from m pieces of picked-up image data, the luminance value of a pixel picked up at a position where the image is to be picked up is picked up in high-resolution image data There is a method of setting the luminance value of the pixel corresponding to the same position as that of the target.

  Since the high resolution image data is data having a higher resolution than the captured image data, one pixel of the captured image data corresponds to a plurality of pixels of the high resolution image data. For example, when the resolution of the captured image data is tripled, one pixel of the captured image data may be set to 3 × 3 pixels of the high resolution image data.

  Further, in the above interpolation processing method, since the luminance values in a plurality of pixels of the high-resolution image data corresponding to one pixel of the captured image data are all set to the same luminance value, the processing is simple, but the captured image data is There is also a drawback that it is difficult to accurately reflect the high-resolution image data. Therefore, the luminance values at a plurality of pixels of the high-resolution image data corresponding to one pixel of the captured image data may be changed according to the luminance values of the pixels adjacent to the one pixel of the captured image data. Good.

  For example, when the resolution of the captured image data is tripled, the luminance value of the pixel that captured the position where the image is to be captured in any one captured image data selected from the m captured image data is captured. What is necessary is just to set it as the luminance value of the pixel located in the center of 9 pixels of 3x3 corresponding to the same position as the above of the object. Then, in the high-resolution image data, the luminance values of the eight pixels around the pixel located at the center of the 3 × 3 nine pixels are adjacent to the pixel that has imaged the same position as the above in the captured image data. What is necessary is just to determine using the luminance value of a pixel.

  According to the above method, since high-resolution image data in which captured image data is reflected more accurately is obtained, it is possible to further reduce the number of repetitive operations in super-resolution processing, and to achieve high-precision high-speed in a short time. It becomes possible to generate resolution image data.

  Further, by using a plurality of captured image data, it is possible to obtain high resolution image data in which the captured image data is reflected more accurately. For example, by mapping the luminance value at each pixel of a set of captured image sets to high resolution image data, it is possible to obtain high resolution image data in which the captured image data is reflected more accurately. The above-described processing for generating high-resolution image data by mapping using a set of captured image sets is so-called image shift processing.

  Furthermore, when at least one of the shape and color of the imaging target is known in advance, the design is image data indicating at least one of the shape and color of the imaging target and image data having a desired resolution (number of pixels) The data can also be used as initial high resolution image data. By using the design data, the design value of the imaging target can be reflected in the high-resolution image data, so the number of repeated operations in the super-resolution processing can be further reduced, and high accuracy with high accuracy in a short time. It becomes possible to generate resolution image data.

  The super-resolution processing unit 25 performs super-resolution processing for generating high-resolution image data by increasing the resolution of the captured image data stored in the captured image storage unit. Specifically, the super-resolution processing unit 25 generates a new high resolution based on the captured image data stored in the captured image storage unit and the high resolution image data 29 stored in the high resolution image storage unit 24. Image data is generated, and the generated high resolution image data is stored in the high resolution image storage unit 24. Note that details of the super-resolution processing executed by the super-resolution processing unit 25 will be described in detail later.

[Configuration of Super-Resolution Processing Unit 25]
Next, a detailed configuration of the super-resolution processing unit 25 will be described with reference to FIG. FIG. 1 is a block diagram illustrating a main configuration of the super-resolution processing unit 25. As illustrated, the super-resolution processing unit 25 includes a normalization unit 31, an image synthesis unit 32, a difference image generation unit (difference image generation unit) 33, a pseudo low-resolution image generation unit 34, and an area extraction unit (appropriate exposure pixel). A determination unit) 35, a weighted difference image generation unit (weighting difference image generation unit) 36, and an image update unit (image update unit) 37.

  When the captured image set includes images captured under different imaging conditions, the normalization unit 31 sets all the captured image data included in the captured image set to one of the different imaging conditions. The luminance value of each pixel of each captured image data is corrected so that it can be considered that the image is captured under one imaging condition. This makes it possible to perform super-resolution processing using captured image sets including captured image data captured under different imaging conditions.

  For example, consider the case where there are captured image data A, B, and C, and the exposure times of the captured image data are a, b, and c (a <b <c), respectively. In this case, with the exposure time a as a reference, the brightness values of the captured image data B and C can be expressed by, for example, the following mathematical formula (5).

  That is, the captured image data B with the exposure time b is b / a times as long as the exposure time is b / a times that of the captured image data A with the exposure time a. It is expected to be. Similarly, the captured image data C with the exposure time c is c / a times as long as the exposure time is c / a times that of the captured image data A with the exposure time a. It is expected that

  Therefore, when the captured image data A with an exposure time of a is used as a reference, the brightness value of the captured image data with an exposure time of b is corrected to a / b times, and the brightness of the captured image data with an exposure time of c seconds is corrected. By correcting the value by a / c times, all of the luminance values of the captured image data A to C can be made equal. The reference exposure time can be selected from any of the captured image data A to C.

  Note that the normalization method is not limited to the above example. For example, normalization may be performed by obtaining an average value of the luminance values of the captured image data and performing correction so that the luminance values of the respective captured image data are approximately the same based on the obtained average value. Also, in addition to the exposure time, when the imaging conditions such as the imaging device to be used, the illumination intensity, the imaging sensitivity, the filter to be used, and the iris are different, all the captured image data used for the super-resolution processing is The luminance value of each pixel of each captured image data is corrected (normalized) so that it can be considered that the image is captured under one of the different imaging conditions.

  Although details will be described later, the super-resolution processing unit 25 generates a composite captured image data by combining a plurality of captured image sets having different imaging conditions. Here, in order to synthesize captured image data having different imaging conditions, it is necessary to perform correction so that the imaging conditions of the respective captured image data can be regarded as the same. Therefore, the normalization unit 31 normalizes the luminance value for a plurality of captured image sets with different imaging conditions.

  The image composition unit 32 synthesizes a plurality of captured image sets with different imaging conditions normalized by the normalization unit 31 to generate composite captured image data. When the image synthesis unit 32 synthesizes the captured image data, the number of sets of captured images is 1. Therefore, the number of composite captured image data is m.

  The composite captured image data reflects the captured image data captured under a plurality of different imaging conditions. Therefore, by performing super-resolution processing using the composite captured image data, the dynamic of the generated high resolution image data is performed. The range can be expanded. A method for generating composite captured image data will be described later.

  The difference image generation unit 33 calculates the difference between the captured image data and the pseudo low-resolution image data as difference image data, and sends the calculated difference image data to the weighted difference image generation unit 36. When there are a plurality of captured image sets (n ≧ 2), the difference image generation unit 33 calculates the difference between the combined captured image data generated by the image composition unit 32 and the pseudo low-resolution image data as difference image data. To do. The pseudo low resolution image data generated by the pseudo low resolution image generation unit 34 is used.

  The pseudo low resolution image generation unit 34 reads the high resolution image data stored in the high resolution image storage unit 24 and reduces the resolution so that the resolution of the read high resolution image data is the same as that of the captured image data. . In addition, the pseudo low resolution image generation unit 34 performs conversion by PSF on the high resolution image data, and generates a plurality (m pieces) of pseudo low resolution image data aligned with each of the captured image data. The pseudo low-resolution image data is sent to the difference image generation unit 33 and used for generation of the difference image data.

  The area extraction unit 35 detects an inappropriate exposure area that is inappropriate for the super-resolution processing in the captured image data. The improper exposure area is an area having a luminance value outside the effective dynamic range of the imaging apparatus 2, and is an area where there is a high possibility that blackout or whiteout occurs. When captured image data including such a region is used for the super-resolution processing, there arises a problem that the accuracy of the super-resolution processing is reduced and the processing time required for the repetitive calculation is increased.

  Specifically, the area extraction unit 35 extracts, for each of the m × n pieces of captured image data, a first area whose luminance value is outside a predetermined normal range, and each captured image data of the extracted area Remember the top position. Here, in the captured image data, an area other than the extracted first area is referred to as a second area. That is, when the region extraction unit 35 extracts the first region, each captured image data is divided into a first region having a luminance value outside the normal range and a second region having a luminance value within the normal range. .

  Note that the normal range is a range of luminance values defined by an upper limit value and a lower limit value of luminance values that can be regarded as appropriate exposure. An area that shows a luminance value that exceeds the upper limit is an area where halation such as overexposure is considered to occur, and an area that shows a luminance value that is lower than the lower limit is considered to be in a blackened state. This is an area Therefore, the second region having the luminance value within the normal range is a region having the luminance value obtained by imaging within the effective dynamic range of the imaging device 2.

  For example, when the luminance value range of the captured image data is 256 gradations from 0 to 255, the luminance value range from 10 to 245 may be the normal range, that is, the second region. Then, the luminance value range from 0 to 9 where whiteout is expected to occur and the luminance value range from 246 to 255 where blackout is expected to occur are outside the normal range, that is, the first A region may be used. Of course, the above numerical range is an example, and the numerical range used for region extraction can be appropriately changed as necessary.

  The region extraction may be performed in units of pixels, or may be performed in units of blocks in which a plurality of pixels are one block. For example, the captured image data is divided into a plurality of blocks by setting an area of x × y pixels (x and y are positive numbers) as one block, and each block is first based on an average value of luminance values of pixels included in each block. You may make it judge whether it is classified into an area | region.

  The storage method in which the region extraction unit 35 stores the extracted region is not limited as long as the first region or the second region is indicated for all the pixels of each captured image data. By storing in this way, it is possible to speed up subsequent processing.

That is, the area extraction unit 35 sets a memory space Z _mn having the same size as that of m × n pieces of captured image data. Then, 1 is set in the pixel position on the memory space _Zmn corresponding to the pixel determined to be the first region in the captured image data, and the memory corresponding to the pixel determined to be the second region. 255 is set in the pixel position on the space _Zmn . As a result, in the memory space Z _mn , a value of 1 is stored at the position of the overexposed or blackened pixel, and a value of 255 is stored at the position of the pixel within the dynamic range.

Here, the data generated by the region extraction unit 35 as described above is referred to as extraction region data _Zmn . The extracted area data Z _mn is used for weighting of the difference image data by the weighted difference image generation unit 36 and is used for image composition by the image composition unit 32.

That is, the image compositing unit 32 refers to the extracted area data _Zmn when integrating the captured image data having the same imaging position and different exposure time into one image data. Integration is performed so that one area is reduced. When a plurality of sets of captured image sets are used, when the image composition unit 32 generates composite captured image data, the region extraction unit 35 uses the extracted region data Z _mn for each pixel of the composite captured image data. updating the extracted region data Z _m indicating whether belonging to the second region belongs to the first region.

Specifically, the image composition unit 32 refers to the extraction region data Z _mn to determine whether each pixel of the captured image data having the same imaging position and different exposure time belongs to the first region or the second region. judge. When a pixel belonging to the first region and a pixel belonging to the second region are detected in the pixel at the same position, the image composition unit 32 determines the brightness value of the pixel at the position in the composite captured image data as The luminance values of the pixels belonging to the two areas are adopted. The image synthesizing unit 32 updates the extracted area data Z _m pixel values of the position as the second region.

  When a plurality of pixels belonging to the second region are detected in the pixels at the same position, for example, the luminance value of the pixel having the longest exposure time may be employed. Further, when only pixels belonging to the first region are detected in the pixels at the same position, the luminance value of any one of the pixels is employed. Therefore, for example, with the captured image set having the longest exposure time as a reference, among the pixels included in the captured image set, the pixels belonging to the first region are pixels at the same position in the other captured image sets, and the second What is necessary is just to replace with the luminance value of the pixel which belongs to an area | region.

Thus, m × n post-normalized captured image data is integrated into m composite captured image data, and the extracted region data Z _mn generated based on m × n post-normalized captured image data There is updated to extract region data Z _m corresponding to m pieces of synthetic image data. Similarly to the extraction area data Z _m In extracted area data Z _mn, the position of the pixel in the first region is a value of 1 is stored, the position of the pixel of the second area is stored the value of 255.

  The composite captured image data generated by such composition processing reflects the captured image data captured at a plurality of exposure times, so that the dynamic range is expanded as compared with the captured image data. Therefore, by performing the super-resolution processing using the composite captured image data, it is possible to generate high-resolution image data having a wide dynamic range compared to the case of performing the super-resolution processing using the captured image data. .

  Furthermore, in the composite captured image data generated by such composition processing, if possible, the brightness value of the pixel in the first region is the brightness of the pixel in the second region where the position is the same in the other captured image data. Has been replaced by a value. Therefore, by performing super-resolution processing using the composite captured image data, the ratio of data useful for super-resolution processing is higher than when super-resolution processing is performed using captured image data. The accuracy of the super-resolution processing can be improved, and the calculation time of the super-resolution processing can be shortened.

  Here, a specific example of synthesis of captured image data will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of a method for synthesizing captured image data. In the figure, images are taken at m different image pickup positions from 1 to m at three exposure times a to c (a <b <c), and a set of m picked-up images. The example which acquired 3 sets of is shown. Here, only four regions P1 to P4 will be described for simplicity, but the same processing is performed for each pixel of the captured image data.

  As illustrated, P1 to P3 all have the same position on the captured image data. As described above, in the captured image data included in different captured image sets and having the same imaging position, the synthesis is performed on pixels having the same position in the captured image data (capturing the same position of the imaging target). In the combined captured image data, the result of combining the pixels P1 to P3 is reflected on P4 at the same position as P1 to P3. That is, P1 to P3 are regions to be combined, and the pixel P4 is a region after combining.

  For example, when P1 and P2 belong to the second region and P3 belongs to the first region, among the pixels belonging to the second region, the luminance value of P2 having the longest exposure time is the luminance of P4. Adopted as a value. That is, if at least one pixel to be synthesized belongs to the second area, it becomes the second area after synthesis. When all of P1 to P3 are the first region, any one of the luminance values of P1 to P3 may be set as the luminance value of the pixel after synthesis.

Weighted difference image generation unit 36 based on the extracted region data Z _m the region extraction unit 35 is generated, on the difference image data difference image generation unit 33 calculates, corresponding to the first image region and a second image area Weighting difference image data is generated. When the weighted difference image generation unit 36 performs weighting, the influence of the first region having a luminance value outside the normal range on the super-resolution processing is reduced, so that overexposure and blackout occur in the captured image data. Even in such a case, highly accurate super-resolution processing can be performed. Then, the weighted difference image generation unit 36 sends the generated weighted difference image data to the image update unit 37.

The weighting is not particularly limited as long as it is performed so that the contribution of the first image area to the super-resolution process is smaller than the contribution of the second image area. Here, in the extraction area data Z _m, the position of the pixels of the first area stores a value of 1, the position of the pixel of the second area so that the value of 255 is stored. Therefore, as a weight value obtained by multiplying the 1/255 to extract region data Z _m, multiplied by the difference image data.

  Thereby, in the difference image data, the luminance value of the pixel in the region which is the first region in the composite image data is 1/255. On the other hand, in the difference image data, the luminance value of the pixel in the region that is the second region in the composite image data is not corrected. Therefore, according to the above configuration, the contribution of the pixels in the first area in the composite image data is sufficiently lower than the contribution of the pixels in the second area in the composite image data. can do.

Here, an example has been described for weighting by multiplying the extracted area data Z _m in the difference image data, it is also possible to perform weighting without using the extracted area data Z _m and extracted area data Z _mn. For example, the weighted difference image generation unit 36 refers to the normalized captured image data or the luminance value at each pixel position of the captured image data, and the luminance value of each pixel of the difference image data is converted to the pixel in the first region. It may be determined whether it is calculated based on the pixel of the second region or the pixel of the second region, and a weight may be given based on the determination result.

  Further, here, an example has been described in which weighting is performed for two regions, the first region and the second region, but the region may be divided into three or more regions. For example, when the amount of incident light and the output are not linear in the effective dynamic range of the imaging device 2, it is preferable to divide into three or more regions and give weights corresponding to the regions.

  As a specific example, when the luminance value is 256 gradations from 0 to 255, the weight in the range of the luminance value 0 to 2 is set to 0, the range of the luminance value 3 to 7 and the luminance value is The weight in the range of 248 to 252 may be ¼ of the weight of the second region, and the weight in the range of the luminance value from 253 to 255 may be zero. In this way, high-accuracy and high-precision reflection of captured image data is achieved by performing stepwise weighting so that the luminance value is closer to the effective dynamic range, so that the contribution to the super-resolution processing increases. Resolution image data can be generated.

  The image updating unit 37 updates the high resolution image data using the weighted difference image data. Then, the image update unit 37 stores the updated high resolution image data in the high resolution image storage unit 24.

[Flow of super-resolution processing]
Next, super-resolution processing executed in the image processing system 1 will be described with reference to FIG. FIG. 5 is a flowchart illustrating an example of super-resolution processing executed in the image processing system 1.

  First, the imaging control unit 21 of the control device 3 instructs the imaging device 2 to image an imaging target. Upon receiving the instruction, the imaging control unit 21 reads the imaging condition from the imaging condition storage unit 22, and performs imaging of the imaging target according to the read imaging condition (S1). Specifically, the imaging control unit 21 executes imaging at m different imaging positions while shifting the imaging position little by little for the same imaging target. Then, the imaging control unit 21 changes the exposure time, and again executes imaging at the m different imaging positions under the imaging condition after the change of the exposure time. The m × n sets of captured image data captured in this way are stored in the low-resolution image storage unit 23.

  The m × n sets of captured image data obtained as described above are, for example, as shown in FIG. FIG. 6 is a diagram illustrating an example of a captured image set. Here, as in the conventional example of FIG. 23, an example of generating one piece of high-resolution image data using nine pieces of picked-up image data, that is, m = 9, and the picked-up image data is three times higher. An example of resolution is shown. Each captured image data is assumed to be composed of 25 pixels of 5 pixels in the vertical direction and 5 pixels in the horizontal direction as shown in the figure. As shown in the figure, it is assumed that three sets of picked-up images taken at different exposure times are used.

  That is, the illustrated captured image set 1 is a set of captured image data captured at the exposure time a at nine imaging positions from the imaging position 1 to the imaging position 9. Similarly, the picked-up image set 2 is a set of picked-up image data picked up at the nine pick-up positions from the pick-up position 1 to the pick-up position 9 at the exposure time b. A set of captured image data captured at an exposure time c at nine imaging positions up to the imaging position 9. The exposure time is a <b <c.

  In addition, the image pickup position is displaced by 1/3 pitch of one pixel width of the picked-up image data. That is, the imaging position 2 is a position moved from the imaging position 1 to the right in the figure by 1/3 of one pixel width of the captured image data, and the imaging position 4 is from the imaging position 1 downward in the figure. This is a position moved by 1/3 of one pixel width of the captured image data.

  In this way, by performing imaging by shifting the imaging position at 1/3 pitch of one pixel width of the captured image data, nine imaging operations are performed for each pixel of the captured image data. As a result, each pixel value of the high-resolution image data having three times the resolution of the captured image data, that is, nine times the number of pixels can be obtained. That is, when the resolution is increased x times, the imaging position may be shifted at 1 / x pitch of 1 pixel width of the captured image data.

  As shown in the figure, when imaging is performed with three exposure times and three captured image sets are acquired, captured image data captured with three exposure times are obtained for each of the imaging positions 1 to 9. Generated. Here, for the sake of simplicity, an example in which only the exposure time is changed as the imaging condition will be described. However, imaging conditions other than the exposure time may be changed. Further, the imaging conditions may be different between captured image data included in one set of captured image sets. Furthermore, although an example in which super-resolution processing is performed using a plurality of sets of captured image sets will be described here, super-resolution processing can be performed if there is at least one captured image set.

Now, the description returns to the flowchart. When imaging of m × n sets is completed and m × n captured image data y _mn is stored in the low-resolution image storage unit 23, the region extraction unit 35 and the normalization unit 31 store the m stored above. X n pieces of captured image data y _mn are acquired, and the imaging conditions used for imaging the m × n pieces of captured image data y _mn are read from the imaging condition storage unit 22 (S2).

Area extracting unit 35 acquires the captured image data y _mn, from each of the m × n pieces of captured image data y _mn, based on the luminance value distribution of the captured image data y _mn, overexposure area or underexposure region like The improper exposure area is identified and extracted as the first area (S3). Then, the region extraction unit 35 generates extracted region data Z _mn indicating whether each pixel of the m × n normalized captured image data belongs to the first region or the second region.

For example, in the case of using captured image data of 25 pixels × 9 sheets × 3 sets as shown in FIG. 6, the area extraction unit 35 extracts data having a data area of 25 pixels × 9 sheets × 3 sets = 675 pixels. Area data _Zmn is set. Then, for each pixel position of the set extraction area data _Zmn , if the pixel of the captured image data corresponding to the pixel position is the first area, a value of 1 is stored, and the captured image data corresponding to the pixel position If the pixel is the second region, a value of 255 is stored.

On the other hand, the normalization unit 31 acquires the exposure time of each captured image data y _mn from the acquired imaging conditions, and normalizes the captured image data y _mn based on the acquired exposure time (S4). Then, the normalization unit 31 sends the captured image data y _mn generated by normalization to the image composition unit 32.

  For example, in the example of FIG. 6, the imaging conditions (exposure time) are different for each of the captured image sets 1 to 3, so that each captured image set can be regarded as captured with the same exposure time. Is done. Specifically, the normalization unit 31 corrects the luminance value of the captured image data with the exposure time b to a / b times when the captured image data A with the exposure time a is used as a reference, and the exposure time Corrects the luminance value of the captured image data of c seconds to a / c times. That is, the normalization unit 31 corrects the luminance values of the pixels 1b to 9b included in the captured image set 2 to a / b times, and the luminance values of the pixels 1c to 9c included in the captured image set 3 are corrected. Correct to a / c times. When captured image data with different imaging conditions is included in one captured image set, the normalization unit 31 also normalizes these data. Note that the timing of normalization is not limited to this example. Normalization may be performed before generation of differential image data described later.

Here, when normalization of the captured image data y _mn is completed, the normalizing unit 31 checks whether or not the number n of captured image sets is 1 (S5). When the imaging set number n is not 1 (YES in S5), that is, when multiple sets of imaging are performed, the normalization unit 31 sends the normalized captured image data y _mn to the image composition unit 32. send. On the other hand, when the imaging set number n is 1 (NO in S5), the normalization unit 31 sends the normalized captured image data y _mn to the difference image generation unit 33.

Image synthesizing unit 32 receives the captured image data y _mn, in each captured image set, the captured image data imaged at the same imaging position synthesized to generate a synthesized image data y _m of m single set, Send a synthetic image data y _m of m sheets thus generated in the difference image generation unit 33. The image synthesizing unit 32 instructs the region extraction unit 35, the extraction area data Z _mn corresponding to m × n pieces of captured image data y _mn, corresponding to m pieces of synthetic image data y _m extraction and updates the region data _{Z m.}

For example, when performing the synthesis of the captured image set 1-3 of Figure 6, the captured image data captured by each imaging position 1-9 is synthesized, the synthetic image data y _m is generated. Synthetic image data y _m is as shown in Figure 7, for example. Figure 7 is a diagram showing an example of the synthetic image data y _m.

  In the figure, in the captured image data captured at the exposure time a, it is assumed that all 25 pixels constituting the captured image data are set in the second region (appropriate exposure). In the captured image data captured with the exposure time b having a longer exposure time, one pixel located in the center among the pixels constituting the captured image data is set to the first region (overexposed). It is assumed that the other 24 pixels are set in the first region. In the captured image data captured with the exposure time c having a longer exposure time, the outermost 16 pixels among the pixels constituting the captured image data are set as the second region, and the 9 pixels in the center are It is assumed that the first area (overexposed) is set.

  In this case, since one pixel located at the center is set in the second region only by the captured image data captured at the exposure time a, the luminance value of this pixel is captured at the exposure time a. The brightness values a1 to a9 of the image data are applied. Further, the eight pixels surrounding one pixel located at the center are set in the second region in both the captured image data captured at the exposure time a and the captured image data captured at the exposure time b, so that the exposure is more exposed. Luminance values b1 to b9 of captured image data captured with a long exposure time b are applied. Since the outermost 16 pixels are set as the second region in all the captured image data, the brightness values c1 to c9 of the captured image data captured at the exposure time c having the longest exposure time are applied. In this way, nine pieces of composite captured image data as shown are generated.

  As described above, the composite captured image data is valid data among the luminance value data of each pixel included in the captured image data captured under a plurality of imaging conditions (for example, exposure time), that is, the brightness value data of the second region. It is configured by combining. As a result, the dynamic range of the composite captured image data is expanded as compared with the captured image data.

  Here, the image processing system 1 performs super-resolution processing using the composite captured image data, but may perform image shift processing using the composite captured image data. When image shift processing is performed using composite captured image data, high-resolution image data with a wide dynamic range can be generated compared to when image shift processing is performed using captured image data. Note that the image shift process is performed by mapping the luminance value of each pixel of the composite captured image data to each pixel of the high resolution image data.

  Here, a mapping method will be described. As described above, the mapping is performed by applying the luminance value of each pixel of the composite captured image data to each pixel of the high resolution image data. As shown in FIG. 7, when nine pieces of 5 × 5 pixel composite captured image data are used, the resolution is increased by a factor of 3, so that the high resolution image data is 5 × 3 pixels × horizontal. The image data has 5 × 3 pixels. Therefore, in the illustrated example, this is performed by applying the luminance value of each pixel of the composite captured image data to the luminance value of each pixel of 15 × 15 pixels.

  Specifically, the luminance value (c1) of the pixel located at the upper left in the upper left composite captured image data of FIG. 7 is applied as the luminance value of the upper left pixel of the high resolution image data. Similarly, as the luminance value of the second pixel from the left in the uppermost column of the high resolution image data, the luminance value (c2) of the pixel located at the uppermost left in the second composite captured image data from the left of the uppermost column in FIG. ) Applies.

  High-resolution image data shown in FIG. 8 is obtained by performing the processing as described above and mapping all the pixels of the nine composite captured image data to the high-resolution image data. FIG. 8 is a diagram showing high-resolution image data generated by subjecting the composite captured image data of FIG. 7 to image shift processing. In FIG. 7 and FIG. 8, the size of each pixel is about the same. Actually, however, each pixel of the high-resolution image data is 1/3 the size of each pixel of the composite captured image data. It has become.

Here, in the example of FIG. 7, it is assumed that all the pixels of all the composite captured image data are in the second region. However, in all the captured image data captured at the same imaging position, the first There may be a case where a pixel set in the region is included. When such a pixel is included, it is difficult to perform highly accurate super-resolution processing. Therefore, the super-resolution processing unit 25 updates the extracted area data Z _mn to the extracted area data Z _m corresponding to the m pieces of synthesized captured image data y _m after performing synthesis, and the extracted area data Z _m. Is used to reduce the contribution of the pixels in the first region in the super-resolution processing.

After the image synthesizing unit 32 has completed the generation of the synthesized image data y _m, or in the case where the imaging set number is 1 (S5 in NO), the pseudo low-resolution image generating unit 34, the high-resolution image storage unit 24 The initial high-resolution image data h stored in is acquired (S7).

  Subsequently, the pseudo low-resolution image generation unit 34 aligns the acquired initial high-resolution image data h with each of the m pieces of captured image data, and considers factors such as optical deterioration factors and downsampling. The resolution is reduced to generate pseudo low-resolution image data (S8).

Specifically, the pseudo low resolution image generation unit 34 multiplies the initial high resolution image data h by B _m to generate pseudo low resolution image data B _m h. Although the initial high resolution image data h is one, the pseudo low resolution image data B _m h is aligned with each of the captured image data, so the number of the pseudo low resolution image data B _m h is m. (See FIG. 25). The pseudo low resolution image generation unit 34 then sends the generated pseudo low resolution image data B _m h to the difference image generation unit 33.

Upon receiving the pseudo low-resolution image data B _m h, the difference image generation unit 33 subtracts the composite captured image data y _m generated by the image synthesis unit 32 from the received pseudo low-resolution image data B _m h to obtain m images. Difference image data _Xm is calculated (S9). Then, the difference image generation unit 33 sends the differential image data X _m of m Like the calculated the weighting difference image generation unit 36. When there is only one set of captured image data, the difference image generation unit 33 calculates m pieces of difference image data X _m by subtracting the captured image data y _m from the pseudo low-resolution image data B _m h. To do.

Weighted difference image generation unit 36 receives the difference image data X _m, a weighting based on the extracted region data Z _m the region extraction unit 35 has generated performed (S10). Specifically, the difference image generation unit 33, by multiplying the .alpha.z _m the differential image data X _m, to generate a weighted difference image data X _m .alpha.z _m obtained by correcting the luminance value of each pixel of the difference image data X _m The generated weighted difference image data X _m αZ _m is sent to the image update unit 37. Note that α is a weight to be multiplied by the extraction region data Z _m , and the value of α can be appropriately set as necessary.

For example, here, in the extraction area data Z _m, the luminance values of the pixels of the first area and 1, since the luminance values of the pixels of the second area is assumed to be 255, the value of alpha 1 / It may be set to 255. As a result, the luminance values of the pixels in the first region in the weighted difference image data X _m αZ _m are all corrected to values of 1 or less, while the luminance values of the pixels in the second region are the same as the difference image data X _m. . Therefore, the contribution of the luminance value of the pixels belonging to the first area can be made lower than that of the pixels in the second area, and thereby the accuracy of the super-resolution processing can be improved.

Image update section 37 that has received the weighted difference image data X _m .alpha.z _m substitutes the weighted difference image data X _m .alpha.z _m received the evaluation function, calculates an evaluation value E. As the evaluation function, for example, Equation (3) may be used similarly to the conventional super-resolution processing shown in [Background Art]. Then, the image update unit 37 compares the calculated evaluation value E with a preset threshold value (S12). Note that, similarly to the conventional super-resolution processing, the threshold value can be appropriately changed according to the required accuracy of resolution enhancement.

  If the evaluation value E is smaller than the threshold value (YES in S12), the image update unit 37 ends the super-resolution process. At this time, the high-resolution image data h stored in the high-resolution image storage unit 24 is the high-resolution image data that is the result of the super-resolution processing.

  On the other hand, when the value of the evaluation value E is equal to or greater than the threshold value (NO in S12), the image update unit 37 uses the high resolution used for calculating the evaluation value E so that the value of the evaluation value E is small. The image data h is updated (S13). The image update unit 37 can update the high-resolution image data h by using the above formula (4), for example, similarly to the conventional super-resolution processing.

Then, the image update unit 37 stores the updated high resolution image data h in the high resolution image storage unit 24. When the high-resolution image data h is newly stored in the high-resolution image storage unit 24, the low-resolution image generation unit artificially lowers the resolution and generates pseudo low-resolution image data B _m h (S8).

  Thus, in the super-resolution processing unit 25, the processes from S8 to S13 are repeated until the value of the evaluation value E becomes smaller than the threshold value. As a result, it is possible to obtain high-resolution image data h having a desired accuracy and having a higher resolution at a desired magnification.

Here, as described above, when the pseudo low-resolution image data B _m h is generated from the high-resolution image data h, a PSF convolution process is performed. By the PSF convolution process, the luminance value of the pixel whose luminance value is corrected to a sufficiently small value by weighting is interpolated and updated by the influence of the luminance value of the pixels located around the pixel. Therefore, the pixel values of the pixels in the first region where overexposure and underexposure occur also approach the values reflecting the shape and color of the imaging target by repeating the processing from S8 to S13.

  In addition, the PSF convolution process is also performed on the difference image data (weighted difference image data in the above example) when iterative calculation is performed. That is, here, it is assumed that the high-resolution image data is updated using Expression (4). In Formula (4), since the partial differentiation of the evaluation function E of Formula (3) is taken, Formula (4) after partial differentiation is expressed as the following Formula (6).

As described above, in Equation (6), since the difference image data (B _m h−y _m ) is multiplied by B _m , the difference image data (B _m h−y _m ) is converted to the PSF by this iterative calculation. Will be subjected to the convolution process. As a result, when the high-resolution image data is updated, each pixel of the difference image data is interpolated and updated due to the influence of the pixels located around it.

  In other words, when weighting is performed as described above, in pixels of high-resolution image data, interpolation update is performed for the pixels in which the inappropriate exposure pixels of the captured image data are reflected according to the luminance values of the surrounding normal exposure pixels. Is done. Therefore, according to the above configuration, it is possible to generate high-resolution image data that accurately reflects the shape and color of the imaging target as compared with the case where weighting is not performed.

  In addition, said flowchart shows an example of the super-resolution process of this invention, and this invention is not limited to this example. For example, when only one set of captured image sets is used, the processes of S5 and S6 in the flowchart of FIG. 5 may be omitted. In addition, when all the captured image data used for the super-resolution processing are captured under the same imaging conditions, the normalization processing in S4 may be omitted.

〔Concrete example〕
Subsequently, the super-resolution processing method of the present invention will be described with specific examples. Here, it is assumed that imaging of an imaging target having a shape and color as shown in FIG. 9 is performed. FIG. 9 is a diagram illustrating an example of an imaging target.

  As illustrated, it is assumed here that the imaging target is an image in which six circular regions having different gradations are arranged in two rows and three columns at equal intervals. Further, as shown in the figure, the gradation of the six circular areas located on the right side of the figure is lower, and the three circular areas arranged on the upper side are the three circular areas arranged on the lower side. The gradation is lower than the area. Of course, the shape and color of the imaging target may be anything, and are not limited to the illustrated example.

  By capturing this imaging target at m different imaging positions from 1 to m, it is possible to acquire a set of m captured images. Further, n sets of captured image sets can be acquired by changing the imaging conditions and imaging at m different imaging positions from 1 to m. For simplicity, an example in which only the exposure time is changed as the imaging condition will be described.

  The captured image data of m sheets × n sets is captured image data as shown in 10, for example. FIG. 10 is a diagram illustrating an example of captured image data. As illustrated, since the m pieces of captured image data included in the same captured image set are captured at different imaging positions, the images of the imaging targets reflected in the captured image data are also different from each other. Further, as illustrated, since the resolution of the imaging device 2 is not sufficient, the captured image data is image data in which an image to be captured is blurred. Note that, in FIG. 10, in order to indicate that the imaging position of each captured image data is shifted, the positional shift is exaggerated, but actually, the imaging position shift is the maximum of the captured image data. The width is one pixel.

  Further, since n sets of images are captured here, n pieces of captured image data captured at the same imaging position are obtained as shown in the figure. Since the n pieces of captured image data captured at the same imaging position have different exposure times, the brightness values of the captured image data also differ according to the exposure time. That is, the brightness value of the captured image data increases as the captured image data is captured under an imaging condition with a longer exposure time.

Here, the areas indicated by D ₁ and D ₃ in FIG. 10 are overexposed, and the areas indicated by D ₂ and D ₄ are blacked out. This occurs due to overexposure or underexposure depending on conditions such as exposure time. In the areas D _{1 to} D ₄ , colors different from those of the imaging target shown in FIG. 9 are displayed. That is, the regions D _{1 to} D ₄ do not accurately reflect the shape and color of the imaging target, and if super-resolution processing is performed using luminance value data of such regions, super-resolution processing is performed. The accuracy of will decrease.

  In the super-resolution processing of the present invention, the captured image data is normalized. By performing normalization, for example, image data as shown in FIG. 11 is obtained. FIG. 11 is a diagram illustrating an example of normalized captured image data. As shown in the figure, by performing normalization, the luminance values of the captured image data of each set are approximately the same. By performing normalization in this way, it can be considered that all captured image data used for the super-resolution processing are captured under the same imaging conditions, and thus obtained by imaging under various imaging conditions. Super-resolution processing can be performed using captured image data.

  Then, after the normalization, the synthesized images of the captured image data at the same imaging position are synthesized to generate synthesized captured image data. As in this example, normalization may be performed on the captured image data, or normalization may be performed on the combined captured image data after the captured image data is synthesized. Good.

  Next, the initial high resolution image data used in this example will be described. As described above, various image data can be used as the initial high-resolution image data. For example, image data as shown in FIGS. 12A to 12D can be used. 12A to 12D are diagrams illustrating an example of initial high-resolution image data.

  That is, as shown in FIG. 5A, a white image with the entire surface of the image data being white can be used as the initial high resolution image data. Similarly, a black image in which the entire image data is black can be used as the initial high resolution image data. Further, as shown in FIG. 5B, a gray image having an intermediate gradation of white and black on the entire surface of the image data may be used as the initial high resolution image data.

  Furthermore, as shown in FIG. 5C, it is also possible to use, as initial high-resolution image data, any image data selected from captured image data that has been subjected to interpolation processing. Here, since it is known in advance that the shape and color of the imaging target are as shown in FIG. 9, design data indicating the shape and color of the imaging target as shown in FIG. It can also be used as image data.

  In the super-resolution processing, the high-resolution image data is sequentially updated so that the difference between the high-resolution image data and the captured image data becomes small. Therefore, there is a difference between the initial high-resolution image data and the captured image data. The smaller the value is, the faster the super-resolution processing is completed. Accordingly, among the initial high resolution image data shown in FIGS. 12A to 12D, those shown in FIGS. 12C and 12D are more than those shown in FIGS. 12A and 12B. The use can reduce the processing time and calculation amount of the super-resolution processing.

  When the image data shown in FIG. 12C or FIG. 12D is used as the initial high resolution image data, the pseudo low resolution image data is, for example, data as shown in FIG. FIG. 13 is a diagram illustrating an example of pseudo low-resolution image data. As shown in the figure, the pseudo low-resolution image data is composed of m image data at the same imaging position as each of the m captured image data, and the resolution (pixel size) of each image data is the same as that of the captured image data. The same is true (see FIG. 25).

  Note that in the case of performing the pseudo low resolution of the initial high resolution image data as shown in FIGS. 12A and 12B, the image data as shown in FIG. 13 is not generated, and the entire surface is uniform and has the resolution. A black image, a white image, a gray image, or the like having the (number of pixels) equal to the captured image data is generated.

  Difference image data is generated from the difference between the pseudo low-resolution image data and the composite captured image data generated as described above, and the difference image data is weighted to generate weighted difference image data. Then, by repeatedly performing calculation using the weighted difference image data, the initial high-resolution image data is sequentially updated, and high-resolution image data with desired accuracy is generated.

[Reference example]
According to the configuration of the image processing system 1, since a plurality of captured image data having different exposure times are normalized and synthesized to perform super-resolution processing, high-resolution image data with an expanded dynamic range is generated. be able to.

  In addition, according to the configuration of the image processing system 1, the contribution of the first region that is likely to cause overexposure or blackout is smaller than the contribution of the second region having a luminance value within the effective dynamic range. Thus, it is possible to generate high-precision high-resolution image data in a short time with a small amount of calculation.

However, since the image processing system 1 uses a plurality of captured image sets having different exposure times, there is a problem that the time required for imaging becomes long. That is, when the resolution is increased N times in the super-resolution processing, it is necessary to perform N ² times of imaging. Therefore, when imaging is performed with a plurality of exposure times, the imaging time is remarkably increased.

  For example, when the resolution is tripled, it is necessary to perform nine imaging operations. In this case, if the exposure time is set to 100 milliseconds, imaging is completed in 900 milliseconds. However, when the exposure time is set to three types of 100 milliseconds, 200 milliseconds, and 300 milliseconds, it takes a minimum of 5400 milliseconds to complete the imaging.

  Note that the expansion of the dynamic range by image processing has been conventionally performed. For example, “analysis of multi-aperture color image” (Naoki Asada, Takashi Matsuyama, Takatoshi Mochizuki, Transactions of Information Processing Society of Japan, October 1991, Vol. 32, No. 10, p.1338-1348) JP-A-174470 (published on July 5, 1990) and JP-A-2007-49228 (published on February 22, 2007) perform composite processing using a plurality of images having different exposure times. It is described to extend the dynamic range.

  According to the method described in the above-mentioned document, an imaging target that is a target for expanding the dynamic range is imaged with two types of exposure time, that is, short exposure and long exposure. Then, for each image captured with two types of exposure time, improper exposure areas such as overexposure pixels and blackout pixels are detected, and these areas are replaced with short exposure images or long exposure images. Yes.

  For example, Japanese Patent Laid-Open No. 02-174470 has a configuration in which a long exposure image and a short exposure image are alternately output. That is, when outputting a short-exposure image, it is determined that a pixel whose luminance value is equal to or less than a predetermined value is blacked out, and the long-exposure image signal stored in the memory is switched. Similarly, when outputting a long-exposure image, it is determined that there is an overexposure with respect to a pixel whose luminance value is a predetermined value or more, and the short-time exposure signal stored in the memory is switched.

  Note that according to the conventional method, the pixels to be replaced need to have the same imaging position. That is, according to the conventional method, it is necessary to use a set of images in which the imaging positions coincide at the pixel level. This is because, when the above-described conventional method is used with image data whose imaging position is shifted, the place where the pixel is replaced becomes discontinuous with the place where the replacement is not performed.

  Therefore, it has not been considered to apply the above-described conventional dynamic range expansion method in super-resolution processing in which imaging is performed while shifting the imaging position. In addition, as described above, in the super-resolution processing, the high-resolution image data dynamic range can be expanded by using composite captured image data obtained by combining a plurality of captured image sets, but imaging takes time. There are also difficulties.

  Therefore, in this reference example, an example in which high-resolution image data with an expanded dynamic range is generated without increasing the time required for imaging will be described with reference to FIGS. More specifically, in this reference example, when imaging an imaging target, imaging is performed while changing the exposure time according to the imaging position, so that the captured image data captured at different exposure times for each imaging position To get.

  Then, high resolution image data with an expanded dynamic range can be generated by performing super-resolution processing or image shift processing using the captured image data captured in this way. That is, the invention described in this reference example is mainly characterized by an imaging method.

  FIG. 14 is a diagram for explaining the imaging method of this reference example. In the figure, an example is shown in which captured image data composed of 25 pixels of 5 × 5 pixels is three times higher in resolution. When the resolution is increased to three times, nine pieces of captured image data are required as in the example of FIG. In this case, as in the example of FIG. 6, the imaging positions of the nine captured image data are shifted by 1/3 pitch of the width of one pixel of the captured image data.

  Here, as can be seen from a comparison with the example of FIG. 6, the captured image data of FIG. 14 has the exposure time a of the uppermost stage and the lowermost stage of FIG. Time is b. That is, according to the imaging method of the present reference example, captured image data captured at a plurality of different exposure times is included in a captured image set for generating one piece of high-resolution image data. For simplicity, an example in which only the exposure time is changed as an imaging condition will be described.

  Such a captured image set can be acquired by the image processing system shown in FIG. That is, the actuator control unit 26 drives the actuator 13 to move the image sensor 12 to a predetermined imaging position. Then, at this imaging position, the imaging timing control unit 27 images the imaging target for the exposure time a. As a result, the captured image data shown on the leftmost side of the uppermost stage in FIG. 14 is captured.

  When the imaging is completed, the actuator control unit 26 drives the actuator 13 to move the imaging device 12 so that the imaging region moves to the right in FIG. 14 by 1/3 of one pixel width of the image image data. When the movement is completed, the imaging timing control unit 27 causes the imaging target to be imaged for the exposure time a as in the above case. As a result, even at an imaging position moved by 1/3 of one pixel width, imaging is performed with the exposure time a, and the captured image data shown in the center of the uppermost stage in FIG. 14 is captured. Similarly, imaging is performed at an exposure time a at an imaging position further moved by 1/3 of one pixel width, and the captured image data shown at the rightmost position in the uppermost stage in FIG. 14 is captured.

  Here, when the imaging of the three pieces of captured image data described in the uppermost part of FIG. 14 is completed, the actuator control unit 26 drives the actuator 13 to have a one-pixel width of the image image data downward in FIG. The image pickup element 12 is moved so that the image pickup area is moved by 1/3. When the movement is completed, the imaging timing control unit 27 images the imaging target for the exposure time b. Thereby, the captured image data shown at the rightmost level in the second row of FIG. 14 is captured. In other words, the uppermost captured image data in FIG. 14 is captured at the exposure time a, but the exposure time is changed to b in the second captured image data in FIG.

  Similarly, imaging is performed with an exposure time b while moving the imaging position at 1/3 pitch of one pixel width, and the captured image data described in the center and the leftmost part of the second row in FIG. 14 is acquired. Then, the actuator control unit 26 drives the actuator 13 to move the imaging element 12 so that the imaging region moves downward by 1/3 of one pixel width of the captured image data in FIG. Here, the imaging timing control unit 27 returns the exposure time to a again to execute imaging. Then, in the same manner as described above, imaging is performed with the exposure time b while moving the imaging position to the right in FIG. 14 at 1/3 pitch of the captured image data, and the three images shown in the bottom row of FIG. The captured image data is acquired.

  Thus, according to the imaging method of the present invention, captured image data captured with a plurality of exposure times (imaging conditions) is included in one captured image set for generating high-resolution image data. become. For example, in the example of FIG. 14, among the nine captured image data included in the captured image set, 6 images are captured at the exposure time a, and 3 images are captured at the exposure time b. .

  By using the captured image data captured at such a plurality of exposure times (imaging conditions), the dynamic range is expanded compared to the case of using only captured image data captured at one type of exposure time. High resolution image data can be generated. Note that the high-resolution image data may be generated by an image shift process or may be generated by a reconstruction type super-resolution process.

  Here, an example in which high-resolution image data with an expanded dynamic range is first generated by image shift processing will be described. As described above, the image shift process is performed by mapping the luminance value of each pixel of the captured image data to each pixel position of the high resolution image data.

  For example, when the image shift process is performed using the nine captured image data shown in FIG. 14, the high-resolution image data shown in FIG. 15 is generated. FIG. 15 is a diagram showing high-resolution image data generated by performing image shift processing using the captured image data of FIG.

  As shown in the figure, when the image shift processing is performed using the nine captured image data of FIG. 14, the captured image data captured at the exposure time a is applied to each pixel in the uppermost row of the high resolution image data. Are mapped. The brightness value of the captured image data captured at the exposure time b is mapped to the second line from the top, and the captured image data captured at the exposure time a is mapped to the third and fourth lines from the top. Luminance values are mapped. Similarly, the luminance value of the captured image data is mapped to each row of the high resolution image data in the order of exposure time b, exposure time a, exposure time a, and exposure time b.

  Here, it is determined whether or not the luminance value of each pixel of the high-resolution image data generated as described above is within a normal range. That is, here, it is determined whether each pixel of the high-resolution image data is the normal exposure second region or the improper exposure first region. Since the determination method is as described in the above embodiment, the description thereof is omitted here.

  Subsequently, an interpolation process is performed on the pixels determined to be inappropriate exposure pixels in the determination process. Interpolation processing is performed on normal exposure pixels in the discrimination processing among the pixels adjacent to the inappropriate exposure pixel (for example, pixels adjacent in four directions of up / down / left / right, or up / down / left / right diagonal directions) of the inappropriate exposure pixel. This is performed by replacing the luminance value of the inappropriately exposed pixel with the average value or intermediate value of the luminance values of the pixels determined to be present.

  By performing the interpolation processing in this way, for example, even when overexposure or underexposure occurs in captured image data captured in either the exposure time b or the exposure time a, the normal exposure pixels Interpolated and updated. That is, the dynamic range of the high-resolution image data is expanded by performing the interpolation process.

  Since the pixels of the high-resolution image data after the interpolation processing include pixels derived from captured image data captured under different imaging conditions (for example, exposure time), normalization of each pixel is performed. Do. The normalization method is as described in the above embodiment. Thereby, high-resolution image data with an expanded dynamic range is generated.

  Next, an example of generating high-resolution image data with an expanded dynamic range by super-resolution processing will be described. According to the super-resolution processing unit 25 of the above-described embodiment, the super-resolution processing of this reference example can be executed. That is, the super-resolution processing of this reference example is executed by performing the processing described in the flowchart of FIG. 5 in the super-resolution processing unit 25 shown in FIG.

  That is, as the captured image data in the flowchart of FIG. 5, by performing super-resolution processing using a set of captured image sets acquired by the above-described imaging method, a dynamic image can be obtained without using a plurality of captured image sets. High-resolution image data with an expanded range can be generated. In this reference example, since only one set of captured image sets is used, the processes of S5 and S6 in the flowchart of FIG. 5 are not performed.

  Here, in the flowchart of FIG. 5, the high-resolution image data updated in S13 is aligned with the captured image data and the resolution is reduced, and pseudo low-resolution image data is generated (S8). At this time, each pixel of the high resolution image data is down-sampled by PSF. For example, when the high-resolution image data in FIG. 15 is reduced in pseudo low resolution, the pixel values of nine pixels included in the vertical 3 × horizontal 3 pixel region are the luminance value of one pixel in the pseudo low-resolution image data. Become.

  As a result, each pixel of the high-resolution image data is interpolated and updated with the luminance values of the pixels located around the pixel when the resolution is reduced. As a result, when pixels whose contribution has been reduced by weighting are included in the pixels of the high-resolution image data, and normally exposed pixels are arranged around the pixels. This pixel is updated with the pixel values of surrounding pixels.

  Therefore, the plurality of pixels of the high-resolution image data included in the region corresponding to one pixel of the pseudo low-resolution image data may include pixels derived from the captured image data captured under each imaging condition evenly. preferable. For example, in the example of FIG. 15, each of nine pixels included in a region of 3 × 3 pixels of high resolution image data (a region corresponding to one pixel of pseudo low resolution image data) has an exposure time a. Six pixels derived from the captured image data and three pixels derived from the captured image data captured at the exposure time b are included.

  Thus, for example, even if the captured image data captured at the exposure time b is overexposed, if the captured image data captured at the exposure time a is captured at normal exposure, the exposure time b The pixel value of the pixel derived from the captured image data is interpolated and updated by the pixel value of the pixel derived from the captured image data captured at the exposure time a when generating the pseudo low resolution image data. As a result, high-resolution image data with an expanded dynamic range is generated.

  In addition, when imaging is performed by the imaging method of FIG. 14, as illustrated in FIG. 15, the pixel is derived from the captured image data captured at the exposure time a and the captured image data captured at the exposure time b. The pixels to be arranged are arranged in stripes. In this case, for example, when an exposure failure occurs in a wide area of the captured image data captured with either the exposure time a or b, as shown in FIG. 15, the horizontal direction of each pixel of the high resolution image data It is also conceivable that sufficient interpolation update is difficult to perform because the exposure times of the pixels adjacent to the pixel are equal.

  In addition, when imaging is performed by the imaging method of FIG. 14, as shown in FIG. 15, the pixels derived from the captured image data captured at the exposure time “a” are captured image data captured at the exposure time “b”. The number is larger than the pixels derived from Therefore, when exposure failure occurs in a wide area of the captured image data captured at the exposure time a, it may be difficult to perform sufficient interpolation update.

  Therefore, the above-described problems can be solved by using captured image data as shown in FIG. FIG. 16 is a diagram illustrating an imaging method different from the above in this reference example. The imaging method of the captured image data shown in FIG. 16 is almost the same as the example of FIG. 14, but the timing for changing the exposure time is different.

  That is, in the example of FIG. 16, imaging is performed so that the exposure times (imaging conditions) of captured image data whose imaging positions are adjacent in the vertical direction or the horizontal direction are different. High-resolution image data shown in FIG. 17 is generated by mapping each pixel value of the captured image data captured in this way to the pixel position of the high-resolution image data.

  Note that in the reconstruction-type super-resolution processing, mapping processing is not performed, but each pixel of the captured image data is reflected at the same position as when mapping processing is performed. Accordingly, when super-resolution processing is performed using the captured image set of FIG. 16, high-resolution image data as shown in FIG. 17 is generated.

  FIG. 17 is a diagram showing high-resolution image data generated from the captured image data of FIG. As shown in the figure, in the example of FIG. 17, the pixels derived from the captured image data captured at the exposure time a and the pixels derived from the captured image data captured at the exposure time b are in the horizontal direction and the vertical direction in FIG. Appear alternately.

  Here, in the example of FIG. 15, the pixels derived from the captured image data captured with different exposure times in the vertical direction and the oblique direction are alternately arranged, but are captured with the same exposure time in the left-right direction. Only the pixels derived from the captured image data are arranged. Therefore, there is a possibility that the interpolation process between adjacent pixels in the left-right direction is not sufficiently performed.

  On the other hand, in the example of FIG. 17, pixels derived from captured image data captured at different exposure times are alternately arranged in any of the vertical direction, the horizontal direction, and the diagonal direction. Therefore, according to this arrangement, the interpolation processing between adjacent pixels is sufficiently performed, and as a result, it is possible to generate high-resolution image data with higher accuracy than in the example of FIG.

  In the example of FIG. 15, each of nine pixels included in a region of 3 × vertical 3 pixels of high resolution image data (a region corresponding to one pixel of pseudo low resolution image data) has an exposure time a. Six pixels derived from the captured image data and three pixels derived from the captured image data captured at the exposure time b were included. That is, in the example of FIG. 15, the ratio of the pixels derived from the captured image data captured at the exposure time a is high, and therefore the influence of the pixels derived from the captured image data captured at a is large.

  On the other hand, in the example of FIG. 17, each of nine pixels included in an area of 3 × 3 pixels of the high resolution image data (an area corresponding to one pixel of the pseudo low resolution image data) has an exposure time. 5 pixels derived from the captured image data captured at a and 4 pixels derived from the captured image data captured at the exposure time b are included.

  That is, in the example of FIG. 17, in the high resolution image data, the difference in the number of pixels derived from the captured image data with different exposure times included in the region corresponding to one pixel of the pseudo low resolution image data is the example of FIG. Is smaller than As a result, even if an exposure failure occurs in a wide area of captured image data captured with one exposure time, sufficient interpolation update is performed, so high-resolution image data with an expanded dynamic range can be accurately obtained. Can be generated.

  By the way, in the example of FIGS. 14-17, when nine captured image data are used, it image | photographs with two types of exposure time of a and b. For this reason, in high-resolution image data, the pixels derived from the captured image data captured at the exposure time a and the pixels derived from the captured image data captured at the exposure time b cannot be made equal, and any of the exposure times The influence of the picked-up image data picked up in (1) becomes larger than the influence of the picked-up image data picked up at the other exposure time.

  Therefore, when nine pieces of captured image data are used, imaging may be performed with three types or nine types of exposure times. Thereby, in the high resolution image data, the number of pixels derived from the captured image data captured at each exposure time can be made equal. FIG. 18 is a diagram for explaining an imaging method when imaging is performed with three types of exposure times. FIG. 18 also shows an example in which the resolution of captured image data composed of 25 pixels of 5 × 5 pixels is increased three times as in FIG. 14 and FIG. 16.

  As shown in the figure, also in the case of imaging with three types of exposure time, the method of moving the imaging position is the same as in the case of FIGS. However, in the example of FIG. 18, the top three captured image data, the second three captured image data, and the bottom three captured image data are captured with different exposure times. ing. That is, in the example of FIG. 18, the uppermost three captured image data are captured at the exposure time a, and the second three captured image data are captured at the exposure time b, and the lowermost three images are captured. The captured image data is captured at the exposure time c.

  Then, by using the captured image data in the example of FIG. 18, the high-resolution image data shown in FIG. 19 can be obtained. FIG. 19 is a diagram showing high-resolution image data generated from the captured image data of FIG.

  As shown in the figure, each of nine pixels included in a 3 × 3 pixel area of high resolution image data (an area corresponding to one pixel of the pseudo low resolution image data) is imaged at exposure times a to c. Three pixels derived from the captured image data are included. Thereby, the contribution of the pixel originating in the captured image data imaged by exposure time ac in the high resolution image data can be made equal. As a result, for example, even when an exposure failure occurs in a wide area of captured image data captured at any exposure time, sufficient interpolation update is performed, so high-resolution image data with an expanded dynamic range is obtained. It can be generated with high accuracy.

  Here, as illustrated, in the example of FIG. 19, pixels derived from captured image data captured at each exposure time are alternately arranged in the vertical direction and the diagonal direction, but the same in the horizontal direction. Only the pixels derived from the captured image data captured at the exposure time are arranged. Accordingly, in the example of FIG. 19, as in the example of FIG. 15, there is a possibility that the interpolation process between adjacent pixels in the left-right direction is not sufficiently performed.

  Therefore, for example, imaging may be performed by the imaging method shown in FIG. FIG. 20 is a diagram for explaining an imaging method further different from the above. Note that FIG. 20 also shows an example in which imaging is performed with three types of exposure times and nine captured image data are acquired, as in FIG. Each captured image data is composed of 25 pixels of 5 × 5 pixels, and it is assumed that the resolution of the captured image data is increased three times.

  As shown in the figure, also in the example of FIG. 20, the moving method of the imaging position is the same as in the case of FIG. 14, FIG. 16, and FIG. However, in the example of FIG. 20, three pieces of captured image data in each stage from the uppermost stage to the lowermost stage are imaged with different exposure times. In the example of FIG. 20, the three captured image data in each column from the leftmost column to the rightmost column have different exposure times. That is, in the example of FIG. 20, the captured image data of the uppermost leftmost column, the second second column, and the third rightmost column are captured at the exposure time a, and the uppermost second column, The captured image data in the second rightmost column and the third leftmost column are captured at the exposure time b, and the uppermost rightmost column, the second leftmost column, and the third second row The captured image data of the eye is captured at the exposure time c.

  And the high resolution image data shown in FIG. 21 can be obtained by using the captured image data of the example of FIG. FIG. 21 is a diagram showing high-resolution image data generated from the captured image data of FIG.

  As shown in the figure, each of nine pixels included in a 3 × 3 pixel area of high resolution image data (an area corresponding to one pixel of the pseudo low resolution image data) is imaged at exposure times a to c. Three pixels derived from the captured image data are included. Thereby, the contribution of the pixel originating in the captured image data imaged by exposure time ac in the high resolution image data can be made equal. As a result, for example, even when an exposure failure occurs in a wide area of captured image data captured at any exposure time, sufficient interpolation update is performed, so high-resolution image data with an expanded dynamic range is obtained. It can be generated with high accuracy.

  Further, as illustrated, in the example of FIG. 21, pixels derived from captured image data captured at each exposure time are alternately arranged in the vertical direction and the horizontal direction. Therefore, in the example of FIG. 21, the interpolation processing is sufficiently performed in the vertical direction and the horizontal direction. As a result, in the example of FIG. 21, high-resolution image data with an expanded dynamic range can be generated with high accuracy.

  In the example of FIG. 21, only pixels derived from captured image data captured with the same exposure time are arranged in the oblique direction, but the arrangement of the example of FIG. 21 has the same exposure time in the left-right direction. This is more preferable than the example of FIG. 19 in which only pixels derived from captured image data are arranged. This is because the PSF used in the super-resolution processing is a point spread function having a Gaussian distribution, and therefore it is preferable that there are as many normal values as possible in the vicinity of the pixel of interest during the convolution processing. Also, in the interpolation processing when performing image shift, there are cases where four pixels located above, below, left, and right of the target pixel are used, so that the captured image data captured with the same exposure time on the top, bottom, left, and right of the target pixel as described above. It is preferable that the derived pixels are not arranged.

  As described above, according to the imaging method of the present reference example, high-resolution image data with an extended dynamic range can be generated even when a set of captured image data is used. Of course, the above-mentioned method shows an example of the imaging method of this reference example, and the imaging method of this reference example is not limited to this example.

  That is, in the above-described example, the example of increasing the resolution of the captured image data composed of 25 pixels of 5 × 5 pixels by 3 times is shown. However, the number of pixels of the captured image data and the magnification for increasing the resolution are necessary. It can be changed as appropriate according to the situation. In the above-described example, only the exposure time is changed as the imaging condition. However, at least one of the imaging conditions such as the imaging device to be used, the illumination intensity, the imaging sensitivity, the filter to be used, and the iris is changed. It may be.

  As described above, in the imaging method of the present reference example, the dynamic range is extended if a set of captured image sets includes captured image data having imaging conditions different from at least one other captured image data. be able to.

  Then, in the high-resolution image data, by adjusting the arrangement of the pixels derived from the captured image data captured under different imaging conditions included in the region corresponding to one pixel of the captured image data, it is possible to achieve higher accuracy. It becomes possible to generate resolution image data.

  That is, in the imaging method of the present reference example, in the high-resolution image data, in a plurality of pixels included in a region corresponding to one pixel of the captured image data, pixels derived from captured image data captured under different imaging conditions are used. It is preferable to control the imaging conditions of each captured image data so that the difference in the number of pixels is minimized.

  For example, in the example of FIG. 15, in the high-resolution image data, in the region corresponding to one pixel of the captured image data, 6 pixels derived from the captured image data captured at the exposure time a are captured at the exposure time b. Three pixels derived from the captured image data are included. On the other hand, in the example of FIG. 17, in the high-resolution image data, the area corresponding to one pixel of the captured image data includes five pixels derived from the captured image data captured at the exposure time a, and the exposure time. Four pixels derived from the captured image data imaged in b are included.

  That is, in the example of FIG. 17, the difference in the number of pixels derived from captured image data captured under different imaging conditions is minimized. Therefore, the imaging method of FIG. 16 in which the high-resolution image data of the example of FIG. 17 is generated is preferable to the imaging method of FIG. 14 in which the high-resolution image data of the example of FIG. 15 is generated.

  In the imaging method of the present reference example, in the high-resolution image data, each of the above-described regions in pixels derived from the captured image data captured under the same imaging condition included in the region corresponding to one pixel of the captured image data. It is preferable to control the imaging conditions so that the number of pixels continuously arranged in the vertical direction or the horizontal direction is minimized.

  As described above, when the captured image data generated by the imaging method of the present reference example is used for (reconstruction type) super-resolution processing, when generating the pseudo low-resolution image data, Pixels derived from different captured image data included in an area corresponding to one pixel are updated by interpolation. Therefore, it is preferable that the pixels derived from the captured image data captured under different imaging conditions included in the high resolution image data are uniformly arranged on the entire surface of the high resolution image data.

  In other words, the pixels derived from the captured image data captured under different imaging conditions included in the high-resolution image data are uniformly distributed in the horizontal direction, the vertical direction, and the diagonal direction in the high-resolution image data. Moreover, it is preferable that the difference in the distribution density of pixels derived from captured image data captured under different imaging conditions is small.

  For example, in the example of FIG. 15, in the high-resolution image data, the pixels derived from the captured image data captured at the exposure time “a” and the pixels derived from the captured image data captured at the exposure time “b” are in the horizontal direction of FIG. Fifteen in a row. On the other hand, in the example of FIG. 17, the pixels derived from the captured image data captured at the exposure time a and the pixels derived from the captured image data captured at the exposure time b are continuous in the horizontal direction of FIG. The maximum number of elements arranged is 2.

  That is, in the example of FIG. 17, the number of pixels arranged continuously in the horizontal direction is minimized. Therefore, the imaging method of FIG. 16 in which the high-resolution image data of the example of FIG. 17 is generated is preferable to the imaging method of FIG. 14 in which the high-resolution image data of the example of FIG. 15 is generated.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope shown in the claims. That is, embodiments obtained by combining technical means appropriately modified within the scope of the claims are also included in the technical scope of the present invention.

  Finally, each block of the control device 3, in particular, the imaging control unit 21 and the super-resolution processing unit 25 may be configured by hardware logic, or may be realized by software using a CPU as follows. .

  That is, the control device 3 includes a CPU (central processing unit) that executes instructions of a control program for realizing each function, a ROM (read only memory) that stores the program, a RAM (random access memory) that expands the program, A storage device (recording medium) such as a memory for storing the program and various data is provided. An object of the present invention is to provide a recording medium in which a program code (execution format program, intermediate code program, source program) of a control program of the control device 3 that is software that realizes the above-described functions is recorded so as to be readable by a computer. This can also be achieved by supplying the control device 3 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

  Examples of the recording medium include a tape system such as a magnetic tape and a cassette tape, a magnetic disk such as a floppy (registered trademark) disk / hard disk, and an optical disk such as a CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

  Further, the control device 3 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Further, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

As described above, for a plurality of captured image sets with different image capturing conditions, the luminance value is corrected in accordance with the image capturing conditions, and these are combined to further increase the dynamic range compared to the captured image data. Image data can be obtained. Then, by generating high resolution image data using the composite captured image data, high resolution image data with an expanded dynamic range is generated.

  However, when a plurality of captured image sets having different imaging conditions are used, the number of captured image data to be captured increases, which causes a problem that the time required for imaging becomes long.

  Therefore, in order to solve the above problem, the imaging control device of the invention according to the reference example has a high resolution in which the imaging target is increased based on the number of low resolution captured image data corresponding to the resolution increasing magnification. An image pickup control apparatus that controls an image pickup apparatus that picks up the image pickup target to generate image data, and each of regions corresponding to one pixel of the picked-up image data in the high-resolution image data It is characterized by comprising imaging condition control means for controlling the imaging conditions of the respective captured image data so as to include pixels of high resolution image data derived from captured image data captured under different imaging conditions.

  In addition, in order to solve the above-described problem, the control method of the imaging control apparatus according to the reference example described above increases the resolution of the imaging target based on the number of low-resolution captured image data corresponding to the resolution enhancement magnification. In order to generate the high-resolution image data, the control method of the imaging control device that controls the imaging device that images the imaging target, and corresponds to one pixel of the captured image data in the high-resolution image data Including an imaging condition control step for controlling the imaging conditions of each of the captured image data so that each region includes pixels of high-resolution image data derived from captured image data captured under different imaging conditions. It is a feature.

  According to the above configuration, in the high-resolution image data, each region corresponding to one pixel of the captured image data includes pixels of the high-resolution image data derived from the captured image data captured under different imaging conditions. As described above, the imaging condition of each captured image data is controlled. The imaging conditions are at least one of exposure time (shutter speed), imaging device to be used, illumination intensity, imaging sensitivity, filter to be used, and iris.

  Therefore, when a high-resolution image is generated using the low-resolution captured image data corresponding to the high-resolution magnification imaged with the above configuration, the region corresponding to one pixel of the captured image data High-resolution image data including pixels of high-resolution image data derived from captured image data captured under different imaging conditions is generated.

  Since this high resolution image data reflects captured image data captured under different imaging conditions, compared to the case where high resolution image data is generated using captured image data captured under the same imaging conditions, The dynamic range has been expanded.

  Therefore, according to the above configuration, it is possible to generate high-resolution image data with an extended dynamic range without increasing captured image data necessary to generate high-resolution image data with a desired high-resolution magnification. .

  Further, the imaging condition control means is a pixel derived from captured image data captured under the same imaging condition included in each area, and is continuously arranged in the vertical direction or the horizontal direction in each area. It is preferable to control the imaging conditions of each of the captured image data so that the number of images is minimized.

  Here, when an improper exposure pixel occurs in a wide area of captured image data captured under a certain imaging condition, pixels derived from the captured image data captured under the imaging condition are vertically aligned in each area. If the pixels are continuously arranged in the horizontal direction or the horizontal direction, improperly exposed pixels tend to be noticeable in the high-resolution image data.

  For example, when pixels derived from captured image data generated in a region where overexposure is caused by wide exposure are arranged continuously in the vertical direction or the horizontal direction in each region, in the high-resolution image data, White stripes are visible in the horizontal direction.

  Therefore, according to the above configuration, the number of pixels in which pixels derived from captured image data captured under the same imaging conditions are continuously arranged in the vertical direction or the horizontal direction in each region is minimized. I have to. As a result, even when inappropriate exposure pixels are generated in a wide area of captured image data captured under specific imaging conditions, pixels derived from inappropriate exposure pixels are less noticeable in high-resolution image data. .

  In addition, when super-resolution processing is performed using captured image data captured based on the control of the above-described imaging control apparatus, the high-resolution image data is included in an area corresponding to one pixel of the captured image data. A plurality of pixels to be interpolated are updated. Thereby, the luminance value of the pixel derived from the inappropriate exposure pixel is updated to an appropriate value by the luminance value of the normal exposure pixel adjacent to the pixel, so that high-precision high-resolution image data can be generated. .

  Further, the imaging condition control means is configured to capture the imaging conditions of the captured image data so that a difference in the number of pixels derived from captured image data captured under different imaging conditions included in the regions is minimized. Is preferably controlled.

  Here, in the case where improper exposure pixels occur in a wide area of captured image data captured under a certain imaging condition, each pixel area derived from the captured image data captured under the above imaging condition When the proportion is high, improperly exposed pixels tend to be noticeable in high resolution image data.

  Therefore, according to the above configuration, the difference in the number of pixels derived from captured image data captured under different imaging conditions is minimized. As a result, even when inappropriate exposure pixels are generated in a wide area of captured image data captured under specific imaging conditions, pixels derived from inappropriate exposure pixels are less noticeable in high-resolution image data. .

  In addition, when super-resolution processing is performed using captured image data captured based on the control of the above-described imaging control apparatus, the high-resolution image data is included in an area corresponding to one pixel of the captured image data. A plurality of pixels to be interpolated are updated. Thereby, the luminance value of the pixel derived from the inappropriate exposure pixel is updated to an appropriate value by the luminance value of the normal exposure pixel adjacent to the pixel, so that high-precision high-resolution image data can be generated. .

  In addition, each imaging is performed so that all of a set of captured image data captured by the control of the imaging control device can be regarded as being captured under one imaging condition selected from the different imaging conditions. Captured image correction means for correcting the luminance value of each pixel of image data, and super-resolution processing means for performing super-resolution processing using a set of captured image data corrected by the captured image correction means. The image processing apparatus can generate high-resolution image data with an extended dynamic range without increasing the number of captured image data necessary to generate high-resolution image data with a desired high-resolution magnification. .

  This is because when performing super-resolution processing, pixels of high-resolution image data derived from captured image data captured under different imaging conditions are simulated in each of the regions corresponding to one pixel of captured image data. This is because it is integrated into one pixel when generating low-resolution image data.

  That is, the luminance values of the pixels of the high-resolution image data derived from the captured image data captured under different imaging conditions are integrated as one pixel of the pseudo-decrease image data, and the high-resolution image data is updated. By being used, the updated high-resolution image data reflects the brightness value of the captured image data captured under different imaging conditions. As a result, the dynamic range of the high resolution image data is expanded.

  A mapping unit configured to map each pixel of the set of captured image data captured by the control of the imaging control device to each pixel position of the high-resolution image data and generate the high-resolution image data; and the mapping unit For each pixel of the high-resolution image data generated by the above, the appropriate exposure pixel determination means for determining whether or not the luminance value is within a predetermined normal range, and the appropriate exposure pixel determination means in the high-resolution image data Interpolation means for interpolating the luminance value of the pixel determined not within the normal range based on the luminance value of the pixel adjacent to the pixel determined by the appropriate exposure pixel determination means to be within the normal range; , All the pixels of the high-resolution image data generated by the mapping means are considered to have been imaged under one imaging condition selected from the different imaging conditions. A high-resolution image having a desired high-resolution magnification, so long as the image processing apparatus includes a captured image correction unit that corrects the luminance value of each pixel of the high-resolution image data. High-resolution image data with an extended dynamic range can be generated without increasing the captured image data necessary for generating the data.

  According to the present invention, high-resolution image data with an expanded dynamic range can be generated, and therefore, the present invention can be suitably applied to an apparatus for increasing the resolution of a still image or a moving image.

1, showing an embodiment of the present invention, is a block diagram illustrating a main configuration of a super-resolution processing unit. FIG. 1, showing an embodiment of the present invention, is a block diagram illustrating a main configuration of an image processing system. FIG. It is a figure explaining the positional relationship of an image sensor and the pixel of high resolution image data. FIG. 4 is a diagram illustrating an example of a method for synthesizing captured image data according to an exemplary embodiment of the present invention. It is a flowchart which shows an example of the super-resolution process performed in the said image processing system. It is a figure which shows an example of the captured image set used in the said image processing system. It is a figure which shows an example of the composite picked-up image data produced | generated by combining the picked-up image data contained in the said picked-up image set. It is a figure which shows the high resolution image data produced | generated by using the said composite picked-up image data for an image shift process. It is a figure which shows an example of the imaging target used with the said image processing system. It is a figure which shows an example of the captured image data obtained by imaging the said imaging target. It is a figure which shows an example of the state which normalized the said captured image data. FIGS. 4A to 4D are diagrams showing an example of initial high-resolution image data used in the image processing system. It is a figure which shows an example of the pseudo | simulation low resolution image data produced | generated by the said image processing system. It is a figure explaining the imaging method in the reference example of this invention. It is a figure which shows the high resolution image data produced | generated by performing an image shift process using the captured image data imaged with the said imaging method. It is a figure which shows the imaging method different from the above in the reference example of this invention. It is a figure which shows the high resolution image data produced | generated from the captured image data imaged with the said imaging method. In the reference example of this invention, it is a figure explaining the imaging method in the case of imaging with three types of exposure time. It is a figure which shows the high resolution image data produced | generated from the captured image data imaged with the said imaging method. It is a figure explaining the imaging method further different from the above in the reference example of this invention. It is a figure which shows the high resolution image data produced | generated from the captured image data imaged with the said imaging method. It is a flowchart which shows an example of the conventional super-resolution processing method. It is a figure which shows an example of the imaging method of the captured image data used for a super-resolution process. It is a figure which shows an example of the initial high resolution image data used for a super-resolution process. It is a figure which shows an example of the pseudo | simulation low resolution image data used for a super-resolution process.

DESCRIPTION OF SYMBOLS 1 Image processing system 2 Imaging apparatus 3 Control apparatus 21 Imaging control part 25 Super-resolution processing part 31 Normalization part 32 Image composition part 33 Difference image generation part (difference image generation means)
34 pseudo low-resolution image generation unit 35 region extraction unit (appropriate exposure pixel determination means)
36 Weighted difference image generating unit (weighted difference image generating means)
37 Image update unit (image update means)

Claims (7)

An image processing apparatus that generates high-resolution image data obtained by increasing the resolution of an imaging target based on a number of low-resolution captured image data corresponding to the resolution-enhancing magnification,
Points obtained from the camera model of the imaging device that images the captured image data by aligning the high-resolution image data with one of the captured image data, converting it to the same resolution as the captured image data Multiplying the spread function to generate pseudo low-resolution image data (processing order), and performing a process of taking the difference between the pseudo low-resolution image data and the captured image data for each of the plurality of captured image data Differential image generation means for generating a plurality of difference image data;
Based on the plurality of difference image data generated by the difference image generation means, an evaluation value indicating an error between the high resolution image data and the plurality of captured image data is calculated, and the evaluation value is a predetermined threshold value. In the case of the above, the image update means for updating the high-resolution image data so that the evaluation value becomes small,
For each pixel of the plurality of captured image data, appropriate exposure pixel determination means for determining whether or not the luminance value is within a predetermined normal range;
In one difference image data corresponding to one of the plurality of captured image data, the appropriate exposure pixel determining unit exists at a position corresponding to a pixel determined to be not within the normal range in one of the plurality of captured image data. Weighting difference image generation means for performing weighting on the target pixel so as to reduce the contribution of the target pixel to the evaluation value to each of the plurality of difference image data,
The image processing apparatus, wherein the image update means updates the high-resolution image data using the plurality of difference image data weighted by the weighted difference image generation means. The plurality of captured image data includes captured image data captured under different imaging conditions,
After the appropriate exposure pixel determining means determines whether or not the luminance value is within a predetermined normal range, the one set of captured image data of the set of captured image data is selected under one imaging condition selected from the different imaging conditions. The image processing apparatus according to claim 1, further comprising a captured image correction unit that corrects the luminance value of each pixel of each captured image data so that it can be regarded that all of the captured images have been captured. An image processing apparatus that generates high-resolution image data obtained by increasing the resolution of an imaging target based on a number of low-resolution captured image data corresponding to the resolution-enhancing magnification,
The high-resolution image data is generated on the basis of a plurality of sets of picked-up image data including a set of picked-up image data of the number corresponding to the high-resolution magnification, and the image pick-up included in each picked-up image set The image data is captured under the same imaging conditions, and the captured image data included in different captured image sets is captured under different imaging conditions.
Appropriate exposure pixel determining means for determining whether or not the luminance value is within a predetermined normal range for each pixel of each captured image data included in the plurality of captured image sets;
After the appropriate exposure pixel determining means determines whether or not the luminance value is within a predetermined normal range, the plurality of sets of captured image sets are set under one imaging condition selected from the different imaging conditions. Captured image correction means for correcting the luminance value of each pixel of each captured image data so that all of the included captured image data can be regarded as captured;
In the pixels of each captured image data included in one set selected from the plurality of captured image sets, the luminance value of the replacement target pixel determined by the appropriate exposure pixel determining unit not being within the normal range is set to another image. In the image set, when there is a replacement candidate pixel that is located at a position corresponding to the replacement target pixel and is determined to be within the normal range by the appropriate exposure pixel determination unit, the luminance value of the replacement target pixel is set. Image synthesizing means for generating a set of combined captured image data by replacing with the luminance value of the replacement candidate pixel;
The high-resolution image data is aligned with one of the composite captured image data included in the set of composite captured image data, converted to the same resolution as the composite captured image data, and the plurality of captured images The process of taking the difference between the pseudo low-resolution image data generated by multiplying the point spread function obtained from the camera model of the imaging device that captured the data and the composite captured image data, and generating the difference image data Differential image generation means for generating a set of differential image data by performing each of the composite captured image data included in the set of composite captured image data;
Based on the set of difference image data generated by the difference image generation means, an evaluation value indicating an error between the high-resolution image data and the set of combined captured image data is calculated. An image updating means for updating the high-resolution image data so that the evaluation value becomes smaller when the threshold value is equal to or greater than a predetermined threshold;
The appropriate exposure pixel determination means further determines whether or not the luminance value is within a predetermined normal range for each pixel of the composite captured image data included in the set of composite captured image data,
With respect to the evaluation value of the pixel of the difference image data existing at the position corresponding to the pixel of the composite captured image data determined by the appropriate exposure pixel determination means not being within the normal range in the set of composite captured image data. Weighting difference image generation means for weighting each pixel of the set of difference image data generated by the difference image generation means so as to reduce the contribution,
The image processing apparatus, wherein the image update means updates the high-resolution image data using the plurality of difference image data weighted by the weighted difference image generation means.   An image processing system comprising the image processing device according to claim 1 and the imaging device. A method for controlling an image processing apparatus that generates high-resolution image data obtained by increasing the resolution of an imaging target based on a number of low-resolution captured image data corresponding to the resolution-enhancing magnification,
Points obtained from the camera model of the imaging device that images the captured image data by aligning the high-resolution image data with one of the captured image data, converting it to the same resolution as the captured image data Multiplying the spread function to generate pseudo low-resolution image data, and performing a process of taking the difference between the pseudo low-resolution image data and the captured image data for each of the plurality of captured image data, A difference image generation step for generating image data;
Based on the plurality of difference image data generated in the difference image generation step, an evaluation value indicating an error between the high-resolution image data and the plurality of captured image data is calculated, and the evaluation value is predetermined. An image update step of updating the high-resolution image data so that the evaluation value becomes smaller when the value is equal to or greater than a threshold value,
For each pixel of the plurality of captured image data, a proper exposure pixel determination step for determining whether or not the luminance value is within a predetermined normal range;
In one difference image data corresponding to one of the plurality of captured image data, in a position corresponding to a pixel determined to be not within a normal range in one of the plurality of captured image data in the appropriate exposure pixel determination step. A weighted difference image generation step for weighting each of the plurality of difference image data, so as to reduce the contribution of the existing target pixel to the evaluation value,
In the image update step, the high-resolution image data is updated using the plurality of difference image data weighted in the weighted difference image generation step. An image processing apparatus control program for operating the image processing apparatus according to claim 1,
An image processing apparatus control program for causing a computer to function as each of the above means.   A computer-readable recording medium on which the image processing apparatus control program according to claim 6 is recorded.