JP2009181508A - Image processing device, inspection system, image processing method, image processing program, computer-readable recording medium recording the program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform super-resolution processing while reducing the effect of a noise component contained in pick-up image data. <P>SOLUTION: The super-resolution processing part 12 includes an image update part 12c which repeats update processing for updating high-resolution image data so as to minimize the value of an evaluation function showing an error between the high-resolution image data higher in resolution than pick-up image data, which is generated based on a plurality of pick-up image data, and noise removed image data obtained by removing the noise component from each of the plurality of pickup image data until the error becomes a predetermined threshold or less. Since the image update part 12c generates the noise removed image data by subtracting from each of the pick-up image data the noise component contained therein, and updates each noise image data so as to minimize the error, the effect of the noise component in super-resolution processing can be reduced. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that generates a high-resolution image from a plurality of low-resolution images, and more particularly to a reconstruction type super-resolution process.

  In imaging devices for capturing high-definition images (moving images or still images), automatic inspection devices that automatically inspect defects in inspection objects, and so on, the details of the imaging object (or inspection object) are clear. Need to image. In order to perform clear imaging in detail, a high-resolution imaging device that can capture a high-resolution image is required.

  However, it is sufficient to use the latest area sensor when outputting an image obtained by imaging an imaging target to a high-definition display device, or when performing a more detailed inspection on the details of the inspection object. It is becoming impossible to obtain the resolution.

  Therefore, a technique for acquiring an image with a necessary resolution by performing a high resolution process for generating a single high resolution image from a plurality of low resolution images by super-resolution processing, image shift processing, or the like, A technique for acquiring an image with a necessary resolution by increasing the number and reducing the imaging area per imaging apparatus is used.

  Compared with the latter method using a plurality of imaging devices, the former high resolution method has the following disadvantages: 1) It takes a lot of imaging time, and 2) Image processing for generating a high resolution image is necessary. However, there are merits such as 1) the cost of the imaging device, for example, the camera is reduced, and 2) good maintainability.

  The present invention relates to the resolution enhancement technique. The high resolution technique is a technique for generating one high resolution image from a plurality of low resolution images having a positional shift, and is roughly classified into two types, image shift processing and super-resolution processing.

  The image shift process is a technique for mapping a pixel corresponding to a pixel of a captured low resolution image to a high resolution image based on the luminance value of the pixel of the low resolution image in the generated high resolution image.

  On the other hand, the super-resolution processing is processing for estimating one high-resolution image from a plurality of low-resolution images. For example, the ML (Maximum-likelihood) method, the MAP (Maximum A Posterior) method, the POCS (Projection On to Various super-resolution processing methods such as the Convex Set method have been proposed.

  The ML method uses a square error between a pixel value of a low-resolution image estimated from a high-resolution image and an actually observed pixel value as an evaluation function for increasing the resolution, and a high level that minimizes this evaluation function. This is a method of using a resolution image as an estimated image. That is, the ML method is a super-resolution processing method based on the principle of maximum likelihood estimation.

  The MAP method is a method for estimating a high-resolution image that minimizes an evaluation function obtained by adding probability information of a high-resolution image to a square error. That is, the ML method is a super-resolution processing method that estimates a high-resolution image as an optimization problem that maximizes the posterior probability by using some foresight information for the high-resolution image.

  The POCS method is a super-resolution processing method for obtaining a high-resolution image by creating simultaneous equations for pixel values of a high-resolution image and a low-resolution image and solving the equations sequentially.

  These super-resolution processes first assume a high-resolution image, and based on the point spread function (PSF function) obtained from the assumed high-resolution image from the camera model of the imaging apparatus, The pixel value of the pixel of the resolution image is estimated, and a process of searching for a high-resolution image that reduces the difference between the estimated value and the observed pixel value (observed value) is included. Therefore, these super-resolution processes are called reconfigurable super-resolution processing methods.

  In the image shift process, it is necessary to map all the pixels of the low-resolution image to the pixels of the high-resolution image. On the other hand, in the super-resolution processing, the imaging position of the low resolution image only needs to be known, and the number of low resolution images to be used is not limited. Note that the pixel value estimation process performed in the super-resolution process is an iterative operation, and thus the super-resolution process generally requires higher calculation cost than the image shift process.

  Here, the principle of conventional super-resolution processing will be described. As described above, the super-resolution processing is formulated as an optimization problem of the evaluation function defined for the generated high-resolution image. That is, the reconstruction type super-resolution processing is reduced to an optimization problem of an evaluation function (error term) based on a square error between the “estimated low-resolution image” and the “observed observed image”.

  Note that since the super-resolution processing is an optimization calculation with a very large number of unknowns, for example, an iterative calculation method such as the steepest descent method is often used to optimize the evaluation function. At this time, it is necessary to calculate the evaluation function and the derivative of the evaluation function with respect to the high-resolution image for each iteration.

  When the characteristics of the high resolution image data are known, an evaluation function (constraint term) based on the characteristics of the high resolution image data may be added to the evaluation function (error term). For example, when it is known in advance that the edge component of the high resolution image data is small, an evaluation function (constraint term) based on image data obtained by multiplying the high resolution image data by a Laplacian filter may be added. As a result, the fact that the edge component of the high-resolution image data is small can be reflected in the super-resolution processing, so that the accuracy of increasing the resolution can be improved.

  Details of the conventional super-resolution processing method will be described below. Here, it is assumed that one piece of high-resolution image data is generated from n pieces of captured image data. In this case, the evaluation function used for the super-resolution processing is expressed by, for example, the following mathematical formula (1), and the super-resolution processing is expressed as processing for minimizing the evaluation function.

The above mathematical formula (1) is an evaluation function used in a super-resolution processing method called a MAP method. Equation (1) is an equation comprised of a constraint term and the error term as described above, the error term is included characters B _n, h, and y _n, the restraint section w, L, and h Is included. In the MAP method, high-resolution image data is generated by performing an operation so that Equation (1) is minimized.

h is obtained by expressing the high resolution image data generated from the n pieces of captured image data in vector, y _n is a representation by a vector of all n pieces of captured image data. y _n is expressed by the following mathematical formula (2), and in the mathematical formula (2), x _i (i = 1 to n) represents each captured image data from the first image to the n-th image as a vector. .

  Note that the vector representation of image data refers to a representation of image data components as a single row vector arranged in a line. For example, when the size of the elements constituting the image data is W × H, the image data is expressed as a vector of 1 row × (W × H) columns.

B _n is a point spread function (PSF function) obtained from the camera model of the imaging apparatus used to acquire the captured image data after the high-resolution image data is aligned with the imaging position of each captured image data. The process of convolution of the kernel to reduce the resolution is expressed as a matrix, and is expressed by the following formula (3).

In Equation (3), M _i (i = 1 to n) is the luminance of the pixel corresponding to the position of each image sensor of the captured image data after the high-resolution image data is aligned with the imaging position of the captured image data. A matrix for sampling values. Therefore, by multiplying the M _i, the high resolution image data, while being aligned with the imaging position of the captured image data, the resolution (number of elements) is the same as the captured image data. In Equation (1), C is a matrix representing a process of convolving the PSF of the camera model to blur the high resolution image data. By multiplying C, the influence of optical blur is added to the high-resolution image data.

Therefore, by convolving B _n with the high resolution image data h, the high resolution image data is aligned with the captured image data, the resolution is reduced, and the image data B _n h reflecting the influence of optical blur and Become. Here, this image data B _n h is referred to as pseudo low-resolution image data. The difference between the pseudo low-resolution image data B _n h and the captured image data y _n is an error term in Equation (1).

  On the other hand, the constraint term in the above formula (1) is represented by a matrix L representing prior probability information of the high resolution image data, high resolution image data h, and w which is a parameter representing the strength of the constraint term. By incorporating such a constraint term into the high resolution evaluation function, it is possible to reflect the prior probability information of the high resolution image data and perform a more accurate super-resolution process.

  A conventional super-resolution processing method using such an evaluation function (formula (1)) will be described with reference to FIG. FIG. 9 is a diagram for explaining a conventional super-resolution processing method. The super-resolution processing is performed by repeatedly calculating until the difference between the n pieces of captured image data and the high-resolution image data estimated from the n pieces of captured image data is smaller than a predetermined value.

  Therefore, the super-resolution processing can be expressed as processing for generating high-resolution image data having a small difference from n pieces of captured image data. The smaller the difference from n pieces of captured image data, the more n pieces of image data. It can be said that it is high-resolution image data with high accuracy that reflects the captured image data more accurately.

  As shown in the figure, in the super-resolution processing, first, difference image data is generated by subtracting input captured image data from pseudo-low resolution image data obtained by optical degradation simulation of high resolution image data. Is done. The optical degradation simulation is a process performed to calculate the difference between the high resolution image data and the captured image data. The high resolution image data is aligned with the captured image data and the resolution of the high resolution image data is determined. To generate a plurality of pseudo low-resolution image data by multiplying the point spread function obtained from the camera model of the imaging device used to acquire the captured image data (pseudo-resolution reduction) Process).

That is, the pseudo low-resolution image data in FIG. 9 corresponds to B _{n h} in Equation (1), the captured image data corresponds to y _n. The difference image data corresponds to (B _n h−y _n ). The value of the evaluation function can be calculated by substituting the difference image data calculated in this way into Equation (1).

  Here, as described above, in the super-resolution processing, the smaller the value of the evaluation function, the higher the resolution is achieved. Therefore, in the super-resolution processing, the evaluation function is compared with a predetermined threshold value, and when the value of the evaluation function is smaller than the threshold value, it is determined that sufficiently high-resolution image data has been generated and super-resolution is performed. The process ends.

  On the other hand, if the value of the evaluation function is equal to or greater than the threshold value, the high resolution image data is updated by determining that the high resolution accuracy of the high resolution image data is not sufficient. The high resolution image data is updated so that the value of the evaluation function becomes small. For this processing, for example, a known iterative calculation method such as the steepest descent method can be applied.

  In the super-resolution processing, the new high-resolution image data obtained in this way is optically simulated, and difference with the captured image data is generated to generate difference image data. By repeating this, high-resolution image data having desired high resolution accuracy is generated.

Incidentally, as described in Patent Document 1 below, captured image data used for super-resolution processing includes a noise component. Examples of the cause of including the noise component include the following (i) to (iii).
(I) Sensitivity variation for each element of the image sensor (ii) Fluctuation of luminance value for each image capture in the same image sensor (iii) Noise randomly generated by data transfer after imaging Using captured image data including such noise components Therefore, the high-resolution image data generated when the super-resolution processing is performed naturally includes the influence of the noise component, and as a result, there is a problem that the S / N is lowered.

As a method for reducing the influence of the noise component, a method for incorporating captured image data information as a constraint term as described in Patent Document 2 below has been proposed. Further, as described in Patent Document 3 below, a method of removing the influence of high frequency noise components by smoothing processing has also been proposed.
JP 2002-247454 A (published on August 30, 2002) JP 2006-309649 A (published on November 9, 2006) JP 2006-203717 A (released on August 3, 2006)

  However, in the method described in Patent Document 2, although the influence of noise is reduced by adding a constraint term, the signal (output of high-resolution image data) is also relatively lowered. It is difficult to significantly improve N. Also in the method described in Patent Document 3, when the imaging target of the captured image data includes a high-frequency component, the high-frequency component is removed together with the noise component, so that the S / N is greatly improved. It is difficult to let

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an image processing apparatus capable of performing super-resolution processing with high accuracy by reducing the influence of noise components included in captured image data. It is in realizing.

  In order to solve the above problems, an image processing apparatus according to the present invention generates high-resolution image data having a higher resolution than the original image data generated based on the plurality of original image data, and the plurality of original image data. An image processing apparatus that repeatedly performs update processing for updating the high-resolution image data so that an error with noise-removed image data from which noise components have been removed from each of the image data becomes smaller than a predetermined threshold value. When performing the update process, noise removing means for generating the noise-removed image data by subtracting noise image data indicating a noise component included in the original image data from each of the original image data, When the error is greater than or equal to the threshold value in the update process, the noise image data is updated so that the error is reduced. Is characterized by comprising a noise image updating means that.

  Further, in order to solve the above problems, the image processing method of the present invention generates high-resolution image data having a higher resolution than the original image data generated based on the plurality of original image data, and the plurality of original images. An image processing apparatus that repeatedly performs update processing for updating the high-resolution image data so that an error from noise-removed image data obtained by removing noise components from each of the image data is reduced until the error becomes smaller than a predetermined threshold value. When the update process is performed, the noise-removed image data is generated by subtracting noise image data indicating a noise component included in the original image data from each of the original image data. Noise removal step, and when the error is greater than or equal to the threshold in the update process, the error is reduced. It is characterized in that it comprises a noise image update step of updating the noise image data.

  As described above, an update process for updating the high-resolution image data so that an error between the high-resolution image data generated based on the plurality of original image data and the plurality of original image data is reduced. It has been conventionally performed to generate high-resolution image data with desired high resolution accuracy by repeatedly performing until the value becomes smaller than a predetermined threshold value.

  Specifically, the high resolution image data is updated as follows. That is, the difference image data between the high resolution image data and the original image data is generated, and the generated difference image data is substituted into a predetermined evaluation function to evaluate the error between the high resolution image data and the original image data. A value is calculated, and the calculated evaluation value is compared with the threshold value. And when the said evaluation value is more than a threshold value, high resolution image data is updated so that the said evaluation value may become small using well-known repeated calculation methods, such as the steepest descent method, for example.

  Here, according to the configuration of the present invention, noise-removed image data generated by subtracting noise image data indicating a noise component of the original image data from the original image data is used instead of the original image data.

  That is, in the configuration of the present invention, the update process is performed so that the error between the high-resolution image data and the noise-removed image data obtained by removing the noise component from each of the plurality of original image data is reduced.

  Therefore, the influence of noise components included in the original image data on the update of the high-resolution image data is reduced, and high-definition high-resolution image data that accurately reflects the shape and color of the subject of the original image data is generated. It becomes possible.

  More specifically, in the present invention, difference image data between the high-resolution image data and the noise-removed image data is generated, and the generated difference image data is substituted into a predetermined evaluation function to obtain the high-resolution image. An evaluation value indicating an error between the data and the noise-removed image data is calculated, and the calculated evaluation value is compared with the threshold value. And when the said evaluation value is more than a threshold value, high resolution image data is updated so that the said evaluation value may become small using well-known repeated calculation methods, such as the steepest descent method, for example.

  The noise image data has an evaluation value that is an error between the high-resolution image data and the noise-removed image data, that is, an evaluation function value obtained by substituting the high-resolution image data and the noise-removed image data into the evaluation function in advance. When it is equal to or greater than a predetermined threshold, the error is updated so as to reduce the error.

  Therefore, according to the configuration of the present invention, the high-resolution image data is updated so that the plurality of original image data is accurately reflected in the high-resolution image data, and the noise image also includes the plurality of original image data. It is updated so that it is accurately reflected in the image data.

  As a result, the noise image data is also updated to image data reflecting a plurality of original image data, so that the noise components included in the plurality of original image data are selected by using the updated noise image data. Can be removed. As a result, it is possible to reduce the influence of noise components included in the original image data and generate high-resolution image data with very high accuracy.

  The plurality of original image data is obtained by imaging the imaging target little by little at a shifted position. By using original image data obtained by imaging the same part of the imaging target at a plurality of different imaging positions, it is possible to generate high-resolution image data having a higher resolution than the individual original image data.

A known method can be applied as a method for capturing original image data for generating high-resolution image data, and is not particularly limited. For example, in a state where the plane on which the imaging elements of the imaging apparatus are arranged is parallel to the imaging target surface to be imaged, 1/1 pixel of the imaging element in the first direction parallel to the imaging target surface. The image is taken by moving the image pickup position N times at an N pitch and further moving the image pickup position N times at a 1 / N pitch in a second direction parallel to the image pickup surface and perpendicular to the first direction. The obtained N ² pieces of image data can be used as the plurality of original image data.

Note that N is a positive number and is determined by the magnification for increasing the resolution. For example, when the resolution of the original image data is doubled, N may be 2 or more. When the resolution of the original image data is increased 10 times, N may be 10 or more. N can be appropriately set as necessary. However, the imaging method and the number of images to be captured are not limited to the above example, and it is also possible to generate high-resolution image data with a resolution increased N times using original image data fewer than N ² images.

For example, when a registration method such as registration is used, it is possible to generate high-resolution image data with an arbitrary high-resolution magnification by using an arbitrary number of original image data. In this case, high-resolution image data having N times the resolution of the original image data can be generated using less than N ² original image data.

  The threshold value can be changed as appropriate according to the required accuracy of resolution enhancement. For example, when higher resolution and higher resolution are required, the threshold value may be set small, and higher resolution accuracy is not required, but when a rapid processing speed is required, the threshold value is set. Should be set larger. That is, the threshold value is a value for determining the accuracy of resolution enhancement, and can be appropriately set according to various conditions required for the generated high resolution image data.

  Further, when the characteristics of the high resolution image data are known, the error may be obtained in consideration of a constraint term based on the characteristics of the high resolution image data. By taking the constraint term into consideration, it is possible to generate high-resolution image data with higher accuracy by reflecting the characteristics of the high-resolution image data. Specifically, the error (evaluation value) may be obtained using a function obtained by adding a constraint term to the evaluation function.

  Further, it is not necessary to update the high-resolution image data and the noise image data the same number of times. When the number of updates of noise image data is less than the number of updates of high-resolution image data, the amount of calculation can be reduced.

  The noise image data is image data having a higher resolution than the original image data. The noise image data is aligned with each of the plurality of original image data, and the noise image data is the same as the original image data. An alignment noise image data generating unit that converts the resolution into a plurality of alignment noise image data is provided, and the noise removing unit aligns with the original image data from each of the plurality of original image data. It is preferable that the noise-removed image data is generated by subtracting the alignment noise image data generated in this way.

  According to the above configuration, noise image data having a higher resolution than the original image data, that is, more accurately reflecting the position and shape of noise is used. Therefore, it is possible to more accurately remove the noise component of the original image data.

  The noise image data is represented by the sum of products of noise base component data set in advance to indicate the position of noise included in each original image data and noise intensity at each noise position. It may be a thing.

  The imaging apparatus includes a plurality of imaging elements for capturing an imaging target as original image data, but there may be variations in imaging sensitivity of the imaging elements. This variation is one of the factors that include noise components in the original image data.

  Such sensitivity variation for each image sensor is determined by the characteristics of the image sensor and the like, so the position where noise occurs in the original image data can be acquired in advance. Therefore, the noise generation position acquired in advance can be used as the noise base component.

  Note that the noise generation position can be acquired in advance. However, when a plurality of original image data is used as in the present invention, the image of the imaging target is captured by the same imaging device of the same imaging device. Even so, it is difficult to obtain the intensity of the noise in advance because the luminance value of the captured image varies every time the image is taken.

  Therefore, in the above configuration, the noise image data is represented by the sum of the products of the noise base component data and the noise intensity at each noise position. As described above, when the error (evaluation value) is equal to or greater than a predetermined threshold, the noise image data is updated so that the error is reduced.

  Therefore, according to the above configuration, it is possible to generate noise image data with higher accuracy by reflecting the luminance value of the actual original image data, thereby further improving the accuracy of resolution enhancement. .

  Note that, as described in [Background Art], noise in original image data (captured image data) includes, for example, (i) variation in sensitivity for each element of the image sensor, and (ii) luminance value for each image capture in the same image sensor. There are noise that occurs due to factors derived from the imaging device, such as fluctuations, and (iii) noise that occurs randomly due to data transfer after imaging.

  The patterns in which the noises (ii) and (iii) are generated in the original image data are preliminarily determined based on the characteristics of the imaging device used for acquiring the original image data, as in the case of the noises (i). You can ask for it. Therefore, also for such noise, it is possible to obtain a noise generation position acquired in advance as a noise base component.

  The noise image data may be a set of frequency components that are censored as a quantization error by a point spread function obtained from a camera model of an imaging apparatus used to acquire the original image data.

  In super-resolution processing, in order to generate a blurred image from a high-resolution image, a point spread function (PSF function) obtained from an ideal camera model of the imaging apparatus may be used. The blurred image data generated by multiplying the image data by this point spread function includes only frequency components larger than a predetermined value. On the other hand, the frequency component equal to or lower than the predetermined value is not included in the blurred image data because it is truncated as a quantization error.

  Here, since the original image data is image data to be increased in resolution, the resolution is relatively low. Therefore, in the original image data, it is considered that a frequency component that is censored as a quantization error by the point spread function is not reflected.

  Therefore, when a frequency component that is censored as a quantization error by the point spread function is included in the original image data, the frequency component is not obtained by imaging of the imaging target, but is caused by noise. It is believed that there is.

  Here, according to the above configuration, the frequency component that is censored as a quantization error by the point spread function is removed from the original image data as noise image data. The noise image data is updated so that the value of the evaluation function becomes smaller when the value of the evaluation function is greater than or equal to a predetermined threshold value.

  Therefore, according to the above configuration, it is possible to generate noise image data with higher accuracy by reflecting the luminance value of the actual original image data, thereby further improving the accuracy of resolution enhancement. .

  Furthermore, the image processing apparatus, the imaging apparatus that captures the imaging target and acquires the plurality of original image data, and the inspection that performs the defect inspection of the imaging target based on the high-resolution image data generated by the image processing apparatus In the case of an inspection system including an apparatus, defect inspection can be performed using high-resolution image data from which the influence of noise has been removed, so that more accurate defect determination can be performed.

  The image processing apparatus may be realized by a computer. In this case, a control program for causing the image processing apparatus to be realized by a computer by causing the computer to operate as each unit of the image processing apparatus, and A computer-readable recording medium on which it is recorded also falls within the scope of the present invention.

  As described above, when performing the update process, the image processing apparatus of the present invention subtracts the noise image data indicating the noise component included in the original image data from each of the original image data, thereby removing the noise-removed image data. And a noise image updating unit that updates the noise image data so that the error is reduced when the error is greater than or equal to a threshold value in the update process.

  In addition, as described above, the image processing method of the present invention removes noise by subtracting noise image data indicating noise components included in the original image data from each of the original image data when performing update processing. The configuration includes a noise removing step for generating image data, and a noise image updating step for updating the noise image data so that the error is reduced when the error is greater than or equal to a threshold in the updating process.

  Therefore, noise components included in a plurality of original image data can be selectively removed, so that the influence of noise components included in the original image data can be reduced and highly accurate high-resolution image data can be generated. Is possible.

  An embodiment of the present invention will be described below with reference to FIGS. First, an outline of the imaging system of the present embodiment will be described based on FIG. As shown in the figure, the imaging system 1 has an imaging target P as an imaging target and includes a frame 2, an imaging device 4, a stage 5, a control device 6, and a monitor 7. Here, the vertical direction in FIG. 2 is the Z-axis direction, the horizontal direction is the X-axis direction, and the depth direction is the Y-axis direction.

  The imaging target P is a subject of the imaging device 4 and is fixed to the frame 2 so that the surface to be imaged faces the imaging device 4. That is, in the imaging system 1, the imaging target surface of the imaging target P is imaged by the imaging device 4. Note that the imaging target P may be any object, and a person, a plant, or a plant may be used as the imaging target P.

  Although details will be described later, according to the imaging system 1, the captured image data (original image data) obtained by imaging with the imaging device 4 is accurately increased in resolution to generate high-definition high-resolution image data. Can do. Therefore, the imaging system 1 may be used for an inspection system that inspects the presence or absence of a defect in the imaging target P based on high-resolution image data generated from captured image data obtained by imaging the imaging target P, for example. That is, an inspection system including the imaging system 1 and an inspection apparatus that inspects the presence or absence of a defect of the imaging target P based on high-resolution image data generated by the imaging system 1 also falls within the scope of the present invention.

  The imaging device 4 is a device that images the imaging target P, and includes a housing 4a, a lens 4b, a solid-state imaging device 4c, and an actuator 4d. The lens 4b focuses light (image of the imaging target P) that enters the inside of the housing 4a on the imaging surface of the solid-state imaging device 4c. The solid-state imaging device 4c is composed of a plurality of imaging devices, samples the optically imaged image of the imaging target P while spatially discretely sampling it, and converts these images into image signals. As the solid-state imaging element 4c, for example, a CCD (Charge Coupled Devices) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like can be applied. In the present embodiment, it is assumed that the imaging device 4 is an area sensor, but may be a line sensor.

  The actuator 4d changes the imaging position of the imaging target P by changing the position of the solid-state imaging element 4c. Specifically, the actuator 4d is fixed to the inner wall of the housing 4a, and the solid-state imaging device 4c is supported by the actuator 4d. Therefore, when the actuator 4d displaces the position of the solid-state imaging device 4c, the position of the solid-state imaging device 4c is displaced relative to the housing 4a, and the imaging position for imaging the imaging target P is also displaced accordingly. .

  Here, it is assumed that the actuator 4d moves the solid-state imaging device 4c two-dimensionally on a plane (YZ plane in the illustrated example) perpendicular to the optical axis connecting the imaging target P and the lens 4b. However, this movement may be a three-dimensional movement including rotation. As the actuator 4d, for example, a piezo actuator or a stepping motor can be used. In this embodiment, it is assumed that a piezo actuator is used.

  The imaging device 4 is fixed on the stage 5 so as to face the imaging target surface of the imaging target P. That is, in the present embodiment, the relative position between the imaging device 4 and the imaging target P is fixed, and it is assumed that the image of the imaging target P is captured by moving the solid-state imaging device 4c by the actuator 4d. Yes. In addition, by moving the stage 5 and / or the frame 2, the image of the imaging target P may be captured while changing the relative position between the imaging device 4 and the imaging target P.

  The control device 6 includes an imaging control unit 11 that controls the imaging device 4, a captured image storage unit 13 that stores captured image data captured by the imaging device 4, and high-resolution image data obtained by increasing the resolution of the captured image data. A super-resolution processing unit 12 to be generated is provided.

  That is, in the imaging system 1, the imaging device 4 images the imaging target P based on the control of the control device 6, and the captured image data obtained by imaging the imaging target P is the captured image storage unit 13 of the control device 6. To be stored in. Then, the super-resolution processing unit 12 performs super-resolution processing using the captured image data, and generates high-resolution image data having a higher resolution than the captured image data.

  The monitor 7 displays captured image data acquired by the imaging device 4, high-resolution image data generated from the captured image data, and the like. The monitor 7 may be any device that can display an image, and various display devices such as liquid crystal, organic EL, and CRT (Cathode Ray Tube) can be used as the monitor 7.

[Configuration of control device]
Then, the structure of the control apparatus 6 is demonstrated based on FIG. FIG. 3 is a block diagram showing a main configuration of the control device 6. As illustrated, the control device 6 includes an imaging control unit 11, a super-resolution processing unit (image processing device) 12, a captured image storage unit 13, a high-resolution image storage unit 14, an image output control unit 17, and a noise image storage. A portion 18 is provided.

  The imaging control unit 11 sends a control signal to the imaging device 4 to instruct the imaging timing control unit 11a that controls the timing of imaging (exposure) by the solid-state imaging device 4c (imaging start time, length of exposure time, etc.). Then, the imaging of the imaging target P is started at a predetermined timing.

  In the present embodiment, it is assumed that the imaging timing control unit 11a controls the imaging timing by changing the voltage of a control signal sent to the imaging device 4. In addition, the imaging timing control unit 11a instructs the actuator control unit 11b that controls the movement start timing, the moving speed, the moving distance, and the like of the actuator 4d, and according to the exposure time of the solid-state imaging device 4c included in the imaging device 4. The solid-state image sensor 4c is moved.

  And the imaging control part 11 respond | corresponds with the imaging position (position of the solid-state image sensor 4c) at the time of acquiring the said captured image data about the captured image data of the imaging target P sampled by the solid-state image sensor 4c of the imaging device 4. The attached captured image data 13 a is stored in the captured image storage unit 13.

  Note that in the super-resolution processing, high-resolution image data is generated based on a plurality of captured image data in which the solid-state imaging device 4c is not an integer assigned to one display pixel of the imaging target P. Therefore, it is necessary to capture the imaging target P while shifting the position little by little by the actuator 4d to acquire a plurality of captured image data, and it is also necessary to acquire the imaging position of each captured image data.

Specifically, the plurality of pieces of captured image data are parallel to the surface to be imaged in a state where the plane on which the solid-state image sensor 4c is arranged is parallel to the surface to be imaged of the imaging target P. A process of performing imaging by moving the imaging position N times at 1 / N pitch of one pixel of the imaging element in one direction (for example, the Z-axis direction in FIG. 2) parallel to the imaging surface and in the first direction N ² pieces of image data obtained by moving the imaging position N times at a 1 / N pitch in a second direction (eg, the Y-axis direction in FIG. 2) perpendicular to the image data. Then, recording the imaging position of the captured the captured image data for each of the N ² pieces of captured image data captured as described above.

  Note that N is a positive number and is determined by the magnification for increasing the resolution. For example, when the resolution of the captured image data is doubled, N = 2 may be used, and when the resolution of the captured image data is increased 10 times, N = 10 may be sufficient. N can be appropriately set as necessary.

  The super-resolution processing unit 12 performs super-resolution processing for generating high-resolution image data by increasing the resolution of the captured image data 13 a stored in the captured image storage unit 13. Specifically, the super-resolution processing unit 12 includes the captured image data 13 a stored in the captured image storage unit 13, the high-resolution image data 14 a stored in the high-resolution image storage unit 14, and the noise image storage unit 18. The new high-resolution image data 14a is generated based on the noise image data 18a stored in, and the generated high-resolution image data 14a is stored in the high-resolution image storage unit 14. Note that details of the super-resolution processing executed by the super-resolution processing unit 12 will be described in detail later.

  The image output control unit 17 acquires the high resolution image data 14 a stored in the high resolution image storage unit 14 and displays it on the monitor 7. Thereby, the user of the imaging system 1 can easily confirm the image of the imaging target P or the like.

[Flow of processing in imaging system]
Next, the flow of processing in the imaging system 1 having the above configuration will be described with reference to FIG. FIG. 4 is a flowchart illustrating an example of processing in the imaging system 1. First, when the imaging target P is fixed to the frame 2, the imaging device 4 captures the imaging target P in accordance with an instruction from the imaging control unit 11 of the control device 6. The captured image data 13a obtained by imaging is stored in the captured image storage unit 13 (S1).

  When the captured image data 13a is stored in the captured image storage unit 13, the super resolution processing unit 12 executes a super resolution process (S2). Specifically, the super-resolution processing unit 12 includes the high-resolution image data 14a stored in the high-resolution image storage unit 14 and the captured image stored in the captured image storage unit 13 when the super-resolution process is executed. Based on the data and the noise image data 18a stored in the noise image storage unit 18, new high-resolution image data 14a is generated. Then, the super-resolution processing unit 12 stores the generated new high-resolution image data 14a in the high-resolution image storage unit 14 (S3).

  The initial value of the high-resolution image data may be any value, but for example, gray image data in which the entire image data is gray can be used. Alternatively, image data obtained by selecting an arbitrary image from the captured image data 13a and interpolating the selected captured image data to expand the image size to high resolution image data may be used.

  Then, the image output control unit 17 displays the stored high resolution image data 14a on the monitor 7 in response to the high resolution image data 14a being stored in the high resolution image storage unit 14 (S4).

[Principle of super-resolution processing of the present invention]
Next, the principle of the super-resolution processing of the present invention will be described. In the super-resolution processing of the present invention, the evaluation function (error term) based on the square error between the “estimated low-resolution image” and the “observed observed image” is optimized as in the conventional super-resolution processing. However, in the super-resolution processing of the present invention, the evaluation function used is different from the conventional super-resolution processing.

  Specifically, in the super-resolution processing of the present invention, the influence of noise is removed by using the following formula (4) as an evaluation function. The principle of the super-resolution processing of the present invention will be described below.

The above mathematical formula (4) shows an example of the evaluation function used in the super-resolution processing of the present invention. Comparing Equation (4) with Equation (1), which is a conventional evaluation function, is different in that an αN _n Kz term is added to the error term. Note that the mathematical expression (4) includes a constraint term, but the constraint term may be omitted when the characteristics of the high-resolution image data cannot be acquired in advance. However, since the evaluation function includes a constraint term, it is possible to perform a higher-resolution super-resolution process. Therefore, the evaluation function preferably includes the constraint term.

Section of this alpha N n _Kz shows the noise component of the captured image data, the in equation (4), the noise component of the captured image data y _n by subtracting the alpha N n _Kz from the captured image data y _n is removed It has become so. Referred to herein as an image data noise removed image data obtained by subtracting the alpha N _n Kz from the captured image data _{y n (y n -αN n Kz} ).

In the term αN _n Kz indicating the noise component, K is a matrix representing the noise base component data. Also, z is an arbitrary vector, and z is referred to as a noise synthesis vector here. Here, Kz obtained by multiplying each column component of the matrix K by the element z is referred to as noise image data. A method for setting the matrix K will be described later.

N _n is a matrix for sampling the luminance value of the pixel of the noise image data at the position of each imaging element of the captured image data after aligning the noise image data to the captured position of each captured image data. . N _n can be expressed as the following formula (5) using the matrix M _i described above. Therefore, by multiplying N _n , the noise image data is aligned with the imaging position of the captured image data, and the resolution (number of elements) is the same as that of the captured image data. Α represents a weight for subtracting noise image data from captured image data.

  Next, a method for obtaining the matrix K representing the noise basis component data will be described. Since K represents the base component of noise, for example, it can be expressed as a matrix in which the number of rows is the number of pixels of the high-resolution image data and the number of columns is the number of base components. In this case, each column component of K represents the base component of the noise image data in the high resolution image data. The base component of each column can be expressed as spatial domain data or frequency domain data.

  For example, if the appearance location and shape of noise are known, the base component can be represented by a spatial domain. In this case, the estimated noise image data Kz is expressed as a linear sum of individual base components, that is, a product of K and an arbitrary noise synthesis vector z. When this is expressed by a mathematical expression, the following mathematical expression (6) is obtained.

In the above equation _(6), k 1 _{~k n} is a base component elements constituting the noise base component data _K, the value of k 1 to k _n, occurrence and shape of the noise is represented. That is, the noise base component data K indicates the position of noise included in the captured image data. Here, each of the k ₁ to k _n, is assumed to be the same resolution as the high resolution image data (number of elements). In the above mathematical formula (6), z _{1 to} z _n are elements constituting the noise synthesis vector z and indicate the intensity of noise at each position.

By multiplying the noise synthesis vector z to the noise base component data K, the noise image data is expressed as the sum of the product of the elements z ₁ to z _n of the noise base component element k ₁ to k _n and noise synthesis vector.

  Here, as described in [Background Art], the cause of noise generation includes, for example, (i) sensitivity variation for each element of the image sensor (ii) fluctuation in luminance value for each image capture in the same image sensor. . Noise generated by these factors is always observed at the same position in the captured image data. For example, consider a case where an imaging device in which imaging elements are arranged in a matrix is used, and imaging sensitivity is different between an odd-numbered imaging element and an even-numbered imaging element. In this case, all of the captured image data captured by the imaging apparatus is image data including a luminance difference derived from a sensitivity difference of the image sensor between the odd-numbered pixels and the even-numbered pixels.

In this way, noise image data having the same resolution as the captured image data can be used. In this case, since it is not necessary to align the captured image data and the noise image data, and it is not necessary to match the resolution, the calculation can be simplified by omitting N _n in Equation (4). it can.

  As in the example shown in the present embodiment, when the noise image data has the same resolution as the high-resolution image data, for example, noise image data having the same resolution as the captured image data obtained from the characteristics of the image sensor is What has been increased in resolution by interpolation processing or enlargement processing may be used.

  On the other hand, when the noise is defined in the frequency domain, the noise base component data K can be obtained from the PSF of the camera model, for example. That is, when high-resolution image data is blurred with PSF (multiplying by matrix C), it is considered that a high-frequency component that is censored as a quantization error in luminance gradation is not expressed in captured image data with low resolution. Therefore, when such a high frequency component appears in the captured image data, this component can be regarded as a noise component.

  In this case, the following equation (7) is established with respect to the matrix C for blurring the image by convolving the PSF of the camera model and the noise base component data K. In Equation (7), e is a vector whose elements are all 1. Therefore, when the noise image data Kz satisfying Expression (7) is blurred by PSF (multiplied by the matrix C), all the elements thereof are 1 or less, and therefore, the quantization error of the luminance gradation is cut off.

By multiplying ^{C -1} to both sides of the equation (7), a relational expression between Kz ≦ ^{C -1} e is obtained. Therefore, by using z ′, which is a vector in which all elements are in the range of 0 to 1, the noise image data Kz can be defined as in the following formula (8). That is, in this case, C ⁻¹ z ′ can be used in place of Kz in Equation (4) and the like.

  As described above, although the value differs depending on whether it is expressed as data in the spatial domain or as data in the frequency domain, the noise base component data is obtained in advance before performing the super-resolution processing. be able to.

  However, since the noise synthesis vectors z and z ′ are unknown numbers, it is necessary to obtain them when performing super-resolution processing. That is, in the super-resolution processing according to the present invention, the evaluation function represented by Equation (4) includes two unknowns, the high-resolution image data h and the noise synthesis vector z.

  Therefore, in the super-resolution processing of the present invention, in addition to the conventional super-resolution processing step, processing for obtaining the noise synthesis vector z or z ′ is required. Hereinafter, an example of a method for deriving the high-resolution image data h and the noise synthesis vector z will be described. In the following, an example of deriving the noise synthesis vector z will be described, but the deriving method is the same even if the noise synthesis vector is z ′.

  As described above, the super-resolution processing is processing for minimizing the value of the evaluation function, and this processing corresponds to processing for bringing the gradient of the evaluation function with respect to the high-resolution image data h and the noise synthesis vector z close to zero. To do. That is, the noise synthesis vector z can be calculated in the same manner as the high resolution image data h. This can be expressed by the following mathematical formulas (9) and (10). Note that T in Expressions (9) and (10) represents transposition of a matrix.

  The high-resolution image data h satisfying Equation (9) and the noise synthesis vector z satisfying Equation (10) can be obtained by a known iterative calculation method. For example, when the steepest descent method is used for the repetitive calculation, the high resolution image data h satisfying the formula (9) can be asymptotically obtained by the following formula (11).

  Similarly, it is possible to asymptotically determine the noise synthesis vector z that satisfies the formula (10) by the following formula (12). Note that β in the following equations (11) and (12) controls the convergence speed in the steepest descent method, and the same value β may be applied to both equations, or different values of β may be applied. May be.

[Flow of super-resolution processing of the present invention]
A super-resolution processing method according to the present invention using Equation (4) as an evaluation function will be described with reference to FIG. FIG. 5 is a diagram for explaining the super-resolution processing method of the present invention. As shown in the figure, in the super-resolution processing of the present invention, in addition to the conventional super-resolution processing step shown in FIG. 9, there is a step (loop on the right side in FIG. 5) for generating alignment noise image data. Have been added. As a result, it is possible to remove noise components from the captured image data and perform highly accurate super-resolution processing.

In the super-resolution processing of the present invention, as shown in the figure, difference image data is generated by subtracting noise-removed image data obtained by subtracting the alignment noise image data from the captured image data from the pseudo low-resolution image data. ing. That is, the pseudo low-resolution image data is equivalent to B _{n h} of equation (4), the captured image data corresponds to y _n, alignment noise image data corresponds to alpha N n _Kz, noise-free image data (y _n -αN _n Kz). Therefore, the difference image data is represented as {B _n h− (y _n −αN _n Kz)}.

Here, as shown in the figure, it is assumed that the noise image data is a single image data with high resolution. Therefore, it is not possible to directly calculate the difference from the captured image data which is low resolution and n pieces of image data. Therefore, by multiplying the noise image data by the matrix N _n , n pieces of alignment noise image data having the same resolution and the same image pickup position as the image pickup image data are generated, and each image pickup image data is aligned with the image pickup image data. By subtracting the alignment noise image data generated in this way, noise-removed image data that is the difference between the captured image data and the noise image data is generated.

Similarly, since the high resolution image data is high resolution and single image data, it is not possible to directly calculate a difference from the captured image data that is low resolution and n image data. Therefore, by multiplying the high-resolution image data by the matrix _Bn , pseudo low-resolution image data that is aligned with each captured image data and reduced in resolution is generated. Then, the difference image data is generated by subtracting the pseudo low-resolution image data generated by aligning with the captured image data that is the original data of the noise-removed image data from each noise-removed image data.

  Note that when noise image data having the same number of images, image pickup position, and resolution is used as each image pickup image data, it is not necessary to generate alignment noise image data. The difference between the data and each noise image data may be directly calculated. In addition, the difference calculation of the image data is performed by subtracting the pixel values of the pixels corresponding to each other in the image data to be subjected to the difference calculation, that is, the pixels indicating the same position on the imaging target P.

  Here, as described above, in the super-resolution processing, the smaller the value of the evaluation function, that is, the evaluation value, the higher the resolution is achieved. Therefore, even in the super-resolution processing of the present invention, the evaluation value is compared with a predetermined threshold value, and when the evaluation value is smaller than the threshold value, it is determined that high-resolution image data with sufficient accuracy has been generated. The resolution process ends.

  On the other hand, when the evaluation value is equal to or greater than the threshold value, it is determined that the high resolution image data is not sufficiently accurate, and the high resolution image data is updated and the noise image data is updated. The high resolution image data and the noise image data are updated so that the value of the evaluation function becomes small. For updating the high-resolution image data and the noise image data, a known iterative calculation method can be appropriately used. For example, the high-resolution image data can be updated by using the equation (11). By using (12), the noise image data can be updated.

  In the super-resolution processing of the present invention, pseudo low-resolution image data is generated from the updated high-resolution image data, and alignment noise image data is generated from the updated noise image data to generate difference image data. To do. By repeating this, high-resolution image data having desired high resolution accuracy is generated.

  As described above, the super-resolution processing method of the present invention removes the influence of noise from captured image data, thereby improving the S / N of high-resolution image data. In the control device 6 of this embodiment, the super-resolution processing unit 12 executes the super-resolution processing method as described above. Hereinafter, a more detailed configuration of the super-resolution processing unit 12 and specific processing contents of the super-resolution processing executed by the super-resolution processing unit 12 will be described with reference to FIGS. 1 and 6.

[Details of super-resolution processing unit]
FIG. 1 is a block diagram illustrating a main configuration of the super-resolution processing unit 12. As illustrated, the super-resolution processing unit 12 includes a high-resolution image registration unit 12a, a noise image registration unit (positioning noise image generation unit) 12b, and an image update unit (noise removal unit, noise image update unit). ) 12c.

  The high resolution image alignment unit 12a aligns the high resolution image data with each captured image data, and generates alignment high resolution image data corresponding to each captured image data. Then, the high resolution image alignment unit 12a artificially lowers the resolution of the alignment high resolution image data and generates pseudo low resolution image data.

  Specifically, the high resolution image image alignment unit 12a first calculates the relative position of the captured image data with respect to the high resolution image data. Here, in the imaging system 1, as described above, the imaging position of the captured image data is acquired by the imaging control unit 11, and the captured image data is stored in the captured image storage unit 13 as captured image data 13a associated with the imaging position. It is assumed to be stored.

  Therefore, the high-resolution image image alignment unit 12a calculates a relative imaging position of each captured image data with respect to the high-resolution image data based on the captured image data 13a stored in the captured image storage unit 13. Can do. When the imaging position of the captured image data is not known, the relative imaging position of the captured image data with respect to the high-resolution image data may be detected by a registration method based on pattern matching or luminance difference.

Then, the high resolution image image alignment unit 12a aligns the high resolution image data with the imaging position of each captured image data based on the relative imaging position of the captured image data with respect to the calculated high resolution image data. Thereafter, a matrix B _n indicating a process for reducing the resolution by convolving a kernel of a point spread function (PSF function) obtained from the camera model of the imaging device 4 is calculated.

That is, by multiplying the matrix B _n high-resolution image data h, high resolution image data h is aligned with the respective captured image data y _n, the pseudo low is low resolution to the same resolution as image data y _n The resolution image data B _n h is generated.

The high-resolution image image alignment unit 12a multiplies the matrix B _n calculated in this way by the high-resolution image data stored in the high-resolution image storage unit 14 to calculate pseudo low-resolution image data B _n h, The calculated pseudo low-resolution image data B _n h is output to the image update unit 12c.

  The noise image alignment unit 12b performs processing for aligning noise image data to each captured image data and resolution of the noise image data when the captured image data and the noise image data have different imaging positions and resolutions. Is performed at least one of the processes for matching the captured image data to generate the alignment noise image data.

  In the present embodiment, since it is assumed that one piece of noise image data having a higher resolution than the captured image data (the same imaging position and the same resolution as the high-resolution image data) is used, the noise image alignment unit 12b Then, the noise image data is subjected to alignment and resolution reduction processing to generate a plurality of alignment noise image data corresponding to each captured image data.

Specifically, the noise image alignment unit 12b aligns the noise image data with the imaging position of each captured image data, and then calculates the luminance value of the pixel of the noise image data at the position of each imaging element of the captured image data. N _n which is a matrix for sampling is obtained. A specific method for obtaining N _n is the same as the case where the high-resolution image image alignment unit 12a obtains B _n .

Note that when the resolution and imaging position of the noise image data are the same as those of the high-resolution image data as in this embodiment, the components M _{1 to} M _n that are the components of the matrix N _n are the components of the matrix B _n . the same value as _M 1 ~M _n is. Therefore, the noise image alignment unit 12b can obtain the matrix N _n using M _{1 to} M _n calculated by the high-resolution image image alignment unit 12a without performing calculation to obtain the matrix N _n .

Then, the noise image image alignment unit 12a multiplies the noise image data stored in the noise image storage unit 18 by the matrix N _n calculated in this way, and further multiplies the weight α to align the noise image data αN _n. Kz is calculated, and the calculated alignment noise image data αN _n Kz is output to the image updating unit 12c.

  The image update unit 12c generates noise-removed image data by subtracting the alignment noise image data from each of the plurality of captured image data, and subtracts the generated noise-removed image data from the high-resolution image data to obtain difference image data. And the value of the evaluation function is calculated using the generated difference image data. Then, when the value of the evaluation function is equal to or greater than a predetermined threshold, the image update unit 12c repeats the update process for updating the high-resolution image data and the noise image data until the value of the evaluation function becomes smaller than the threshold. Do.

Specifically, the image updating unit 12c generates a noise-free image data by subtracting the alignment noise image data alpha N _n Kz from the captured image data _{y n} obtained from the captured image storage unit 13 _{(y n} -αN n Kz) Then, the difference image data (B _n h−y _n + αN _n Kz) is calculated by subtracting the noise-removed image data from the pseudo low resolution image data B _n h received from the high resolution image alignment unit 12a.

  Then, the image update unit 12c substitutes the calculated difference image data into the evaluation function to obtain an evaluation function value, that is, an evaluation value, and determines whether or not the evaluation function value is smaller than a predetermined threshold value. If it is smaller than the threshold value, the image updating unit 12c transmits the end of the super-resolution processing to the defect inspection processing unit 15 and ends the process. On the other hand, when the evaluation value is equal to or greater than a predetermined threshold, the image update unit 12c updates the high-resolution image data and the noise image data based on the obtained difference image data.

  The threshold value can be changed as appropriate according to the required accuracy of resolution enhancement. For example, when higher resolution and higher resolution are required, the threshold value may be set small, and higher resolution accuracy is not required, but when a rapid processing speed is required, the threshold value is set. Should be set larger. That is, the threshold value is a value for determining the accuracy of the super-resolution processing, and can be set as appropriate according to various conditions required for the super-resolution processing.

[Flow of super-resolution processing]
Next, the flow of super-resolution processing in the super-resolution processing unit 12 will be described based on FIG. FIG. 6 is a flowchart illustrating an example of super-resolution processing. Note that the flowchart of FIG. 6 corresponds to the process of S2 in the flowchart of FIG.

When captured image data y _n of n sheets used in the super-resolution processing is stored in the captured image storage unit 13, the super-resolution processing section 12 obtains the stored image data y _n (S11). Image data y _n of the super-resolution processing unit 12 has acquired is sent to the high-resolution image registration unit 12a and the noise image registration unit 12b.

High-resolution image registration unit 12a that has received the image data y _n is the high-resolution image storage unit 14 to acquire a high-resolution image data h stored in the high resolution image data h each captured image data obtained After alignment with the imaging position of y _n , a matrix B _n indicating a process for reducing the resolution by convolving the kernel of the PSF function obtained from the camera model of the imaging device 4 is calculated. Then, the high-resolution image registration unit 12a generates a pseudo low-resolution image data B _{n h} is multiplied by a B _n calculated above to the high-resolution image data h (S12). The high resolution image alignment unit 12a sends the generated pseudo low resolution image data B _n h to the image update unit 12c.

The noise image registration unit 12b that has received the image data y _n obtains noise image data Kz stored in the noise image storage unit 18, the acquired noise image data Kz each captured image data y _n Alignment noise image data αN _n Kz is generated (S13).

  In addition, although the noise basis component K of the noise image data Kz uses a value obtained in advance, the noise synthesis vector z is updated by iterative calculation as shown in FIG. Such a value may be used. For example, the initial value of the noise synthesis vector z may be set so that the entire noise image data Kz is a black black image.

Then, the noise image alignment unit 12b sends the generated alignment noise image data αN _n Kz to the image update unit 12c. In the illustrated example, the process of S13 is performed after S12. However, the process of S13 may be performed first, or the process of S12 and the process of S13 may be performed simultaneously. .

The image updating unit 12c that has received the pseudo low-resolution image data B _n h and the alignment noise image data αN _n Kz generates noise-removed image data by subtracting the noise image data αN _n Kz from the captured image data y _n . Then, by subtracting the noise-removed image data (y _n −αN _n Kz) from the pseudo low resolution image data B _n h, difference image data (Bnh−y _n + αN _n Kz) is generated (S14).

  Subsequently, the image update unit 12c calculates the value (evaluation value) E of the evaluation function by substituting the generated difference image data into the equation (4) (S15). Then, the image update unit 12c compares the obtained evaluation value E with a predetermined threshold value (S16). When the evaluation value E is smaller than the threshold value (Yes in S16), the image update unit 12c transmits the end of the super-resolution processing to the defect inspection processing unit 15 and ends the processing.

  On the other hand, when the evaluation value E is equal to or greater than the threshold (No in S16), the image update unit 12c updates the high-resolution image data h and the noise image data Kz based on the evaluation value E (S17). Specifically, the image update unit 12c performs high-speed image data h by repeatedly performing calculations using the above formula (11), and updates the high-resolution image data h after the update to the high-resolution image storage unit 14. The high-resolution image data h stored in is overwritten. Further, the image update unit 12c performs the calculation repeatedly using the above formula (12), thereby updating the noise image data Kz, and the noise image data Kz after the update is stored in the noise image storage unit 18. The image data Kz is overwritten.

When the updated high-resolution image data h and noise image data Kz are overwritten and saved, the image update unit 12c indicates that the high-resolution image data h and noise image data Kz are updated. It transmits to the matching part 12b. Upon receiving this transmission, the high resolution image alignment unit 12a and the noise image alignment unit 12b again generate the pseudo low resolution image data B _n h and the alignment noise image data αN _n Kz (S12, 13).

  As described above, according to the super-resolution processing method performed by the super-resolution processing unit 12, when the evaluation value is equal to or greater than the threshold value, the noise image data Kz is updated together with the high-resolution image data h. Yes. Therefore, according to the super-resolution processing method, the super-resolution processing can be performed based on the noise image data Kz that more accurately reflects the noise included in the captured image data. It becomes possible to perform highly accurate super-resolution processing by reducing the influence of noise components.

  The above flow chart shows an embodiment of the present invention, and the super-resolution processing of the present invention is not limited to the processing content of this embodiment. For example, the above flowchart shows an aspect in which the update of the high-resolution image data and the update of the noise image data are performed the same number of times, but the number of updates is not necessarily the same. When the number of updates of the noise image data is less than the number of updates of the high-resolution image data, the amount of calculation for super-resolution processing can be reduced.

  Below, how advantageous the present invention is over the prior art will be described with reference to FIGS. FIG. 7A is a diagram illustrating processing target image data 31 that is a target of super-resolution processing in the present embodiment. The processing target image data 31 corresponds to the imaging target P in the above-described embodiment, and is data in which bright portions 31a are arranged in a matrix as illustrated. Here, it is assumed that among the nine bright portions 31a arranged in the vertical 3 × horizontal 3, the central bright portion 31b has a higher luminance value than the other bright portions 31a.

  In this embodiment, instead of actually capturing the processing target image data 31 with the imaging device, the processing target image data 31 is optically degraded, and the processing target image data 31 is applied to the imaging device by multiplying noise generated by the imaging. Thus, captured image data equivalent to that obtained by imaging is generated.

  Specifically, first, the processing target image data 31 was optically degraded with a PSF corresponding to imaging with the imaging element allocation shown in FIG. FIG. 4B is a diagram showing the position of each image sensor with respect to the processing target image data 31. In the figure, each region 32a surrounded by a broken line 32 indicates the position of one image sensor. Thus, by optically degrading, it is possible to generate image data equivalent to a case where an image is picked up by assignment of the image pickup element indicated by the broken line 32.

  Next, the image data generated in this way was multiplied by the sensitivity variation in a checkered pattern as shown in FIG. FIG. 7C is a diagram showing variation in sensitivity multiplied by the processing target image data. As shown in the figure, the sensitivity variation pattern 33 is a pattern in which the hatched region 33a and the hatched region 33b are alternately arranged for each of the regions 32a shown in FIG. ing. Here, it is assumed that the sensitivity variation of −n [%] occurs in the region 33a, and the sensitivity variation of + n [%] occurs in the region 33b.

  Then, after the image data generated as described above is further multiplied by noise, the image data obtained by sampling at the position of each image sensor is used as the captured image data. Here, as the PSF, a Gaussian PSF having a standard deviation of 0.56 times the size of the image sensor was used. In addition, Gaussian noise with a standard deviation of luminance of 1.43 was used as the noise.

  In this embodiment, the conventional super-resolution processing method, the method of performing the smoothing process after the super-resolution processing by the conventional method, and the super-resolution processing method of the present invention are used. The captured image data generated in this way was increased in resolution three times in the vertical direction and three times in the horizontal direction to generate high-resolution image data. In the super-resolution processing method of the present invention, 2 is used as the value of the weight α multiplied by the noise image data in Equation (4).

  FIG. 8 shows the relationship between S / N and imaging sensitivity variation in the high-resolution image data generated as described above. As illustrated, the conventional super-resolution processing method has the lowest S / N ratio among the three methods. On the other hand, according to the method of performing the smoothing process (smoothing) after the super-resolution process by the conventional method, the S / N is improved as compared with the conventional super-resolution process method. Then, according to the super-resolution processing method of the present invention, it can be seen that the S / N is further improved as compared with the method of performing the smoothing process after the super-resolution process by the conventional method.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope shown in the claims. That is, embodiments obtained by combining technical means appropriately modified within the scope of the claims are also included in the technical scope of the present invention.

  Moreover, in the said embodiment, the super-resolution process part 12 which performs a super-resolution process, the captured image storage part 13 which memorize | stores various data so that reading is possible, the high-resolution image storage part 14, the noise image storage part 18, and a defect The configuration in which the inspection image storage unit 16 and the defect inspection processing unit 15 that performs defect inspection are included in the control device 6 has been shown, but some or all of these components are configured separately. You can also Furthermore, in the above-described embodiment, an example in which the high-resolution image alignment unit 12a and the noise image alignment unit 12b are configured separately has been described, but these may be integrated into one component.

  In the present embodiment, an example in which super-resolution processing is directly performed on captured image data captured using a solid-state imaging device has been described. However, an image to be processed is data captured in advance. There may be. That is, in the present invention, high-definition high-resolution image data in which the influence of noise is reduced can be generated by supplying captured image data captured by an arbitrary imaging device to the super-resolution processing unit 12.

  The present invention can also be expressed as follows. That is, the image processing apparatus of the present invention converts a plurality of first image data (captured image data) having a first resolution into second image data (high-resolution image) having a second resolution higher than the first resolution. An image processing unit for generating registration data for a predetermined reference position of the first image data, and a configuration of imaging noise included in the image data to be processed Noise basis component generation means for generating noise basis component data representing a component, noise-removed image data obtained by removing noise estimated by combining the noise basis component data from the first image data, and the second By repeatedly calculating until the value of the mathematical expression (evaluation function) for calculating the difference after alignment with the image data becomes smaller than a predetermined value It is characterized in that it comprises a high resolution image generation means for generating the second image data.

  The noise base component data is determined in advance when multiplied by an optical degradation function determined by imaging of the lens of the imaging device used for imaging the first image data and discretization by a solid-state imaging device. It is preferably expressed as a set of frequency components that are less than or equal to the specified value.

  The noise base component data may be expressed as a set of image data having a pattern smaller than the pixel size of the first image data on the second image data.

  The present invention can also be expressed as follows. That is, the image processing apparatus according to the present invention generates high-resolution image data having a higher resolution than the captured image data based on a plurality of captured image data obtained by imaging one imaging target using the imaging apparatus. A high-resolution image storage unit that stores high-resolution image data, and noise image data that indicates noise components generated in each of the plurality of captured image data by performing imaging with the imaging device A high-resolution image data stored in the noise image storage unit and the high-resolution image storage unit is aligned with each of the plurality of captured image data, converted to the same resolution as the captured image data, and the imaging device A pseudo low-resolution image generating means for generating a plurality of pseudo low-resolution image data by multiplying a point spread function obtained from the camera model of Difference image generation means for generating a plurality of difference image data by subtracting pseudo low-resolution image data and noise image data generated by aligning the captured image data with each of the captured image data, and Based on the generated plurality of difference image data, an evaluation function value indicating an error between the high-resolution image data and the plurality of captured image data is calculated, and the evaluation function value is equal to or greater than a predetermined threshold value. In some cases, the high-resolution image data and the noise image data are updated so as to reduce the value of the evaluation function, and the image update unit stores the data in the high-resolution image storage unit and the noise image storage unit, respectively. It is characterized by having.

  According to the above configuration, the high-resolution image data and the noise image data are updated so that the value of the evaluation function becomes small when the value of the evaluation function is equal to or greater than a predetermined threshold value. That is, when the value of the evaluation function is equal to or greater than a predetermined threshold, the high resolution image data and the noise image data are updated so that the plurality of captured image data are accurately reflected in the high resolution image data.

  Then, based on the updated high resolution image data and noise image data, the value of the evaluation function is calculated again, and if the value of the evaluation function is equal to or greater than a predetermined threshold, the high resolution image data and the noise image data are It will be updated again. By repeating such processing, the high-resolution image data and the noise image data are updated until the value of the evaluation function becomes smaller than a predetermined threshold value.

  Therefore, when the value of the evaluation function becomes smaller than a predetermined threshold, the high resolution image storage unit stores high resolution image data that accurately reflects a plurality of captured image data.

  Since the noise image data is also updated in the same manner as the high resolution image data, according to the above configuration, the value of the evaluation function is calculated based on the noise image data that accurately reflects the plurality of captured image data. The

  As a result, noise components included in a plurality of captured image data can be selectively removed, so that the influence of the noise components included in the captured image data is reduced and highly accurate high-resolution image data is generated. It becomes possible.

  Finally, each block of the control device 6, in particular, the imaging control unit 11, the super-resolution processing unit 12, and the defect inspection processing unit 15 may be configured by hardware logic or using a CPU as follows. It may be realized by software.

  That is, the control device 6 includes a CPU (central processing unit) that executes instructions of a control program that realizes each function, a ROM (read only memory) that stores the program, a RAM (random access memory) that expands the program, A storage device (recording medium) such as a memory for storing the program and various data is provided. An object of the present invention is to provide a recording medium in which a program code (execution format program, intermediate code program, source program) of a control program of the control device 6 which is software that realizes the above-described functions is recorded so as to be readable by a computer. This can also be achieved by supplying the control device 6 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

  Examples of the recording medium include a tape system such as a magnetic tape and a cassette tape, a magnetic disk such as a floppy (registered trademark) disk / hard disk, and an optical disk such as a CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

  The control device 6 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Also, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  According to the present invention, the influence of noise components included in captured image data can be reduced and super-resolution processing can be performed, so that the resolution of image data is increased without being limited to an apparatus for inspecting product defects. Applicable to various devices. Further, the target of higher resolution is not limited to still images, and it is possible to increase the resolution of moving images as well.

1, showing an embodiment of the present invention, is a block diagram illustrating a main configuration of a super-resolution processing unit. FIG. 1 is a diagram illustrating an outline of an imaging system according to an embodiment of the present invention. It is a block diagram which shows the principal part structure of the control apparatus with which the said imaging system is provided. It is a flowchart which shows an example of the process in the said imaging system. It is a figure for demonstrating the super-resolution processing method of this invention. It is a flowchart which shows an example of the super-resolution process performed in the said super-resolution process part. FIG. 4A is a diagram showing processing target image data that is a target of super-resolution processing in the embodiment of the present invention, and FIG. 4B is a diagram showing a position of each image sensor with respect to the processing target image data. FIG. 6C is a diagram showing variation in sensitivity multiplied by the processing target image data. S in three types of high-resolution image data generated using each of the conventional super-resolution processing method, the method of performing the smoothing process after the super-resolution processing by the conventional method, and the super-resolution processing method of the present invention, It is a figure which shows the relationship between / N and the dispersion | variation in imaging sensitivity. It is a figure for demonstrating the conventional super-resolution processing method.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging system 4 Imaging apparatus 12 Super-resolution processing part (image processing apparatus)
12a High-resolution image alignment unit 12b Noise image alignment unit (alignment noise image generation means)
12c Image update unit (noise removal means, noise image update means)
13 Captured image storage unit 14 High-resolution image storage unit 18 Noise image storage unit

Claims (8)

The error between the high-resolution image data having a higher resolution than the original image data and the noise-removed image data obtained by removing the noise component from each of the plurality of original image data is small. An image processing apparatus that repeatedly performs update processing for updating the high-resolution image data until the error becomes smaller than a predetermined threshold value,
Noise removing means for generating the noise-removed image data by subtracting noise image data indicating a noise component included in the original image data from each of the original image data when performing the update process;
An image processing apparatus comprising: a noise image updating unit configured to update the noise image data so that the error is reduced when the error is equal to or greater than the threshold value in the update process. The noise image data is image data having a higher resolution than the original image data,
Alignment noise image data generating means for aligning the noise image data with each of the plurality of original image data and generating the plurality of alignment noise image data by converting the noise image data to the same resolution as the original image data With
The noise removal means generates the noise-removed image data by subtracting the alignment noise image data generated by alignment with the original image data from each of the plurality of original image data. The image processing apparatus according to claim 1.   The noise image data is represented by a sum of products of noise base component data set in advance to indicate the position of noise included in each original image data and noise intensity at each noise position. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   The noise image data is a set of frequency components that are censored as a quantization error by a point spread function obtained from a camera model of an imaging apparatus used for acquiring the original image data. 2. The image processing apparatus according to 2. An image processing apparatus according to any one of claims 1 to 4,
An imaging device that captures an imaging target and obtains the plurality of original image data;
An inspection system including: an inspection device that performs defect inspection of the imaging target based on high-resolution image data generated by the image processing device. The error between the high-resolution image data having a higher resolution than the original image data and the noise-removed image data obtained by removing the noise component from each of the plurality of original image data is small. An image processing method of an image processing apparatus for repeatedly performing update processing for updating the high-resolution image data until the error becomes smaller than a predetermined threshold,
A noise removal step of generating the noise-removed image data by subtracting noise image data indicating a noise component included in the original image data from each of the original image data when performing the update process;
And a noise image updating step of updating the noise image data so that the error is reduced when the error is greater than or equal to the threshold value in the update process.   An image processing program for operating the image processing apparatus according to any one of claims 1 to 5, wherein the image processing program causes a computer to function as each of the means.   A computer-readable recording medium on which the image processing program according to claim 7 is recorded.

Images