JP2009071383A - Image processing apparatus, inspection system, image processing method, image processing program, and computer-readable recording medium recording this program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus reducing an operation amount required for ultra-resolution processing by obtaining second high-resolution image data corresponding to (n+1) sheets of image data obtained by adding processing target image data to n-sheets of image data, in an operation for two sheets of image data such as first high resolution image data and processing target image data. <P>SOLUTION: A controller 6 in the image processing apparatus is provided with: a compared image generating section 12b for generating compared image data (k<SB>n+1</SB>) indicating a post-positioning difference between the first high-resolution image data (h<SB>1→n</SB>) generated from n-sheets of picked-up image data, and the processing target image data (x<SB>n+1</SB>); and an image updating section 12c for generating image updated data by repeating operations, until the value of formula (8) obtained by adding the compared image data to a product of the image update data (g<SB>n+1</SB>) and a predetermined constant, the image update data (g<SB>n+1</SB>) being equivalent to the difference between the second high resolution image data (h<SB>1→n+1</SB>) generated from the n-sheets of picked-up image data and the processing target data and the first high-resolution image data, becomes smaller than a predetermined value, and by adding the generated image update data to the first high-resolution image data to generate the second high resolution image data. Thus, the controller 6 reduces the operation quantity required for ultra-resolution processing. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that generates a high resolution image from a plurality of low resolution images.

  In an automatic inspection device that automatically inspects a display device or the like, generally, an inspection image obtained by imaging an inspection object using an imaging device such as an area sensor is processed by a predetermined algorithm to detect defects. Etc. For this reason, in an automatic inspection device, the higher the inspection image resolution is, the more detailed inspection is possible. Therefore, a high-resolution imaging device capable of capturing an image with a high resolution is required.

  However, for example, when the inspection target is a liquid crystal display panel, the number of display pixels of the liquid crystal display panel has increased dramatically with the recent increase in the panel size of the liquid crystal display panel. However, it has become impossible to obtain sufficient resolution for inspection. Therefore, in the inspection apparatus, it is sufficient to increase the number of imaging devices and techniques for performing high resolution processing for generating one high resolution image from a plurality of low resolution images by super-resolution processing, image shift processing, etc. A technique for imaging at a high resolution is used.

  Compared with the latter method using a plurality of imaging devices, the former high resolution method has the following disadvantages: 1) It takes a lot of imaging time, and 2) Image processing for generating a high resolution image is necessary. However, there are advantages such as 1) the cost of the inspection apparatus is reduced, and 2) good maintainability.

  In recent years, high-resolution techniques have the merit that cost reduction is possible due to the increasing demand for cost reduction of inspection equipment installed in domestic factories and the need for cheaper inspection equipment for overseas deployment. Many studies have been conducted. The high resolution technique is a technique for generating one high resolution image from a plurality of low resolution images having a positional shift, and is roughly classified into two types, image shift processing and super-resolution processing.

  The image shift process is a technique for mapping a pixel corresponding to a pixel of a captured low resolution image to a high resolution image based on the luminance value of the pixel of the low resolution image in the generated high resolution image.

  On the other hand, the super-resolution processing is processing for estimating one high-resolution image from a plurality of low-resolution images. For example, the ML (Maximum-likelihood) method, the MAP (Maximum A Posterior) method, the POCS (Projection On to Various super-resolution processing methods such as the Convex Set method have been proposed.

  The ML method uses a square error between a pixel value of a low resolution image estimated from a high resolution image and an actually observed pixel value as an evaluation function, and estimates a high resolution image that minimizes this evaluation function. It is a method of making an image. That is, the ML method is a super-resolution processing method based on the principle of maximum likelihood estimation.

  The MAP method is a method for estimating a high-resolution image that minimizes an evaluation function obtained by adding probability information of a high-resolution image to a square error. That is, the ML method is a super-resolution processing method that estimates a high-resolution image as an optimization problem that maximizes the posterior probability by using some foresight information for the high-resolution image.

  The POCS method is a super-resolution processing method for obtaining a high-resolution image by creating simultaneous equations for pixel values of a high-resolution image and a low-resolution image and solving the equations sequentially.

  These super-resolution processes first assume a high-resolution image, and based on the point spread function (PSF function) obtained from the assumed high-resolution image from the camera model of the imaging device, the pixels of all the low-resolution images , The pixel value is estimated, and a process of searching for a high-resolution image that reduces the difference between the estimated value and the observed pixel value (observed value) is included. Therefore, these super-resolution processes are called reconfigurable super-resolution processing methods.

  In the image shift process, it is necessary to map all the pixels of the low-resolution image to the pixels of the high-resolution image. On the other hand, in the super-resolution processing, the imaging position of the low resolution image only needs to be known, and the number of low resolution images to be used is not limited.

  Since the pixel value estimation process performed in the super-resolution process is an iterative operation, the super-resolution process generally has a higher calculation cost than the image shift process, and the time required for the super-resolution process increases accordingly. End up. Therefore, it is required to reduce the amount of calculation required for the super-resolution processing and shorten the time required for the super-resolution processing. For example, in Patent Document 1 below, the calculation cost is reduced by reducing the number of low-resolution images used to generate a high-resolution image.

Specifically, in Patent Document 1, super-resolution processing for estimating a super-resolution image data sequence {z _k } from a low-resolution input image data sequence {L _k } at time k is performed. Here, in Patent Document 1, as the data used for the super-resolution processing for estimating the super-resolution image data sequence {z _k } at time k, a piece of low-resolution image data L _l (k−a) before and after time k. ≦ l ≦ k + a) and the result image data z _{k−1 of the} previous frame are used.

That is, in the conventional general super-resolution processing, a total of k low-resolution images from the first to the k-th are used for z _k estimation. By using the image data z _k-1 generated in 1, the number of low-resolution images to be used is reduced to the total (2a + 1) of (ka) to (k + a), thereby reducing the calculation cost. is doing. In this super-resolution processing method, a new high-resolution image generated from N low-resolution images, an M (N ≦ M) low-resolution image, and an L (L ≧ 1) high-resolution image Does not necessarily match the new high-resolution image generated from.
JP 2007-52672 A (published March 1, 2007)

  As described above, according to the super-resolution processing method of Patent Document 1, the number of images required for the super-resolution processing can be reduced, and the calculation cost can be reduced. However, in the super-resolution processing method described in Patent Document 1, the number of images used for the super-resolution processing is still large and the calculation amount is large.

  That is, in the super-resolution processing, if the number of low-resolution images used for processing increases, the amount of computation increases, and the time required for the super-resolution processing increases accordingly. When the number of captured images is reduced for the purpose of suppression, there is a problem that a high-resolution image having a desired restoration accuracy may not be obtained.

  The present invention has been made in view of the above problems, and an object of the present invention is to reduce the amount of calculation required for super-resolution processing and generate a high-resolution image having restoration accuracy equivalent to that of the conventional method. It is to realize an image processing apparatus and the like capable of performing the above.

  In order to solve the above problems, the image processing apparatus of the present invention generates high-resolution image data having a higher resolution than the image data based on a plurality of image data obtained by imaging one imaging target. In the image processing apparatus, the first high-resolution image data generated from the n pieces of image data and the processing target image data different from the image data obtained by imaging the imaging target are registered. This corresponds to a difference between the comparison image generation means for generating comparison image data indicating the difference, the second high resolution image data generated from the n pieces of image data and the processing target image data, and the first high resolution image data. By repeating the calculation until the value of the mathematical formula obtained by adding the comparison image data to the product of the image update data and a predetermined constant becomes smaller than the predetermined value. Image update data generating means for generating the image update data, and image update means for adding the generated image update data and the first high resolution image data to generate second high resolution image data. It is characterized by that.

  Further, in order to solve the above problems, the image processing method of the present invention is based on a plurality of image data obtained by imaging one imaging target, and high-resolution image data having a higher resolution than the image data. In the image processing method of the image processing apparatus for generating image data, first high-resolution image data generated from n pieces of image data, and processing target image data different from the image data obtained by imaging the imaging target A comparison image generation step for generating comparison image data indicating a difference after alignment, second high resolution image data generated from the n pieces of image data and processing target image data, and the first high resolution image data Until the value of the mathematical formula obtained by adding the comparison image data to the product of the image update data corresponding to the difference between and the predetermined constant becomes smaller than the predetermined value. An image update data generation step for generating the image update data by performing a calculation, and an image update for generating the second high-resolution image data by adding the generated image update data and the first high-resolution image data And a step.

  According to the above configuration, (n + 1) images obtained by adding the processing target image data to the n pieces of image data in the calculation for the two pieces of image data of the first high resolution image data and the processing target image data. Second high-resolution image data corresponding to the data can be obtained. As a result, the amount of calculation required for the super-resolution processing can be reduced, and as a result, the time required for the super-resolution processing can be shortened.

  In the above configuration, the first high-resolution image data that reflects n pieces of image data is used. Therefore, the image data generated using all of the (n + 1) pieces of image data in the calculation for the two pieces of image data. Second high-resolution image data having a resolution equivalent to the above can be generated.

  In the above configuration, the difference indicates a difference in luminance value between pixels corresponding to the position of the image data, in other words, the pixel indicating the same position of the imaging target. N is a natural number of 2 or more. As the above iterative calculation, for example, a known iterative calculation method such as a steepest descent method can be applied. Further, since the first high-resolution image data and the processing target image data have different image resolutions, a process for reducing the resolution of the first high-resolution image data may be performed at the time of alignment.

  Further, in order to solve the above-described problem, another image processing apparatus according to the present invention is based on a plurality of image data obtained by imaging one imaging target, and has a higher resolution than the image data. In an image processing device that generates image data, a difference after alignment between first high-resolution image data generated from n pieces of image data and processing target image data selected from the n pieces of image data Comparison image generation means for generating comparison image data indicating image update data corresponding to a difference between the second high resolution image data generated from image data other than the processing target image data and the first high resolution image data And the predetermined constant are added until the value of the mathematical formula obtained by adding the comparison image data becomes smaller than the predetermined value. Image update data generation means for generating data, and image update means for adding the generated image update data and the first high resolution image data to generate second high resolution image data It is said.

  According to the above configuration, (n−1) sheets obtained by removing the processing target image data from the n pieces of image data in the calculation for the two pieces of image data of the first high resolution image data and the processing target image data. Second high-resolution image data corresponding to the image data can be obtained. As a result, the amount of calculation required for the super-resolution processing can be reduced, and as a result, the time required for the super-resolution processing can be shortened.

  In addition, according to the above configuration, data inappropriate for super-resolution processing (for example, out-of-focus data) or the like can be removed from image data used for image processing by setting it as processing target image data. it can. This makes it possible to acquire the second high-resolution image data that reflects the image data more accurately.

  In order to solve the above-described problem, another image processing apparatus of the present invention is based on a plurality of image data obtained by imaging one imaging target, and has a higher resolution than the image data. In the image processing apparatus for generating resolution image data, first high-resolution image data generated from n pieces of image data and a first processing target image different from the image data obtained by imaging the imaging target The difference between the difference after the alignment with the data and the difference after the alignment between the first high-resolution image data and the second image data to be processed selected from the n pieces of image data is shown. Comparison image generating means for generating comparison image data, and a second high resolution generated from image data obtained by removing the second processing target image data from the n pieces of image data and adding the first processing target image data. The value of the mathematical formula formed by adding the comparison image data to the product of the image update data corresponding to the difference between the first image data and the first high-resolution image data and a predetermined constant is smaller than the predetermined value. The image update data generation means for generating the image update data by repeatedly performing the calculation until it becomes, and the generated image update data and the first high resolution image data are added to generate the second high resolution image data And image updating means.

  According to the above configuration, the second processing target image data is excluded from the n pieces of image data in the calculation for the three pieces of image data of the first high resolution image data and the first and second processing target image data. Second high-resolution image data corresponding to n pieces of image data to which the first processing target image data is added can be obtained. As a result, the amount of calculation required for the super-resolution processing can be reduced, and as a result, the time required for the super-resolution processing can be shortened.

  Moreover, according to said structure, while reflecting 1st process target image data, the 2nd high resolution image data which excluded the contribution of 2nd process target image data can be produced | generated. For example, out-of-focus image data or image data captured first (oldest) among n pieces of image data is set as the second processing target image data, and the latest captured image data is set as the first processing target. Image data may be used. As a result, the second high-resolution image data that always reflects the latest n pieces of image data can be generated.

  In order to solve the above problems, the inspection system of the present invention includes the image processing apparatus, an imaging apparatus that captures an imaging target and generates image data, and a second high resolution generated by the image processing apparatus. And an inspection apparatus for inspecting the defect of the imaging target based on image data.

  According to the above configuration, the image data generated by the imaging device is increased in resolution by the image processing device and second high-resolution image data is generated. Then, the inspection apparatus detects a defect to be imaged by analyzing the second high-resolution image data. As a result, even if the imaging apparatus does not have sufficient resolution for detecting the defect of the imaging target, it is possible to detect the defect of the imaging target with high accuracy.

  In the above inspection system, the image data generated by the imaging device is sequentially supplied to the image processing device, and the image processing device supplies the last (newest) supplied image data among the sequentially supplied image data. It is preferable that the high-resolution image data is sequentially updated using the data as processing target image data (or first processing target image data).

  According to the above configuration, the imaging of the imaging target and the generation of the second resolution-enhanced image data are performed in parallel. Therefore, a predetermined number of image data is reflected in the image processing apparatus as compared with the conventional method of generating high resolution image data reflecting all the image data after supplying the predetermined number of image data. The time until the second high-resolution image data is generated can be shortened.

  The inspection apparatus may be realized by a computer. In this case, a control program for realizing the inspection apparatus by the computer by operating the computer as each unit of the inspection apparatus, and recording the program Such computer-readable recording media also fall within the scope of the present invention.

  As described above, the image processing apparatus of the present invention is a processing target image different from the first high-resolution image data generated from n pieces of image data and the image data obtained by imaging the imaging target. Comparison image generating means for generating comparison image data indicating a difference after alignment with data, second high resolution image data generated from the n pieces of image data and processing target image data, and the first high resolution image The image is obtained by repeatedly calculating until the value of the mathematical formula obtained by adding the comparison image data to the product of the image update data corresponding to the difference from the data and a predetermined constant is smaller than a predetermined value. Image update data generating means for generating update data, and an image for generating second high resolution image data by adding the generated image update data and the first high resolution image data. A structure that a updating means.

  In addition, as described above, the image processing method of the present invention is different from the first high-resolution image data generated from n pieces of image data and the image data obtained by imaging the imaging target. A comparison image generating step for generating comparison image data indicating a difference after alignment with the target image data; second high-resolution image data generated from the n pieces of image data and the processing target image data; and the first high By repeating the calculation until the value of the mathematical formula obtained by adding the comparison image data to the product of the image update data corresponding to the difference from the resolution image data and a predetermined constant becomes smaller than the predetermined value. An image update data generation step for generating the image update data, and the generated image update data and the first high resolution image data are added to obtain the second high resolution image data. A configuration including an image update step of generating.

  Therefore, in the calculation for the two pieces of image data of the first high-resolution image data and the processing target image data, the first corresponding to (n + 1) pieces of image data obtained by adding the processing target image data to the n pieces of image data. 2 High-resolution image data can be obtained, and the amount of calculation required for super-resolution processing can be reduced.

  In the above configuration, the first high-resolution image data that reflects n pieces of image data is used. Therefore, the image data generated using all of the (n + 1) pieces of image data in the calculation for the two pieces of image data. Second high-resolution image data having a resolution equivalent to the above can be generated.

[Embodiment 1]
An embodiment of the present invention will be described below with reference to FIGS. First, the outline | summary of the test | inspection system of this embodiment is demonstrated based on FIG. The inspection system 1 uses the liquid crystal panel P as an inspection target, and includes a frame 2, a panel controller 3, an imaging device 4, a stage 5, a control device 6, and a monitor 7 as illustrated. Here, the vertical direction in FIG. 2 is the Z-axis direction, the horizontal direction is the X-axis direction, and the depth direction is the Y-axis direction.

  The liquid crystal panel P is a display panel that displays a predetermined image by turning on the backlight, switching the display pattern, and the like according to the control of the panel controller 3, and the display surface of the image faces the imaging device 4. As shown in FIG. That is, the inspection system 1 is a system that inspects whether or not the liquid crystal panel P is normally lit. Although not shown in the figure, the image display surface has a depth in the Y-axis direction and is parallel to the YZ plane. The inspection system 1 is not limited to the liquid crystal panel P, and can test various articles.

  The imaging device 4 is a device that images the liquid crystal panel P, and includes a housing 4a, a lens 4b, a solid-state imaging device 4c, and an actuator 4d. The lens 4b focuses light (image of the liquid crystal panel P) that enters the inside of the housing 4a on the imaging surface of the solid-state imaging device 4c. The solid-state imaging device 4c is composed of a plurality of imaging devices, samples the optically imaged liquid crystal panel P by spatially discretely sampling it, and converts these images into image signals. As the solid-state imaging element 4c, for example, a CCD (Charge Coupled Devices) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like can be applied. In the present embodiment, it is assumed that the imaging device 4 is an area sensor, but may be a line sensor.

  The actuator 4d changes the imaging position of the liquid crystal panel P by changing the position of the solid-state imaging device 4c. Specifically, the actuator 4d is fixed to the inner wall of the housing 4a, and the solid-state imaging device 4c is supported by the actuator 4d. Therefore, when the actuator 4d displaces the solid-state image sensor 4c, the solid-state image sensor 4c is displaced relative to the housing 4a, and the imaging position of the liquid crystal panel P is also displaced accordingly.

  Here, it is assumed that the actuator 4d moves the solid-state imaging device 4c two-dimensionally in a plane perpendicular to the optical axis connecting the liquid crystal panel P and the lens 4b. It may be a three-dimensional movement including As the actuator 4d, for example, a piezo actuator or a stepping motor can be used. In this embodiment, it is assumed that a piezo actuator is used.

  The imaging device 4 is fixed on the stage 5 so as to face the image display surface of the liquid crystal panel P. That is, in the present embodiment, the relative position between the imaging device 4 and the liquid crystal panel P is fixed, and it is assumed that an image of the liquid crystal panel P is captured by moving the solid-state imaging device 4c by the actuator 4d. Yes. Note that an image of the liquid crystal panel P may be captured while moving the stage 5 and / or the frame 2 while changing the relative position between the imaging device 4 and the liquid crystal panel P.

  The control device 6 includes an imaging control unit 11 that controls the imaging device 4, a captured image storage unit 13 that stores an image captured by the imaging device 4, and super-resolution that increases the resolution of image data captured by the imaging device 4. A processing unit 12 and the like are provided. That is, in the inspection system 1, the imaging device 4 images the liquid crystal panel P based on the control of the control device 6, and image data obtained by imaging the liquid crystal panel P is stored in the captured image storage unit 13 of the control device 6. The super-resolution processing unit 12 stores the super-resolution processing. Although not shown, the control device 6 has a configuration for performing a defect inspection of the liquid crystal panel using image data that has been increased in resolution by super-resolution processing.

  The monitor 7 displays a captured image acquired by the imaging device 4, a high-resolution image generated from the captured image, a defect inspection image inspected using these images, and the like. The monitor 7 may be any device that can display an image, and various display devices such as liquid crystal, organic EL, and CRT (Cathode Ray Tube) can be applied.

[Configuration of control device]
Then, the structure of the control apparatus 6 is demonstrated based on FIG. FIG. 3 is a block diagram showing a main configuration of the control device 6. As illustrated, the control device 6 includes an imaging control unit 11, a super-resolution processing unit (image processing device) 12, a captured image storage unit 13, a high-resolution image storage unit 14, and a defect inspection processing unit (defect inspection device) 15. A defect inspection image storage unit 16 and an image output control unit 17.

  The imaging control unit 11 instructs the imaging timing control unit 11a that controls the timing of imaging (exposure) by the solid-state imaging device 4c (imaging start time, length of exposure time, etc.) by sending a control signal to the imaging device 4. The imaging of the liquid crystal panel P is started at a predetermined timing. In the present embodiment, it is assumed that the imaging timing control unit 11a controls the imaging timing by changing the voltage of the signal sent to the imaging device 4. In addition, the imaging timing control unit 11a instructs the actuator control unit 11b that controls the movement start timing, the moving speed, the moving distance, and the like of the actuator 4d, and according to the exposure time of the solid-state imaging device 4c included in the imaging device 4. The solid-state image sensor 4c is moved.

  And the imaging control part 11 respond | corresponds to the imaging position (position of the solid-state image sensor 4c) when the captured image data of the liquid crystal panel P sampled by the solid-state image sensor 4c of the imaging device 4 is acquired. The attached captured image data 13 a is stored in the captured image storage unit 13.

  Although details will be described later, in the super-resolution processing, high-resolution image data is generated based on a plurality of captured image data in which the solid-state imaging device 4c is assigned to one display pixel of the liquid crystal panel P that is not an integer. For this reason, it is necessary to capture the liquid crystal panel P while shifting the position little by little by the actuator 4d to acquire a plurality of captured image data, and it is also necessary to acquire the imaging position of each captured image data.

  The super-resolution processing unit 12 performs super-resolution processing for generating high-resolution image data by increasing the resolution of the captured image data 13 a stored in the captured image storage unit 13. Specifically, the super-resolution processing unit 12 performs predetermined processing on the captured image data 13 a stored in the captured image storage unit 13 and the high-resolution image data 14 a stored in the high-resolution image storage unit 14. Thus, new high-resolution image data 14a is generated, and the generated high-resolution image data 14a is stored in the high-resolution image storage unit 14. Note that details of the super-resolution processing executed by the super-resolution processing unit 12 will be described in detail later.

  The defect inspection processing unit 15 acquires the high resolution image data 14a stored in the high resolution image storage unit 14, and performs a predetermined process on the acquired high resolution image data 14a to generate defect inspection image data. Then, the defect inspection processing unit 15 determines the type of defect of the liquid crystal panel P using the generated defect inspection image data.

  For example, the ferret diameter ratio can be used to determine the type of defect. That is, the defect of the liquid crystal panel P to be inspected includes a minute defect caused by a breakage of a color filter constituting the pixel and a pixel size defect caused by a failure of a circuit for driving the pixel. Here, the minute defect is observed in a small circle, whereas the defect having the pixel size is observed in the same size and elongated shape as the pixel. Therefore, it is possible to classify the defects according to the shape of the defect, and the ferret diameter ratio is used when performing the determination.

  When the defect type of the liquid crystal panel P can be determined, the defect inspection processing unit 15 sets, as the defect inspection image data 16a, data in which the defect inspection image data used for the determination is associated with the determination result of the defect type. It is stored in the defect inspection image storage unit 16. On the other hand, when the type of defect cannot be determined, the defect inspection processing unit 15 instructs the imaging control unit 11 to perform imaging of the liquid crystal panel P. As a result, the resolution of the high-resolution image data used for the defect inspection is further improved, and the defect of the liquid crystal panel P can be accurately detected. The details of the defect inspection process executed by the defect inspection processing unit 15 will be described later.

  The image output control unit 17 acquires the defect inspection image data 16 a stored in the defect inspection image storage unit 16 and displays the data indicated by the defect inspection image data 16 a on the monitor 7. Thereby, the user of the inspection system 1 can easily confirm the image indicating the defect of the liquid crystal panel P, the type of defect, and the like.

[Processing flow in inspection system]
Next, the flow of processing in the inspection system 1 having the above configuration will be described with reference to FIG. FIG. 4 is a flowchart illustrating an example of processing in the inspection system 1. First, when the liquid crystal panel P is fixed to the frame 2, the imaging device 4 performs imaging of the liquid crystal panel P in accordance with an instruction from the imaging control unit 11 of the control device 6 (S1). The captured image data 13a obtained by imaging is sequentially stored in the captured image storage unit 13 every time the imaging device 4 performs imaging (S2).

  Then, when the captured image data 13 a is newly stored in the captured image storage unit 13, the super-resolution processing unit 12 executes the super-resolution process, and when executing the super-resolution process, the super-resolution image storage unit 14 performs the super-resolution processing. New high-resolution image data 14a is generated based on the stored high-resolution image data 14a and the captured image data stored in the captured image storage unit 13 (S3). Subsequently, the super-resolution processing unit 12 stores the generated new high-resolution image data 14a in the high-resolution image storage unit 14 (S4).

  Here, an example will be described in which one piece of captured image data selected from the captured image data stored in the captured image storage unit 13 is used to generate new high-resolution image data 14a. One piece of captured image data (captured image data group) can also be used.

  Note that when the first captured image data 13a is stored in the captured image storage unit 13 after the start of imaging, the super-resolution processing unit 12 is a gray color stored in the high-resolution image storage unit 14 in advance. New high-resolution image data 14a is generated using arbitrary image data such as image data as high-resolution image data.

  When the defect inspection processing unit 15 confirms that the high resolution image data 14a is newly stored in the high resolution image storage unit 14, the defect inspection processing unit 15 generates defect inspection image data based on the high resolution image data 14a. A defect inspection process for analyzing the generated defect image data to determine the presence or absence of defects in the liquid crystal panel P is executed (S5). Subsequently, the defect inspection processing unit 15 determines whether or not the type of defect can be identified by the used defect inspection image data (S6).

  If the type of the defect can be identified (Yes in S6), the defect inspection processing unit 15 stores the defect inspection image as the defect inspection image data 16a associated with the detected defect type or the like. It stores in the part 16 (S7). Then, the defect inspection image data 16a stored in the defect inspection image storage unit 16 is displayed on the monitor 7 by the image output control unit 17 (S8).

  On the other hand, if the type of the defect is not identifiable (No in S6), the defect inspection processing unit 15 instructs the imaging processing unit to capture an image of the liquid crystal panel P. As a result, the captured image data is further added to the captured image data storage unit, and as a result, the super-resolution processing is executed again based on the added captured image data. Thus, the defect type is re-determined again based on the high-resolution image data 14a that has been further increased in resolution by reflecting the additional captured image data.

  Although not shown in the figure, if the type of defect cannot be specified even after repeating the loop of S1 to S6 a predetermined number of times, the defect inspection processing unit 15 indicates that the liquid crystal panel P has a defect. If it is not determined, the process is terminated.

  Here, when the number of captured image data is small, it is expected that high-resolution image data having a resolution sufficient for defect inspection cannot be obtained. Therefore, S1 to S4 are repeatedly executed until a predetermined number of captured image data are obtained, and after a predetermined number of captured image data is obtained, the process proceeds to S5 to perform defect inspection processing. May be. As a result, it is possible to reduce the time required for inspection by omitting the defect inspection processing of high-resolution image data that is expected not to have sufficient resolution for defect inspection.

[Purpose of super-resolution processing]
Before describing the super-resolution processing executed by the super-resolution processing unit 12, the purpose of performing the super-resolution processing in the inspection system 1 will be described here. First, a defect of the liquid crystal panel P detected by the inspection system 1 and a method for detecting the defect will be described with reference to FIG. FIG. 5 is a diagram illustrating an example of the arrangement of display pixels included in the liquid crystal panel P.

  As illustrated, the liquid crystal panel P includes, for example, a display pixel 21a composed of pixels of three colors, an R pixel 21b, a G pixel 21c, and a B pixel 21d. These RGB pixels are repeatedly arranged on the image display surface of the liquid crystal panel P at a constant cycle (for example, in the order of RGB-RGB-RGB...). In the present embodiment, the state in which the liquid crystal panel P has a defect refers to a state in which these RGB pixels are not normally lit.

  As a method for detecting a pixel that does not light normally, for example, a method of comparing a luminance difference between adjacent pixels of the same color with a predetermined threshold value can be cited. That is, when the pixels of the same color are normally lit, the luminance is almost equal. Therefore, if the luminance difference of the pixels of the same color exceeds the threshold value, it can be determined that a defect has occurred.

  In the illustrated example, the same color pixel adjacent to the R pixel 21b is the R pixel 21e. A defective pixel in which at least one of the R pixel 21a and the R pixel 21e is not normally lit when the luminance of the R pixel 21b and the luminance of the R pixel 21e are compared and the difference in luminance is larger than a predetermined threshold. Can be determined.

  It is assumed that the defect inspection processing unit 15 inspects all the pixels of the liquid crystal panel P using the captured image data obtained by the imaging device 4 imaging the liquid crystal panel P and detects defective pixels. Therefore, the captured image data needs to accurately reflect the value of the luminance value in each pixel of the liquid crystal panel P.

  Next, the conditions for the captured image data to accurately reflect the brightness value of each pixel of the liquid crystal panel P will be described with reference to FIG. 6A to 6C are diagrams illustrating a correspondence relationship between the pixels of the liquid crystal panel P and the solid-state imaging device 4c of the imaging device 4. In each of FIGS. 4A to 4C, each quadrangle indicated by a dotted line represents each image sensor constituting the solid-state image sensor 4c.

  In order for the captured image data to accurately reflect the value of the luminance value of each pixel of the liquid crystal panel P, RGB pixels are captured by different image sensors from among the plurality of image sensors that constitute the solid-state image sensor 4c. It is desirable. This is because, when a plurality of pixels are imaged by one image sensor, it is difficult to specify an accurate value of the luminance value of each pixel.

  In addition, since it is difficult to completely match the boundary of the image sensor and the boundary of the pixel, in order for the captured image data to accurately reflect the luminance value of each pixel of the liquid crystal panel P, RGB pixels It is desirable to assign a plurality of image sensors to one pixel in consideration of the positional deviation between the image sensor and the image sensor.

  FIG. 4A shows a state in which image sensors sufficient for defect detection are assigned in consideration of positional deviation. As shown in the drawing, at least 6 × 6 imaging elements are assigned to one display pixel of the liquid crystal panel P. Thereby, sufficient image sensors are assigned to each pixel, and captured image data that accurately reflects the luminance value of each pixel of the liquid crystal panel P can be obtained.

  However, in practice, it is difficult to realize the allocation of the imaging elements as shown in FIG. 5A due to problems such as the number of display pixels of the liquid crystal panel P and the resolution of the imaging device 4.

  FIG. 4B shows the pixels of the liquid crystal panel P when the liquid crystal panel P is imaged by the imaging device 4 using a 11 million pixel CCD sensor when the number of display pixels of the liquid crystal panel P is about 1920 × 1080. And the solid-state imaging device 4c of the imaging device 4 are shown. As shown in the figure, the allocation of the image sensor to one display pixel is about 2 × 2, and under this condition, captured image data that accurately reflects the luminance value of each pixel of the liquid crystal panel P is obtained. Is difficult to understand.

  FIG. 5C shows the liquid crystal panel P when the liquid crystal panel P is imaged by the imaging device 4 using a 16 million pixel CCD sensor when the number of display pixels of the liquid crystal panel P is about 1920 × 1080. The correspondence relationship between the pixel and the solid-state imaging device 4c of the imaging device 4 is shown. As shown in the figure, an imaging element of about 2.3 × 2.3 is assigned to one display element. As described above, even when the resolution of the imaging device 4 is improved as compared with FIG. 5B, it is possible to obtain captured image data that accurately reflects the luminance value of each pixel of the liquid crystal panel P. I find it difficult.

  As described above, with the method for improving the resolution of the imaging device 4, it is difficult to obtain captured image data that accurately reflects the luminance value of each pixel of the liquid crystal panel P. Therefore, by performing super-resolution processing on captured image data in which the allocation of imaging elements to display pixels is not sufficient as shown in FIGS. 10B and 10C, for example, a high level as shown in FIG. It is necessary to generate a resolution image.

[Principle of super-resolution processing executed by the super-resolution processor]
Next, the principle of super-resolution processing in the super-resolution processing unit 12 will be described. As described in [Background Art], the super-resolution processing is formulated as an optimization problem of an evaluation function defined for a generated high-resolution image. That is, the reconstruction type super-resolution processing is reduced to an optimization problem of an evaluation function (error term) based on a square error between the “estimated low-resolution image” and the “observed observed image”.

Here, first, consider a case where the number of captured image data stored in the captured image storage unit 13 is n. In this case, the high-resolution evaluation function used for the super-resolution processing is expressed by, for example, the following formula (1), and the super-resolution processing is expressed as processing for minimizing the evaluation function I _{1 → n} .

In the above formula (1), h _{1 → n} represents high-resolution image data as a vector. Further, y _{1 → n} represents all captured image data from the first to the n-th as a vector. y _{1 → n} is expressed by the following formula (2), and in formula (2), x _i (i = 1 to n) represents each captured image data as a vector.

B _{1 → n} is a method of aligning high-resolution image data with the imaging position of each captured image data, and then convolving a kernel of a point spread function (PSF function) obtained from the camera model of the imaging device 4 to reduce the resolution. This processing is expressed as a matrix, and is expressed by the following mathematical formula (3). In Formula (3), A _i (i = 1 to n) is a conversion from high-resolution image data to individual captured image data, that is, the position of the high-resolution image data h _{1 → n} with respect to the captured image data x _i . It is a matrix showing the process of combining and lowering the resolution.

  In Equation (1), C represents a matrix representing prior probability information indicating the characteristics of the high resolution image data, and α represents a constraint parameter representing the strength of the constraint. 1 to n are numbers representing each of n pieces of captured image data, and T represents transposition of a matrix.

The “estimated low-resolution image” refers to an image obtained by multiplying the estimated high-resolution image data by an optical degradation function to reduce the resolution. In this example, the estimated high resolution image data is h _{1 → n} in the following formula (1), and the optical degradation function is A _n (component of B _{1 → n} ). “Image” corresponds to A _n × h _{1 → n} . The “observed observed image” corresponds to x _n (component of y _{1 → n} ) of the same equation.

Here, the process of minimizing the evaluation function of the formula (1) corresponds to the process of setting the gradient of the evaluation function I _{1 → n} to h _{1 → n} to 0. This process is represented by the following mathematical formula (4). That is, when the captured image data is n pieces of up to _x 1 ~x _n is, n pieces of captured image of h _{1 → n} as a value close to 0 in Equation (4) to _x 1 ~x _n A high-resolution image based on the data.

On the other hand, when the captured image data x _{n + 1} is newly stored in the captured image storage unit 13 and the captured image data stored in the captured image storage unit 13 is _{n + 1} from x ₁ to x _{n + 1} , this super The resolution process is expressed as a process for minimizing the evaluation function expressed by the following mathematical formula (5).

As in the case of n resolution images x ₁ to x _n , the process of minimizing the evaluation function represented by the equation (5) is to obtain the gradient of the evaluation function I _{1 → n + 1} with respect to h _{1 → n + 1} . This process corresponds to the process of setting the gradient to 0. This process is expressed by the following formula (6). Therefore, when the number of captured image data is _{n + 1} from x ₁ to x _{n + 1} , the process of obtaining the high resolution image h _{1 → n + 1} by bringing the following formula (6) close to zero corresponds to the super-resolution process. become.

When this mathematical formula (6) is modified, the following mathematical formula (7) is obtained. Then, taking the difference between the above formula (4) and the above formula (7), the formula shown in the first line of the following formula (8) is obtained, and this formula is _expressed as (h _{1 → n + 1} −h _{1 → n} ). When arranged, the mathematical formula shown in the second line of the following mathematical formula (8) is obtained. Furthermore, in the mathematical formula shown in the second line of the mathematical formula (8), by placing (h _{1 → n + 1} −h _{1 → n} = g _{n + 1} ), the mathematical formula (8) = f (g _{n + 1} ) is expressed (math formula) (8) third line).

Equation (8), that is, g _{n + 1} that brings f (g _{n + 1} ) close to zero, can be asymptotically obtained by the following equation (9). As described above, since g _{n + 1} is the difference between the high resolution image data h _{1 → n + 1} and h _{1 → n} , the high resolution image data h _{1 → n} is added to the obtained g _{n + 1} to obtain a high resolution image. Data h _{1 → n + 1} can be obtained. Note that in equation (9), g _{n, k} is a vector representing the k-th value of the image update data g _n in the repeating operation. Α in Expression (9) controls the convergence speed in the steepest descent method. Α in Equation (9) may be the same value as α in Equations (1) and (4) to (8), or may be a different value.

Super-resolution processing unit 12, based on the principle described _above, the high-resolution image data h _{1 → n} generated from n pieces of captured image data to the _{x 1 ~x n, x 1 ~x} n + 1 to n + 1 The high resolution image data h _{1 → n + 1} corresponding to the captured image data of the sheet is obtained. A more detailed configuration of the super-resolution processing unit 12 and specific processing contents of the super-resolution processing executed by the super-resolution processing unit 12 will be described with reference to FIGS. 1, 7, and 8.

[Details of super-resolution processing unit]
FIG. 1 is a block diagram illustrating a main configuration of the super-resolution processing unit 12. As illustrated, the super-resolution processing unit 12 includes an image registration unit 12a, a comparison image generation unit (comparison image generation unit) 12b, and an image update unit (image update data generation unit, image update unit) 12c. I have.

  The image alignment unit 12a performs alignment between the captured image data and the high resolution image data. Then, the image alignment unit 12a artificially lowers the resolution of the high-resolution image data after alignment.

  Specifically, the image alignment unit 12a first calculates the relative position of the captured image data with respect to the high resolution image data. Here, in the inspection system 1, as described above, the imaging position of the captured image data is acquired by the imaging control unit 11, and the captured image data is stored in the captured image storage unit 13 as captured image data 13a associated with the imaging position. It is assumed that it is stored. Therefore, the image alignment unit 12a can calculate the relative position of the captured image data with respect to the high resolution image data based on the captured image data 13a stored in the captured image storage unit 13. When the imaging position of the captured image data is not known, the relative position of the captured image data with respect to the high resolution image data may be detected by a registration method based on pattern matching or luminance difference.

Then, the image alignment unit 12a aligns the high resolution image data with the imaging position of each captured image data based on the relative position of the captured image data with respect to the calculated high resolution image data, and then A matrix B _{1 → n} indicating a process of reducing the resolution by convolving a kernel of a point spread function (PSF function) obtained from the camera model is calculated. The image alignment unit 12a outputs the matrix B _{1 → n} thus calculated to the comparison image generation unit 12b.

When the comparison image generation unit 12b receives the matrix B _{1 → n} , the comparison image generation unit 12b calculates a difference by multiplying each of the high resolution image data h _{1 → n} and the captured image data x _{n + 1} by the weight, and generates comparison image data k _{n + 1} . To do. The comparison image data is data represented by the following mathematical formula (10).

Note that the weights multiplied by the high-resolution image data h _{1 → n} and the weights multiplied by the captured image data x _{n + 1} can be calculated from the received matrix B _{1 → n} . Further, x _{n + 1} and h _{1 → n} can be read from the captured image storage unit 13 and the high-resolution image storage unit 14, respectively. Therefore, the comparison image generation unit 12b can calculate the value of the comparison image data kn _{+ 1} as a numerical value. The comparison image generation unit 12b outputs the value calculated in this way to the image update unit 12c.

Here, as can be seen from a comparison between Equation (8) and Equation (10), Equation (10) is related to (h _{1 → n + 1} −h _{1 → n} ) in the second line of Equation (8). It is a constant term that is not. That is, Equation (8) can be expressed as _{(h 1 → n + 1 -h} 1 → n) and by placing a _{g n + 1, f (g} n + 1) = β × g n + 1 + k n + 1. That is, the comparative image generating unit 12b performs a process for obtaining a constant term of f (gn _{+ 1} ).

Β is a coefficient of (h _{1 → n + 1} −h _{1 → n} ) in the second row of the formula (8). Here, as shown in Equation (8), β is represented by B _{1 → n} , A _i , C, and α. As described above, B _{1 → n} represents processing for reducing the resolution after aligning the high-resolution image data with the imaging position of each captured image data as a matrix, and A _i (i = 1 to 1). n) is a matrix representing processing for aligning and reducing the resolution of the high-resolution image data h _{1 → n} with respect to the captured image data x _i , and C is a matrix representing prior probability information indicating the characteristics of the high-resolution image data Α represents a constraint parameter representing the strength of the constraint. Since these parameters are constants determined by the conditions for performing the super-resolution processing, the value of β is also determined in advance according to the conditions for performing the super-resolution processing.

The image update unit 12c uses the B _{1 → n} received from the image alignment unit 12a and the comparison image data k _{n + 1} received from the comparison image generation unit 12b, and the absolute value of Equation (8) is a predetermined value. The image update data g _{n + 1} is calculated by performing the repetitive calculation represented by the equation (9) until it becomes smaller. Then, the image update unit 12c changes the image update data g _{n + 1} when the absolute value of the mathematical formula (8), that is, f (g _{n + 1} ) is smaller than a predetermined value, to the high resolution image data h _{1 → n} . Addition to generate new high-resolution image data h _{1 → n + 1} .

That is, in the super-resolution processing unit 12, an absolute value of a mathematical expression f (g _{n + 1} ) obtained by adding the comparison image data k _{n + 1} to the product of the image update data g _{n + 1} and a predetermined constant β is predetermined. The image update data g _{n + 1} is obtained by iteratively calculating until it becomes smaller than the value, and the obtained image update data g _{n + 1} is added to the high resolution image data h _{1 → n} to obtain new high resolution image data h _{1 → n + 1} is generated.

  The predetermined value can be changed as appropriate according to the required accuracy of resolution enhancement. For example, when higher resolution and higher resolution are required, the predetermined value may be set to a small value, and high resolution accuracy is not required so much, but quick processing speed is required. May be set larger than the predetermined value. That is, the predetermined value is a value for determining the accuracy of the super-resolution processing, and can be appropriately set according to various conditions required for the super-resolution processing.

Next, the flow of super-resolution processing in the super-resolution processing unit 12 will be described with reference to FIG. FIG. 7 is a flowchart illustrating an example of super-resolution processing. The flowchart of FIG. 7, the process from the state high-resolution image data h _{1 → n} generated based on the n pieces of captured image data to the x ₁ ~x _n is stored in the high-resolution image storage section 14 Shows the flow.

Here, when x _{n + 1} which is the ( _{n + 1) th} captured image data is stored in the captured image storage unit 13, the image alignment unit 12a acquires the captured image data x _{n + 1} from the captured image storage unit 13 (S11). The high-resolution image data h _{1 → n} is acquired from the high-resolution image storage unit 14 (S12). Note that the order of S11 and S12 may be switched or may be performed simultaneously.

Subsequently, the image alignment unit 12a aligns the position of the acquired high-resolution image data h _{1 → n with} the captured image data x _{n + 1} and matrixes for artificially reducing the resolution of the high-resolution image data h _{1 → n.} B _{1 → n} is generated (S13), and the generated matrix B _{1 → n} is sent to the comparison image generation unit 12b.

Next, when the comparison image generation unit 12b receives the matrix B _{1 → n} , the comparison image generation unit 12b acquires the captured image data x _{n + 1} from the captured image storage unit 13 and the high resolution image data h _{1 → n} from the high resolution image storage unit 14. To get. Then, the comparison image generation unit 12b multiplies the captured image data x _{n + 1} and the high resolution image data h _{1 → n} by the weight calculated from the matrix B _{1 → n} , respectively, and the comparison image data k _{n + 1} represented by Expression (10). Is generated (S13). Then, the comparison image generation unit 12b sends the value of the generated comparison image data kn _{+ 1} to the image update unit 12c.

Image update section 12c having received the value of the comparison image data _{k n + 1} is, B _{1 → n} and the value of the comparison image data _{k n + 1} are substituted into Equation (8) to calculate the value of the image update data _{g n + 1} (S15) . Subsequently, the image update unit 12c determines whether or not the absolute value of f (g _{n + 1} ) is smaller than a predetermined value (S16). When the absolute value of f (g _{n + 1} ) is equal to or greater than a predetermined value (No in S16), the image update unit 12c performs calculation using Equation (9) to update the image update data g _{n + 1} . Then (S17), it is determined again in S16 whether or not the absolute value of f (gn _{+ 1} ) is smaller than a predetermined value.

On the other hand, when it is determined that the absolute value of f (g _{n + 1} ) is smaller than the predetermined value (Yes in S16), the image update unit 12c stores the image update data g _{n + 1} in the high resolution image storage unit 14. is added to the high-resolution image data h _{1 → n} being to generate a high-resolution image data new _{h 1 → n + 1 (S18} ), the high resolution image data and the generated h _{1 → n + 1} high-resolution image storage unit 14 To store.

  In the above example, the example in which the repetitive calculation is performed by the steepest descent method using the formula (9) is shown. However, the repetitive calculation is not limited to the steepest descent method, and a known calculation method may be appropriately used. it can.

[Advantages of super-resolution processing of this embodiment]
As described above, the super-resolution processing unit 12 is based on the high-resolution image data h _{1 → n} generated from the first to n-th captured image data and the newly added captured image data x _{n + 1.} Super-resolution processing for generating new high-resolution image data h _{1 → n + 1} corresponding to the first to n + 1-th captured image data is performed. This is illustrated in the drawing as shown in FIG.

FIG. 8 is a diagram illustrating the relationship between captured image data and high-resolution image data. As illustrated, the high-resolution image data corresponding to the first to n-th captured image data is the high-resolution image data h _{1 → n} , and the captured image data corresponding to the first to n + 1-th captured image data. To high resolution image data h _{1 → n + 1} .

Here, in the conventional super-resolution processing, when the captured image data is n + 1 from the first to the (n + 1) th, for example, a high-resolution evaluation function as shown in Equation (1) is used, and the first to the first. The high resolution image data h _{1 → n + 1} has been obtained using all or part of the captured image data up to _{n + 1} .

Therefore, it is necessary to perform an operation for the total number of pixels of all captured image data used for the super-resolution processing as an error term operation for each repetitive operation, and there is a problem that the calculation cost is high. Further, in the conventional super-resolution processing, the procedure for obtaining the high-resolution image data h _{1 → n + 1} in a state where all of the first to n + 1-th captured image data are prepared has been taken. Therefore, the n + 1-th captured image data The super-resolution processing was not performed until was acquired.

Therefore, in the present embodiment, Equation (4) is obtained from Equation (1), which is a high-resolution evaluation function for obtaining high-resolution image data h _{1 → n} from the first to n-th captured image data. Similarly, Equations (5) to (6), which are high-resolution evaluation functions for obtaining high-resolution image data h _{1 → n + 1} from the first to (n + 1) -th captured image data, are obtained.

Here, as described above, Expression (4) is an expression for obtaining the high-resolution image data h _{1 → n} from the first to n-th captured image data, and Expression (6) is the expression from the first to the (n + 1) th. This is a mathematical expression for obtaining high-resolution image data h _{1 → n + 1} from captured image data. Therefore, the mathematical formula derived from the difference between the mathematical formula (4) and the mathematical formula (6) is the image update data g _{n + 1} which is the difference between the high resolution image data h _{1 → n + 1} and h _{1 → n} and the high resolution image data h. Equation (8) expressed by comparison image data kn _{+ 1} which is a difference between _{1 → n} and x _{1 + n} .

In the super-resolution processing of the present embodiment, since this equation (8) is used, the first to n + 1th using one piece of high-resolution image data h _{1 → n} and one piece of captured image data x _{n + 1.} High resolution image h _{1 → n + 1} data corresponding to the captured image data up to can be obtained.

Specifically, in the super-resolution processing of the present embodiment, comparison image data kn _{+ 1} that is a difference after alignment between the high-resolution image data h _{1 → n} and x _{1 + n} is obtained. Then, by substituting the obtained comparison image data k _{n + 1} into the equation (8), the equation (8) has a function f (g) with the image update data g _{n + 1} as a variable and the comparison image data k _{n + 1} as a constant term. _{n + 1} ).

By using this function f (g _{n + 1} ), the calculation of the error term necessary for each repetitive calculation is the calculation for the total number of pixels of one piece of high resolution image data h _{1 → n} . Therefore, when the total number of pixels of all captured image data used for the super-resolution processing is larger than the total number of pixels of the high-resolution image data h _{1 → n} , the calculation cost of the super-resolution processing can be reduced.

Further, in the super-resolution processing of the present embodiment, the image update data g _{n + 1} for updating the high-resolution image data h _{1 → n} is obtained by repetitive calculation. Accordingly, when the high-resolution image data h _{1 → n} is hardly updated by the newly added captured image data x _{n + 1, that} is, the high-resolution image data h _{1 → n} generated from _n captured image data and n + 1 sheets. When the high-resolution image data h _{1 → n + 1} generated from the captured image data is substantially equal, the number of iterations is greatly reduced.

For example, when the resolution of captured image data obtained by capturing an object that does not move, such as the liquid crystal panel P that is the inspection target of the present embodiment, is increased, the high-resolution image data h _{1 → n} And the high-resolution image data h _{1 → n + 1} are small, and the number of repeated operations can be reduced.

In addition, the super-resolution processing of the present embodiment, in which the number of repetitive calculations is smaller as the difference between the high-resolution image data h _{1 → n} and the high-resolution image data h _{1 → n + 1} is smaller, This is particularly suitable when data is sequentially added in time series.

  The reason for this is that as the number of captured image data increases to some extent, the width of change in high-resolution image data decreases. That is, in the super-resolution processing of this embodiment, the super-resolution processing is executed every time the captured image data is sequentially added in time series. Therefore, the captured image used for the super-resolution processing at the start of the super-resolution processing. Since the number of data is small, the number of repeated operations is large. However, the number of repeated operations decreases as the super-resolution processing ends. In the super-resolution process of the present embodiment, the captured image data acquisition and the super-resolution process are performed in parallel. As a result, the time from the start of imaging to the end of super-resolution processing can be reduced.

[Processing in the defect inspection processing section]
Next, details of processing in the defect inspection processing unit 15 will be described with reference to FIG. FIG. 9 is a block diagram showing a main configuration of the defect inspection processing unit 15. As illustrated, the defect inspection processing unit 15 includes a contrast image generation unit 15a, a binarized image generation unit 15b, and a defect type determination unit 15c.

  The contrast image generation unit 15 a generates contrast image data from the high resolution image data acquired from the high resolution image storage unit 14. The contrast image data is image data for comparing the processed image data (here, high-resolution image data) and the reference image data, and the brightness value of each pixel of the processed image data and the reference image data. It is data indicating a difference. Since the liquid crystal panel P to be inspected in this embodiment has a repeating pattern of pixels having the same shape, the luminance value of the pixel and the same color adjacent to the pixel are compared with a certain pixel in the processed image data. An average value with the luminance value of the pixel can be regarded as the luminance value of the reference image data. Therefore, the luminance value of each pixel in the contrast image data can be calculated by, for example, a method of calculating a difference in luminance value between adjacent pixels of the same color.

  The binarized image generation unit 15b binarizes the contrast image data generated by the contrast image generation unit 15a using a predetermined threshold value, and generates binarized image data. The predetermined threshold value may be calculated from a luminance deviation or the like in the entire captured image data.

  The defect type determination unit 15c determines the type of defect based on the size of the defect portion in the binarized image data, the ferret diameter ratio, and the like. Then, the defect type determination unit 15c converts the defect inspection image data 16a, which is a combination of the high-resolution image data to be processed, the defect type, and the feature amount of the defect such as the ferret diameter ratio, to the defect inspection image storage unit. 16.

  An example of processing executed by the defect inspection processing unit 15 having the above configuration will be described below. First, when new high resolution image data is added to the high resolution image storage unit 14, the contrast image generation unit 15 a acquires the added high resolution image data from the high resolution image storage unit 14. Then, the contrast image generation unit 15a generates contrast image data from the acquired high-resolution image data, and sends the generated contrast image data to the binarized image generation unit 15b.

  Next, the binarized image generation unit 15b that has received the contrast image data binarizes the received contrast image data to generate binarized image data, and the generated binarized image data is used as the defect type determination unit 15c. Send to.

  Then, the defect type determination unit 15c that has received the binarized image data determines the type of the defect based on the received binarized data. If the defect type cannot be determined, the defect type determination unit 15c instructs the imaging control unit 11 to further capture an image of the liquid crystal panel P. On the other hand, when the defect type can be determined, the defect inspection image data 16a obtained by combining the determined defect type, feature amount, position, and the like data with the high-resolution image data to be processed is used as the defect inspection image. Store in the storage unit 16.

  As described above, in the inspection system 1 of the present embodiment, the captured image data obtained by imaging the liquid crystal panel P is increased in resolution by the super-resolution processing unit 12, and the captured image data is converted into high-resolution image data that is increased in resolution. Based on this, the defect inspection of the liquid crystal panel P is performed. Therefore, the defect inspection can be accurately performed based on the image data having a resolution sufficient for the defect inspection of the liquid crystal panel P.

  Further, in the inspection system 1, super-resolution processing is executed every time captured image data is added. Therefore, imaging is performed in comparison with the conventional method in which super-resolution processing is performed after all the imaging of the liquid crystal panel P is completed. The time from the start to the end of the super-resolution processing can be shortened.

  Further, in the inspection system 1, when the type of defect cannot be determined using the generated high resolution image data, the liquid crystal panel P is automatically imaged and the captured image data is added. This makes it possible to accurately inspect the defect by increasing the resolution of the high-resolution image data to a level necessary for defect determination.

[Embodiment 2]
In the above embodiment, from the first to _{n + 1} based on the high-resolution image data h _{1 → n} corresponding to the first to n-th captured image data and the newly added n + 1-th captured image data x n + 1. The mode of obtaining the high-resolution image data h _{1 → n + 1} corresponding to the _first captured image data is shown.

In the present embodiment, after generating high-resolution image data h _{1 → n} corresponding to the first to n-th captured image data, any one image is excluded from the first to n-th captured image data. The high-resolution image data h _{1 → n−1} is generated from the n−1 pieces of captured image data. In other words, in this embodiment, the high resolution image data h _{1 → n,} to generate a high-resolution image data h 1 _{→ n-1} which eliminated the information related to any one captured image data.

As a result, data inappropriate for super-resolution processing (for example, out-of-focus data), first high-resolution image data h ₁ that does not reflect captured image data, and the like are used for super-resolution processing. It can be removed from the captured image data, and a more accurate high-resolution image can be acquired.

This embodiment will be described in more detail based on FIG. In addition, about the structure same as the said embodiment, the same reference number is attached | subjected and the description is abbreviate | omitted. FIG. 10 is a diagram illustrating the relationship between captured image data and high-resolution image data. As illustrated, the high-resolution image data corresponding to the first to n-th captured image data is the high-resolution image data h _{1 → n} , and the imaging corresponding to the first to n−1-th captured image data. From the image data to the high resolution image data h _{1 → n−1} .

In the inspection system 1 of the present embodiment, the super-resolution processing unit 12 of the control device 6 includes one piece of high-resolution image data h _{1 → n} and one piece of captured image data x _n (removed target captured image data). obtaining a comparison image data k _n with. The super-resolution processing unit 12 obtains the high resolution image data h _{1 → n-1} on the basis of the comparison image data k _n and the image update data g _n obtained above.

That is, in the super-resolution processing unit 12 of the present embodiment, data to be input is high resolution image data h _{1 → n} and the image data x _n, and the therefore the comparison image data is k _n, and the image update data g _n Except for this point, the same processing as in the above embodiment is performed. Thus, here, first formula is a difference from the above embodiment and the present embodiment, i.e., the process of obtaining a comparison image data k _n and the image update data g _n will be described.

[Process for obtaining a comparison image data k _n and image update data g _n]
First, the process of calculating the high-resolution image data h _{1 → n} is a process of bringing the above equation (4) close to zero, as described above in “Principle of Super-Resolution Processing Performed by Super-Resolution Processing Unit 12”. It corresponds to. Similarly, the process of calculating the high-resolution image data h _{1 → n−1} is expressed as a process of bringing the following formula (11) close to zero.

When the difference between Expression (4) and Expression (11) is calculated, the following Expression (12) is obtained. The f (g _n) process close Equation (12) to zero which is expressed by a super-resolution processing in the embodiment.

Similar to the above embodiment, since the term is not in the above equation (12) _{(h 1 → n -h 1 →} n-1) Kakari' is compared image data _{k n, k n} is the following equation (13 ).

[Flow of super-resolution processing]
Next, the flow of super-resolution processing according to this embodiment will be described. In the super-resolution processing of the present embodiment, first, the image alignment unit 12a aligns the high-resolution image data h _{1 → n} with the captured image data x _n to be removed, and the matrix B ₁ for reducing the resolution. _{→ n} is obtained and sent to the image updating unit 12c. The imaged image data _{xn to be} removed is specified by the user of the inspection system 1 performing a predetermined input operation on the input unit provided in the control device 6, and is limited to the nth imaged image data. Absent.

Then, comparing the image generating unit 12b for receiving the matrix B _{1 → n} is the comparative image data k _n represented by the formula (13), x _n obtained respectively from the captured image storage unit 13 and the high-resolution image storage unit 14 , H _{1 → n} and the weight calculated from the matrix B _{1 → n} are substituted to calculate the value of Equation (13). The comparison image generating section 12b, the value of equation (13) calculated above, i.e. sends the values of the comparison image data k _n to the image update unit 12c.

Image update section 12c having received the comparison image data k _n is a k _n Equation (12), i.e. substituted into f (g _n), performs the repetitive operations until its absolute value is smaller than a predetermined value . Then, the image updating unit 12c subtracts f the image update data g _n when the absolute value is smaller than a predetermined value of (g _n) from high resolution image data h _{1 → n,} new The high resolution image data h _{1 → n−1} is generated.

By the above process, the super-resolution processing unit 12 of the present embodiment, h _{1 → n-1} from the high-resolution image data h _{1 → n} is a novel high-resolution image data which eliminated the information of the captured image data x _n Can be generated.

[Embodiment 3]
In the present embodiment, when the (n + 1) th captured image data is stored, the super-resolution processing is performed based on the second to nth captured image data. In other words, in the present embodiment, when generating the high resolution image data to the new, the addition of the newest image data x _{n + 1,} forgetting oldest captured image data x _1.

  As a result, among the captured image data used for super-resolution processing, the oldest captured image data is forgotten, that is, eliminated, so that it is possible to obtain high-resolution image data that more strongly reflects the newest captured image data. become.

This embodiment will be described in more detail based on FIG. In addition, about the structure same as the said embodiment, the same reference number is attached | subjected and the description is abbreviate | omitted. FIG. 11 is a diagram illustrating the relationship between captured image data and high-resolution image data. As illustrated, the high-resolution image data corresponding to the first to n-th captured image data is the high-resolution image data h _{1 → n} , and the captured image data corresponding to the second to n + 1-th captured image data. To high resolution image data h _{2 → n + 1} .

In the inspection system 1 of the present embodiment, the super-resolution processing unit 12 of the control device 6 includes one piece of high-resolution image data h _{1 → n} , one piece of captured image data x _{n + 1} , and first captured image data. The comparison image data k _n ′ is obtained using x ₁ (the captured image data to be removed). Then, the super-resolution processor 12 obtains high-resolution image data h _{2 → n + 1} based on the obtained comparison image data k _n ′ and image update data g _{n + 1} ′.

That is, in the present embodiment, the comparison image data k _n ′ and the image update data g _{n + 1} ′ are different from the above embodiment. A process for obtaining these mathematical expressions will be described below.

[Process for obtaining comparison image data k _n ′ and image update data g _{n + 1} ′]
First, the process of calculating the high-resolution image data h _{1 → n} is a process of bringing the above equation (4) close to zero, as described above in [Principle of Super-Resolution Processing Performed by Super-Resolution Processing Unit]. Equivalent to. Similarly, the process of calculating the high-resolution image data h _{2 → n + 1} is expressed as a process of bringing the following formula (14) close to zero.

When the difference between Expression (4) and Expression (14) is calculated, the following Expression (15) is obtained. The process of bringing the mathematical expression (15) represented by f (gn _{+ 1} ′) close to zero is the super-resolution process in the present embodiment.

Then, as in the above embodiment, since the above equation (15) _{(h 2 → n + 1 -h} 1 → n) is not Kakari' claim a comparison image data _{k n,} comparing the image data _{k n} 'is below The following equation (16) is obtained. As can be seen from the development of Equation (16), the comparison image data k _n ′ includes the difference after the alignment between the captured image data x _{n + 1} and the high resolution image data h _{1 → n} , the captured image data x ₁ and the high And the difference after the alignment with the resolution image data h _{1 → n} .

[Flow of super-resolution processing]
Next, the flow of super-resolution processing according to this embodiment will be described. In the super-resolution processing according to the present embodiment, when the captured image data _{xn + 1} is stored in the captured image storage unit 13, the image alignment unit 12a captures the newly added high-resolution image data h1 _{→ n.} The matrix B _{1 → n} for aligning with the data x _{n + 1} and reducing the resolution is obtained and sent to the comparison image generating unit 12b.

Subsequently, the comparison image generating unit 12b that has received the matrix B _{1 → n} has the oldest (first stored) high-resolution image data h _{1 → n in the} comparison image data k _n ′ represented by Expression (16). The value of the comparison image data k _n ′ is calculated by substituting the captured image data x ₁ and the weight obtained from the matrix B _{1 → n} . Then, the comparison image generation unit 12b sends the value of the generated comparison image data k _n ′ to the image update unit 12c.

The image updating unit 12c that has received the value of the comparison image data k _n ′ repeatedly performs calculation until the absolute value of the mathematical formula (15), that is, f (g _{n + 1} ′) is smaller than a predetermined value, and updates the image. Data g _{n + 1} ′ is calculated. Specifically, when the absolute value of f (g _{n + 1} ′) is larger than a predetermined value, the image update unit 12c updates the value of g _{n + 1} ′ by the steepest descent method, and f (g _{n + 1} The image update data g _{n + 1} 'when the absolute value of') is smaller than a predetermined value is added to the high resolution image data h _{1 → n} to obtain new high resolution image data h _{2 → n + 1.} Is generated.

  With the above processing, the super-resolution processing unit 12 according to the present embodiment simultaneously adds new captured image data and excludes captured image data that has been held in the past, and always obtains the latest n captured image data. High resolution image data can be generated based on the above.

  The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

  Moreover, in the said embodiment, the super-resolution process part 12 which performs a super-resolution process, the captured image storage part 13 which memorize | stores various data so that reading is possible, the high-resolution image storage part 14, and the defect inspection image storage part 16 Although the configuration in which the defect inspection processing unit 15 that performs the defect inspection is included in the control device 6 has been shown, some or all of these components can also be configured separately.

  Finally, each block of the control device 6, in particular, the imaging control unit 11, the super-resolution processing unit 12, and the defect inspection processing unit 15 may be configured by hardware logic or using a CPU as follows. It may be realized by software.

  That is, the control device 6 includes a CPU (central processing unit) that executes instructions of a control program that realizes each function, a ROM (read only memory) that stores the program, a RAM (random access memory) that expands the program, A storage device (recording medium) such as a memory for storing the program and various data is provided. An object of the present invention is to provide a recording medium in which a program code (execution format program, intermediate code program, source program) of a control program of the control device 6 which is software that realizes the above-described functions is recorded so as to be readable by a computer. This can also be achieved by supplying the control device 6 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

  Examples of the recording medium include a tape system such as a magnetic tape and a cassette tape, a magnetic disk such as a floppy (registered trademark) disk / hard disk, and an optical disk such as a CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

  The control device 6 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Also, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  According to the present invention, the amount of calculation required for the super-resolution processing can be reduced, so that the present invention is not limited to an apparatus for inspecting product defects, and can be applied to various apparatuses for increasing the resolution of image data. Further, the target of higher resolution is not limited to still images, and it is possible to increase the resolution of moving images as well.

1, showing an embodiment of the present invention, is a block diagram illustrating a main configuration of a super-resolution processing unit. FIG. BRIEF DESCRIPTION OF THE DRAWINGS It is a figure which shows embodiment of this invention and shows the outline | summary of a test | inspection system. It is a block diagram which shows the principal part structure of the control apparatus with which the said inspection system is provided. It is a flowchart which shows an example of the process in the said test | inspection system. It is a figure which shows an example of arrangement | positioning of the display pixel which the liquid crystal panel which is a test object of the said test | inspection system has. It is a figure which shows the correspondence of the pixel of the said liquid crystal panel, and the solid-state image sensor which the imaging device of the said inspection system has, The figure (a) shows the state by which the image sensor sufficient for a defect detection is allocated, FIG. 5B shows the correspondence between the pixels and the solid-state imaging device when the number of display pixels of the liquid crystal panel is about 1920 × 1080 and the imaging device is a 11 million pixel CCD sensor, and FIG. The correspondence relationship between the pixels and the solid-state imaging device when the number of display pixels of the panel is about 1920 × 1080 and the imaging device is a 16 million pixel CCD sensor is shown. It is a flowchart which shows an example of the super-resolution process which the said super-resolution process part performs. It is a figure which shows the relationship between the captured image data and high resolution image data which are used with the said inspection system. It is a block diagram which shows the principal part structure of the defect inspection process part with which the said control apparatus is provided. It is a figure which shows the relationship between the captured image data and high resolution image data in embodiment of this invention different from the above. It is a figure which shows the relationship between the captured image data and high resolution image data in another embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Inspection system 4 Imaging device 6 Control apparatus 11 Imaging control part 12 Super-resolution processing part (image processing apparatus)
12a Image registration unit 12b Comparison image generation unit (comparison image generation means)
12c Image update unit (image update data generation means, image update means)
13 Captured image storage unit 14 High-resolution image storage unit 15 Defect inspection processing unit (defect inspection apparatus)

Claims (7)

In an image processing apparatus that generates high-resolution image data having a higher resolution than the image data based on a plurality of image data obtained by imaging one imaging target,
Comparative image data indicating a difference after alignment between the first high-resolution image data generated from n pieces of image data and the processing target image data different from the image data obtained by imaging the imaging target Comparison image generating means for generating
The comparison is made by multiplying image update data corresponding to the difference between the second high-resolution image data generated from the n pieces of image data and the processing target image data and the first high-resolution image data by a predetermined constant. Image update data generation means for generating the image update data by repeatedly performing calculation until the value of the mathematical formula formed by adding the image data is smaller than a predetermined value;
An image processing apparatus comprising: an image update unit configured to add the generated image update data and the first high resolution image data to generate second high resolution image data. In an image processing apparatus that generates high-resolution image data having a higher resolution than the image data based on a plurality of image data obtained by imaging one imaging target,
Comparison image for generating comparison image data indicating a difference after alignment between the first high-resolution image data generated from n pieces of image data and the processing target image data selected from the n pieces of image data Generating means;
The comparison image data is a product of image update data corresponding to a difference between the second high resolution image data generated from image data other than the processing target image data and the first high resolution image data, and a predetermined constant. Image update data generation means for generating the image update data by repeatedly performing an arithmetic operation until the value of the mathematical formula obtained by adding is smaller than a predetermined value;
An image processing apparatus comprising: an image update unit configured to add the generated image update data and the first high resolution image data to generate second high resolution image data. In an image processing apparatus that generates high-resolution image data having a higher resolution than the image data based on a plurality of image data obtained by imaging one imaging target,
the first high-resolution image data generated from the n pieces of image data and the difference after alignment between the first processing target image data different from the image data obtained by imaging the imaging target; Comparison image generation means for generating comparison image data indicating a difference between the first high-resolution image data and the difference after the alignment between the second image data to be processed selected from the n pieces of image data. ,
Difference between second high-resolution image data generated from image data obtained by removing the second processing target image data from the n pieces of image data and adding the first processing target image data, and the first high-resolution image data The image update data is generated by iteratively calculating until the value of the mathematical formula obtained by adding the comparison image data to the product of the image update data corresponding to and a predetermined constant is smaller than a predetermined value. Image update data generating means for performing,
An image processing apparatus comprising: an image update unit configured to add the generated image update data and the first high resolution image data to generate second high resolution image data. The image processing apparatus according to any one of claims 1 to 3,
An imaging device that captures an imaging target and generates image data;
An inspection system including: an inspection apparatus that performs defect inspection of the imaging target based on second high-resolution image data generated by the image processing apparatus. In an image processing method of an image processing apparatus that generates high-resolution image data having a higher resolution than the image data based on a plurality of image data obtained by imaging one imaging target,
Comparative image data indicating a difference after alignment between the first high-resolution image data generated from n pieces of image data and the processing target image data different from the image data obtained by imaging the imaging target A comparison image generation step for generating
The comparison is made by multiplying image update data corresponding to the difference between the second high-resolution image data generated from the n pieces of image data and the processing target image data and the first high-resolution image data by a predetermined constant. An image update data generation step of generating the image update data by repeatedly performing an arithmetic operation until the value of the mathematical formula formed by adding the image data is smaller than a predetermined value;
An image processing method comprising: an image update step of adding the generated image update data and the first high resolution image data to generate second high resolution image data.   An image processing program for operating the image processing apparatus according to any one of claims 1 to 3, wherein the image processing program causes a computer to function as each of the means.   A computer-readable recording medium on which the image processing program according to claim 6 is recorded.

Images