JP6045429B2 - Imaging apparatus, image processing apparatus, and image processing method - Google Patents

Description

  The present invention relates to an imaging apparatus, an image processing apparatus, and an image processing method for acquiring information indicating the structure of the surface of an inspection object from an image obtained by imaging the inspection object.

  As inspection methods for scratches generated on the surface of an object to be inspected such as a substrate, direct visual inspection, inspection using a microscope, inspection using an image taken by a camera or the like are known. In these methods for inspecting scratches, if the surface of the object to be inspected is minute, it is necessary to observe the image at a high magnification, so the range that can be observed at one time becomes narrow, and the inspection takes a long time. There is a problem.

  There is also known a method for inspecting scratches in which the surface of an object to be inspected is colored with a coloring liquid, and observation is performed by filling a minute recess such as a scratch with the coloring liquid. However, in this method for inspecting scratches, the colored liquid may affect the surface of the object to be inspected, such as deterioration. Further, it is necessary to perform work such as removal of the colored liquid after the inspection, which is very troublesome.

  Therefore, in order to inspect minute scratches on the surface of the object to be inspected, a technique for observing an image obtained by irradiating light on the object to be inspected and imaging the object to be inspected has been proposed. For example, Patent Document 1 discloses a technique for irradiating light from a plurality of directions different from each other on an object to be inspected and detecting a flaw based on light and dark caused by unevenness of a flaw portion. Patent Document 2 discloses a technique for generating a composite image in which the contour of a concave portion of a scratched part is emphasized by combining a plurality of images having different light illumination directions with respect to the inspection object. According to these techniques, it is possible to inspect a scratch in a relatively short time without contact with the surface of the inspection object.

JP 2009-99777 A JP 2000-125289 A

  By the way, when imaging is performed by irradiating the inspection object with light from various directions, shading (brightness unevenness) may occur in the obtained image depending on conditions such as the direction, position, and directivity of the light source. is there. However, in Patent Documents 1 and 2, since shading is not considered, when shading is generated in an image, it is impossible to distinguish between light and dark caused by unevenness of a scratched portion and light and dark due to shading. In addition, when automatic detection of a flaw is performed using a computer or the like based on the light and darkness due to the unevenness of the flaw part, shading may reduce the flaw detection accuracy.

  A technique for correcting shading generated in an image by using a shading characteristic acquired in advance by photographing a white board or the like is also known, but in this case, labor and a process for photographing the white board are increased.

  The present invention has been made in view of the above, and based on a plurality of images having different illumination directions, information relating to the structure of the surface of the object to be inspected can be obtained easily and accurately without the need for shading correction. An object is to provide an imaging device, an image processing device, and an image processing method that can be obtained.

  In order to solve the above-described problems and achieve the object, an imaging apparatus according to the present invention includes an illumination unit that sequentially irradiates a subject with light from a plurality of different directions, and each time the subject is irradiated with the light. In addition, the subject is imaged so that at least a part of the imaging field of view is common, and an imaging unit that generates image information of a plurality of images respectively corresponding to a plurality of illumination directions, and each of the plurality of images is captured Regions that respectively extract at least two region images that are partial images of a common region of the subject from at least two images of the plurality of images based on information about the illumination direction of the light at the time An image extraction unit and a synthesis processing unit that synthesizes the extracted region images are provided.

  In the imaging apparatus, the plurality of illumination directions include at least two directions that are orthogonal to each other.

  In the imaging apparatus, the region image extraction unit includes a shading characteristic estimation unit that estimates a shading characteristic generated in each of the plurality of images, and based on the shading characteristics estimated by the shading characteristic estimation unit, the at least two A region image is extracted.

  In the imaging apparatus, the illuminating units are arranged at different positions with respect to the imaging visual field, each of which includes a plurality of light sources or a plurality of light source groups that emit light that irradiates the subject, and the shading characteristic estimation The unit estimates the shading characteristics based on a position and / or orientation of a light source or a light source group that emits light when each of the plurality of images is captured among the plurality of light sources or the plurality of light source groups. It is characterized by.

  In the imaging apparatus, the shading characteristic estimation unit estimates the shading characteristic based on luminance information of each of the plurality of images.

  In the imaging apparatus, the composition processing unit obtains information on a structure depending on each of the plurality of illumination directions by synthesizing the at least two region images.

  In the imaging apparatus, the synthesis processing unit synthesizes the at least two area images by a weighted average process.

  In the imaging apparatus, the synthesis processing unit synthesizes the at least two region images by difference processing.

  In the imaging apparatus, the region image extraction unit includes an image region dividing unit that sets a plurality of region images by dividing each of the plurality of images into a plurality of regions according to the plurality of illumination directions, The at least two area images are extracted by selecting a corresponding area image between the at least two images.

  The image processing apparatus according to the present invention sequentially irradiates a subject with light from a plurality of different directions, and each time the subject is irradiated with the light, the subject is arranged so that at least a part of an imaging field of view is common. In an image processing apparatus that processes a plurality of images respectively corresponding to a plurality of illumination directions based on image information generated by imaging, information on the illumination direction of the light when each of the plurality of images is captured A region image extraction unit that extracts at least two region images, each of which is a partial image showing a common region of the subject, from at least two images of the plurality of images, and the extracted image And a synthesis processing unit that synthesizes the region images.

  The image processing method according to the present invention sequentially irradiates a subject with light from a plurality of different directions, and each time the subject is irradiated with the light, the subject is arranged so that at least a part of an imaging field of view is common. In an image processing method for processing a plurality of images respectively corresponding to a plurality of illumination directions based on image information generated by imaging, information on the illumination direction of the light when each of the plurality of images is captured A region image extraction step for extracting at least two region images, each of which is a partial image showing a common region of the subject, from at least two images of the plurality of images; And a synthesis processing step for synthesizing the region images.

  According to the present invention, since a region image to be combined is extracted from a plurality of images having different illumination directions based on information on the illumination direction of light in each image, the need for shading correction is not required. Information regarding the structure of the surface of the object to be inspected can be obtained easily and accurately.

FIG. 1 is a schematic diagram illustrating a configuration example of an imaging apparatus according to Embodiment 1 of the present invention. FIG. 2 is a block diagram showing a configuration of the region image extraction unit shown in FIG. FIG. 3 is a flowchart showing the imaging method according to Embodiment 1 of the present invention. FIG. 4 is a diagram illustrating a step of acquiring a plurality of images having different illumination directions. FIG. 5 is a schematic diagram showing a different direction illumination image group. FIG. 6 is a schematic diagram for explaining a shadow corresponding to the illumination direction that appears when a structure with unevenness exists on the inspection object. FIG. 7 is a schematic diagram for explaining the step of dividing each image of the different direction illumination image group. FIG. 8 is a schematic diagram showing a synthesized image obtained by synthesizing the image shown in FIG. 7 by a simple average process. FIG. 9 is a photograph showing an example in which a microscope image obtained by an industrial microscope is synthesized by a weighted average process. FIG. 10 is a schematic diagram illustrating a combined image obtained by combining the image illustrated in FIG. 7 by difference processing. FIG. 11 is a schematic diagram illustrating a configuration example of an imaging apparatus according to Embodiment 2 of the present invention. FIG. 12 is a block diagram showing a configuration of the region image extraction unit shown in FIG. FIG. 13 is a flowchart showing an imaging method according to Embodiment 2 of the present invention. FIG. 14 is a schematic diagram illustrating an image acquired by performing imaging by sequentially illuminating the field of view of the imaging unit illustrated in FIG. 4 from the direction of each light source.

  Hereinafter, embodiments of an imaging device, an image processing device, and an image processing method according to the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to these embodiments. Moreover, in description of each drawing, the same code | symbol is attached | subjected and shown to the same part.

  In the following, as an example of the configuration of the imaging apparatus according to the present invention, an example in which the image processing apparatus according to the present invention is combined with an industrial microscope that performs substrate inspection or the like will be given. The image processing apparatus according to the present invention can be configured in combination with various image acquisition means for capturing an image of a subject and generating image information.

(Embodiment 1)
FIG. 1 is a block diagram showing a configuration of an imaging apparatus according to Embodiment 1 of the present invention. As shown in FIG. 1, the imaging apparatus 1 according to Embodiment 1 includes a divided light source unit 11 that irradiates light from a plurality of different directions onto an inspection target (inspected object) 10 that is a subject, Based on the divided light source control unit 12 that controls the operation of the divided light source unit 11, the imaging unit 13 that images the inspection object 10 and generates image information (image data), and the image information generated by the imaging unit 13. And an image processing apparatus 100 that performs image processing on the image of the inspection object 10.

  The divided light source unit 11 is an illuminating unit including a plurality of light sources 11a to 11d provided at positions different from each other with respect to the field of view of the imaging unit 13 described later. The light sources 11a to 11d sequentially emit light under the control of the divided light source control unit 12, and sequentially illuminate the inspection object 10 from different directions.

  In the present embodiment, a combination of a plurality of point light sources (light sources 11a to 11d) as shown in FIG. 1 is used as a configuration of the divided light source unit 11, but illumination light is irradiated from a plurality of directions to the inspection object. However, the configuration is not limited to this. For example, a combination of a plurality of parallel light sources in which an optical system such as a lens is combined with a point light source may be used, or ring illumination in which a plurality of point light sources are arranged in a ring shape may be used. Ring illumination is illumination means used in the field of industrial microscopes, etc., and divides a plurality of point light sources into a plurality of regions, and simultaneously emits the point light source groups included in each region, thereby making the center of gravity of the point light source group Assuming that one virtual light source exists at the position, it is possible to consider the relative positional relationship and orientation with the field of view of the imaging unit 13.

  The divided light source control unit 12 sequentially switches the light sources 11 a to 11 d to emit light under the control of a control unit 107 (described later) included in the image processing apparatus 100. The divided light source control unit 12 captures information such as the relative positional relationship of the emitted light sources 11a to 11d with the image capturing unit 13 and the direction in which the object 10 is irradiated on the image capturing unit 13. Is transmitted to the image processing apparatus 100 as information relating to the illumination direction at the time.

  The imaging unit 13 includes an imaging system such as a CCD or CMOS and an optical system such as an objective lens, for example, and is synchronized with the light emission of the light sources 11a to 11d under the control of a control unit 107 (described later) provided in the image processing apparatus 100. Then, the object 10 is imaged. More specifically, the imaging unit 13 receives light emitted from any of the light sources 11a to 11d and reflected by the inspection object 10 on the light receiving surface of the imaging element via an optical system such as an objective lens, An electric signal representing an image of the inspection object 10 is generated by photoelectric conversion. The imaging unit 13 further generates digital image information (image data) by performing A / D conversion processing on the generated electrical signal, and outputs the digital image information to the image processing apparatus 100.

  The image processing apparatus 100 generates an image of the inspection object 10 by performing predetermined image processing on the image data output from the imaging unit 13, and a scratch generated on the surface of the inspection object 10 based on the image. Acquire information on the structure with irregularities such as. The image processing apparatus 100 controls an input unit 101, an external interface (I / F) unit 102, a storage unit 103, a region image extraction unit 104, a synthesis processing unit 105, an output unit 106, and each of these units. And a control unit 107.

  The input unit 101 includes input devices such as a keyboard, various buttons, and various switches, and pointing devices such as a mouse and a touch panel, and outputs signals corresponding to user operations on these devices to the control unit 107.

  The external interface unit 102 is an interface for connecting an external device to the image processing apparatus 100. In the first embodiment, the divided light source control unit 12 and the imaging unit 13 are connected via the external interface unit 102. Connected to the image processing apparatus 100. The external interface unit 102 transmits a control signal for the imaging operation output from the control unit 107 to the imaging unit 13, receives image data output from the imaging unit 13, and performs processing such as format conversion on the image data. Thus, display image data is generated and stored in the storage unit 103. In addition, the external interface unit 102 transmits a control signal for the light emission operation of the divided light source unit 11 output from the control unit 107 to the divided light source control unit 12 and information on the illumination direction output from the divided light source control unit 12. Is stored in the storage unit 103 in association with the image data.

  The storage unit 103 includes a recording device such as a flash memory, RAM, and ROM that can be updated and recorded, a recording medium such as a hard disk, MO, CD-R, and DVD-R that is built-in or connected by a data communication terminal, and The recording apparatus includes a reading device that reads information recorded on the recording medium. The storage unit 103 stores image data and various information input via the external interface unit 102, and also stores various control programs executed by the control unit 107 and various information used during the execution of the programs. .

  The region image extraction unit 104 extracts region images to be processed in the subsequent synthesis processing unit 105 from a plurality of images having different illumination directions with respect to the inspection object 10. Here, the region image is a partial image that is a part of each image.

  FIG. 2 is a block diagram illustrating a detailed configuration of the region image extraction unit 104. As shown in FIG. 2, the region image extraction unit 104 includes a shading characteristic estimation unit 110, an image region division unit 111, and a region image selection unit 112.

  The shading characteristic estimation unit 110 estimates a shading characteristic according to the illumination direction with respect to the inspection object 10 based on the information about the illumination direction output from the divided light source control unit 12.

  The image area dividing unit 111 divides each image into a plurality of areas based on the information on the illumination direction output from the divided light source control unit 12. Each of the divided areas is set as the above-described area image. When a plurality of illumination directions when the imaging unit 13 captures an image are determined in advance, the image area dividing unit 111 divides each image by a predetermined method without using information on the illumination direction. May be.

  The area image selection unit 112 has a common area on the surface of the inspection object 10 out of area images obtained by dividing each of the plurality of images based on the shading characteristics estimated by the shading characteristic estimation unit 110. A pair of area images that are captured image areas and for which the composition processing unit 105 (to be described later) performs composition processing is selected.

  The composition processing unit 105 generates a composite image by performing composition processing on the pair of region images extracted by the region image extraction unit 104. The synthesis processing unit 105 can execute a weighted average process or a difference process as a synthesis process for a pair of region images, and which process is to be executed is set in advance. In addition, when executing the difference processing, as the output method, it is possible to output the difference image itself or the result of extracting the uneven structure from the difference image (inspection result). It is set in advance. The composition processing and output method executed by the composition processing unit 105 may be configured so that the user can specify from the outside using the input unit 101.

  The output unit 106 includes a display device such as an LCD, an EL display, or a CRT display, and displays the composite image synthesized by the synthesis processing unit 105, the inspection result, and the like. In the first embodiment, the display device is provided inside the image processing device 100. However, the display device may be provided outside the image processing device 100 via the external interface unit 102.

  The control unit 107 is configured by hardware such as a CPU, for example, and reads various control programs stored in the storage unit 103, so that the image processing apparatus 100 and the imaging apparatus 1 are based on various types of information stored in the storage unit 103. Overall control of the overall operation. Specifically, the control unit 107 causes the divided light source control unit 12, the imaging unit 13, and each unit in the image processing apparatus 100 to execute an operation for acquiring information regarding the structure of the surface of the inspection object 10.

Next, an imaging method according to Embodiment 1 will be described. FIG. 3 is a flowchart illustrating the imaging method according to the first embodiment.
First, in step S1, the image processing apparatus 100 sequentially emits the light sources 11a to 11d of the divided light source unit 11 and causes the imaging unit 13 to perform imaging in synchronization with the light emission of the light sources 11a to 11d. A plurality of images (hereinafter also referred to as different-direction illumination image groups) in which at least a part of the field of view of the object 10 is common and the illumination directions are different from each other are acquired. In the first embodiment, an image with the same field of view with respect to the inspection object 10 is acquired.

  In the first embodiment, as shown in FIG. 4, the visual field 13a of the imaging unit 13 is illuminated from the upper side by the light source 11a, illuminated from the lower side by the light source 11b, illuminated from the left side by the light source 11c, and the light source 11d. Thus, the operation of illuminating from the right side direction and imaging the inspection object 10 in the visual field 13a four times in synchronization with the illumination from each of these directions is defined as one cycle. FIG. 4 shows a state where the light source 11a emits light.

  The number and direction of the illumination directions are not particularly limited as long as at least two illumination directions that are orthogonal to each other are included. For example, in FIG. 4, the visual field 13a is sequentially illuminated from the direction of four sides, but four light sources 11a to 11d are arranged in the vicinity of the four vertices of the visual field 13a, and the visual field 13a is arranged at the four vertices. You may illuminate sequentially from the direction.

  In step S2, the image data of the different-direction illumination image group sequentially generated by the imaging operation of the one cycle is sequentially input from the imaging unit 13 to the image processing apparatus 100, and the light sources 11a to 11 emitted at the time of imaging each image. 11d position information (light source position information) is sequentially input from the divided light source control unit 12 to the image processing apparatus 100 as information on the illumination direction.

  FIG. 5 is a schematic diagram showing a different-direction illumination image group corresponding to the image data input to the image processing apparatus 100. Specifically, the image 21 shown in FIG. 5A is an image captured by illuminating the visual field 13a from above in the figure with the light source 11a. An image 22 shown in FIG. 5B is an image captured by illuminating the visual field 13a from below in the figure with the light source 11b. An image 23 shown in FIG. 5C is an image captured by illuminating the visual field 13a from the left direction of the drawing with the light source 11c. An image 24 illustrated in FIG. 5D is an image captured by illuminating the visual field 13a from the right direction of the drawing with the light source 11d.

  Here, as shown in FIG. 6, when a structure 10a having irregularities such as scratches is present on the surface of the inspection object 10, when the structure 10a is illuminated obliquely, a shadow corresponding to the illumination direction appears. For this reason, in each image in the different-direction illumination image group in which the inspection object 10 is copied, light and dark according to the illumination direction, that is, the positions of the emitted light sources 11a to 11d appear. This light and dark appears most strongly when the structure 10a extends in a direction perpendicular to the illumination direction, and hardly appears when the structure 10a extends in a direction parallel to the illumination direction.

  Therefore, in the images 21 and 22 in which the visual field 13a is illuminated from the vertical direction in the figure, the shadow of the structure extending in the left-right direction indicated by the solid line appears strongly, and the shadow of the structure extending in the vertical direction indicated by the broken line hardly appears. On the other hand, in the images 23 and 24 in which the visual field 13a is illuminated from the left-right direction in the figure, the shadow of the structure extending in the up-down direction indicated by the solid line appears strongly, and the shadow of the structure extending in the left-right direction indicated by the broken line hardly appears. Therefore, by illuminating the same visual field 13a from a plurality of directions including two directions that are orthogonal to each other and performing imaging a plurality of times, it is possible to obtain a plurality of images in which the specific difference of the structure 10a appears most. .

  Note that, when the object to be inspected 10 is illuminated from an oblique direction, irregular reflection occurs in an uneven structure portion such as a scratch. Therefore, the structure can be confirmed by observing an image of irregular reflection. In, in order to perform a simple calculation process, the uneven structure is detected from the shadow in the image.

  In step S <b> 3, the image region dividing unit 111 divides each image of the different direction illumination image group into regions based on the light source position information output from the divided light source control unit 12. Specifically, the images 21 to 24 are divided in a direction orthogonal to the illumination direction. For example, in the case of the different direction illumination image group shown in FIG. 5, since the illumination is performed from the vertical direction and the horizontal direction in the drawing, the vertical direction and the horizontal direction of each image 21 to 24 are shown as indicated by the alternate long and short dashed lines in FIG. By dividing each into two, it is divided into four rectangular areas. Thereby, area images 21a to 21d, 22a to 22d, 23a to 23d, and 24a to 24d are obtained from the images 21 to 24, respectively.

  Here, the number of image divisions is not limited to four, and may be arbitrarily set, for example, one or both of the vertical direction and the horizontal direction are divided into three or more. For example, when the light sources 11a to 11d are arranged in the vicinity of the apex of the visual field 13a, each image may be divided into four triangular regions by two diagonal lines orthogonal to the illumination direction.

  In addition, when the illumination direction with respect to the visual field 13a and the image division method are determined in advance, the image area dividing unit 111 may perform image division according to a predetermined method without using the light source position information.

  In step S <b> 4, the shading characteristic estimation unit 110 estimates the shading characteristic in each image of the anisotropic illumination image group based on the light source position information output from the divided light source control unit 12. Here, for example, in an industrial microscope or the like, the intensity of illumination light emitted from a light source is the strongest directly under the light source and decreases as the distance from the light source increases. Therefore, as shown in FIGS. 5A to 5D, in each of the images 21 to 24, the region where the distance between the inspection object 10 and the light sources 11 a to 11 d is closer is brighter, and the region where the distance is farther is darker. Shading occurs. Therefore, the shading characteristic estimation unit 110 estimates the shading characteristics generated in the images 21 to 24 from the positional relationship between the visual field 13a and the light sources 11a to 11d when the images 21 to 24 are captured. For example, in the case of the image 21, since it is illuminated by the light source 11a above the visual field 13a, the shading characteristic that the upper part is bright and the lower part is dark is estimated.

For such shading characteristics, a plurality of shading characteristics for each illumination direction are stored in the storage unit 103 in advance, and the shading characteristic estimation unit 110 reads the corresponding shading characteristics from the storage unit 103 based on the light source position information. Anyway. Alternatively, the shading characteristic estimation unit 110 stores in advance one type of shading characteristic corresponding to the position of a certain light source (for example, the light source 11a), and rotates the shading characteristic based on the light source position information, thereby each image. You may estimate the shading characteristic in 22-24.
In this way, by using the light source position information, it is possible to estimate the shading characteristics occurring in each of the images 21 to 24 without requiring calculation cost.

  In step S5, the region image selection unit 112 is estimated in step S4 from the region images 21a to 21d, 22a to 22d, 23a to 23d, and 24a to 24d (see FIG. 7) obtained by the region division in step S3. Based on the shading characteristics, the area image group to be subjected to the subsequent synthesis processing is selected.

  Specifically, the region image selection unit 112 selects a plurality of pairs of region images having a small difference in shading characteristics between region images from among a plurality of region images in which a common region on the surface of the inspection object 10 is captured. The magnitude of the difference between the shading characteristics between the area images can be determined, for example, by calculating the square sum or the absolute value sum of the differences between the shading characteristic values between the corresponding pixels in the two area images.

  In the case of FIG. 7, among the upper left region images 21a, 22a, 23a, and 24a of the images 21 to 24, the pair of region images 21a and 23a has a small difference in shading characteristics. Accordingly, the pair of area images 21a and 23a is selected as a target for the synthesis process. Similarly, a pair of region images 21b and 24b is selected from the region images 21b, 22b, 23b, and 24b in the upper right of the images 21 to 24. Similarly, a pair of region images 22c, 24c is selected from the lower right region images 21c, 22c, 23c, 24c of the images 21-24. Similarly, a pair of region images 22d and 23d is selected from the lower left region images 21d, 22d, 23d, and 24d of the images 21 to 24.

  In addition, when there are a plurality of pairs of region images having a small difference in shading characteristics with respect to the region image at the same position, it is preferable to select the pair having the higher luminance. For example, in the case of the region images 21a, 22a, 23a, and 24a, the pair of the region images 22a and 24a also has a small difference in shading characteristics, but the pair of the region images 21a and 23a with higher luminance is selected as the target of the synthesis process. The

In this manner, by selecting a pair of region images having a small difference in shading characteristics, it is possible to suppress the influence of shading in the subsequent synthesis processing.
When the region dividing method in step S3 is fixed, the shading characteristics generated in each region image are also constant for each illumination direction and region image position. In this case, for each position of the area image, a difference in shading characteristics generated in area images having different illumination directions may be calculated in advance, and a pair of illumination directions that minimizes the difference in shading characteristics may be obtained. Thereby, in step S5, it becomes possible to easily select a pair of area images to be combined based on only the light source position information.

  In step S6, the composition processing unit 105 selects, for example, a preset process from the weighted average process and the difference process as the composition process for the region image group selected in step S5.

  When the weighted average process is preset as the synthesis process (step S6: weighted average process), the synthesis processing unit 105 performs the weighted average process for each pair on the region image group selected in step S5 (step S6). S7).

For example, the pixel value of a pixel at coordinates (x, y) in one region image (for example, region image 21b) selected as a pair is set to I _u (x, y), and the other region image (for example, region image 24b) When the pixel value of the pixel at the same position of Ir is set to I _r (x, y) and the weight given to each pixel of each area image is set to 0.5 (that is, simple average), the pixel of the pixel in the weighted average image The value I _a (x, y) is given by the following equation (1).

Further, the composition processing unit 105 calculates an edge amount in each pixel of each region image of the selected region image group, and a weight w (x based on a difference in edge amount in each corresponding pixel between the paired region images. , Y) may be added to perform the weighted average processing. In this case, the pixel value I _w (x, y) of the pixel in the weighted average image is given by the following equation (2).

  In Expression (2), the weight w (x, y) is set so that the edge amount of one region image (for example, region image 21b) selected as a pair is the amount of edge of the other region image (for example, region image 24b). If it is sufficiently large, it is set so as to be close to 1, and conversely, if the edge amount of one area image is sufficiently smaller than the edge amount of the other area image, it is set so as to approach 0. Thus, in the weighted average image, the contribution of the pixel value of the region image having a large edge amount is large.

  In this way, by performing the weighted average process according to the edge amount of the region image, it is possible to avoid averaging the pixel value of the pixel showing the uneven structure and the pixel value of the pixel not showing the structure, A composite image in which the structure appears more clearly can be obtained.

In subsequent step S8, the synthesis processing unit 105 outputs a synthesized image (weighted average image) obtained by the weighted average process in step S7. In response to this, the output unit 106 displays the composite image on the screen.
Thereafter, a series of processes in the imaging apparatus 1 is completed.

  On the other hand, when the difference process is preset as the combination process in step S6 (step S6: difference process), the combination processing unit 105 performs the difference process for each pair on the region image group selected in step S5. (Step S9).

  In subsequent step S10, the composition processing unit 105 selects, as an output method, an output of a difference image or an output of an inspection result obtained by extracting a structure according to a preset setting.

  When the output of the difference image is selected as the output method (step S10: difference image), the composition processing unit 105 outputs the difference image obtained in step S9 (step S11). In response to this, the output unit 106 displays the difference image on the screen.

Here, in the difference image, the brightness is high in a pixel having a large pixel value difference between the region images to be processed, and the brightness is low in a pixel having a small pixel value difference. Therefore, when there is a structure depending on the illumination direction (a vertical scratch or a horizontal scratch), the structure appears as a pixel region with high brightness.
Thereafter, a series of processes in the imaging apparatus 1 is completed.

  On the other hand, when the output of the inspection result is selected in step S10 (step S10: inspection result), the composition processing unit 105 performs threshold processing on the difference image obtained in step S9 (step S12). As described above, since the structure depending on the illumination direction appears as a pixel region with high brightness, the structure can be extracted from the difference image by threshold processing, and the presence or absence of the structure can be automatically determined.

  Here, when a structure such as a scratch is extracted from an image obtained by imaging the inspection object 10, edge detection is often performed. However, when a texture component such as a character or a pattern exists on the surface of the inspection object 10 in addition to a structure such as a scratch, it is difficult to accurately extract only the structure by edge detection.

  On the other hand, in the case of the first embodiment, there is only a difference in structure depending on the illumination direction between two region images having different illumination directions, except for the difference in shading characteristics. That is, since the texture components are common, it is possible to accurately extract only the structure of the surface of the inspection object 10 by taking the difference between the two region images.

In subsequent step S13, the composition processing unit 105 outputs the structure extraction result. Specifically, when a pixel having a pixel value equal to or greater than a predetermined threshold is detected as a result of threshold processing on the difference image, the synthesis processing unit 105 determines that there is an uneven structure. The position information of the detected pixel is output. On the other hand, when no pixel having a pixel value equal to or greater than the threshold value is detected, the synthesis processing unit 105 determines that there is no uneven structure and outputs a determination result such as “no flaw”. In response to this, the output unit 106 displays the determination result output from the synthesis processing unit 105 on the screen.
Thereafter, a series of processes in the imaging apparatus 1 is completed.

  In FIG. 8A, a weighted average process (see step S7) is performed on a pair of region images selected from the images 21 to 24 shown in FIG. 7 with a weight w (x, y) of 0.5. It is a schematic diagram which shows the obtained synthesized image. As shown to Fig.8 (a), in this synthesized image, the shadow showing the structure of an unevenness | corrugation appears clearly irrespective of the illumination direction of the original images 21-24. In addition, by selecting a pair of area images for which a weighted average is taken in consideration of shading characteristics, the influence of shading can be suppressed in the entire composite image, and the structure can be clearly identified. .

  FIG. 8B shows, as a comparison, a schematic diagram of an image obtained by combining the image 21 and the image 24 shown in FIG. 5 by the simple average process without considering the shading characteristics. As shown in FIG. 8B, the direction dependency of the structure can be suppressed by combining images with different illumination directions. However, in this composite image, the effect of shading generated in the images 21 and 24 is further increased. Therefore, in the region where the brightness is lowered due to the effect of shading (for example, the lower left region in the figure), the structure is visually recognized. The sex is decreasing.

  FIG. 9 is a photograph showing an example in which microscopic images acquired by an industrial microscope are synthesized. FIG. 9A is a region image in the upper right of the image illuminated from above the field of view (corresponding to the region image 21b in FIG. 7A), and FIG. 9B is illuminated from the right side of the field of view. Is a region image in the upper right of the image subjected to (corresponding to the region image 24b in FIG. 7D), and FIG. 9C is a weighted average image of these region images.

  In the region P shown in FIG. 9A, scratches extending in the substantially horizontal direction are observed, but the same scratches are not observed in the region P in FIG. 9B. That is, the scratches found in the region P in FIG. 9A are scratches depending on the illumination direction. In addition, in the region Q shown in FIG. 9B, scratches extending in a substantially vertical direction are observed, but the same scratches are not observed in the region Q in FIG. 9A. That is, the flaw seen in the area | region Q of FIG.9 (b) is a flaw dependent on an illumination direction. In the weighted average image of these region images, both the scratches in the region P and the region Q can be clearly observed.

  FIG. 10 is a schematic diagram showing a composite image obtained by performing difference processing (see step S9) on a pair of region images (see step S5) selected from the images 21 to 24 shown in FIG. . As described above, in this composite image, a structure depending on the illumination direction appears as a pixel area having high brightness (that is, whitish).

  As described above, according to the first embodiment, each of a plurality of images having different illumination directions is divided into a plurality of region images, and the region images selected based on the shading characteristics are synthesized (weighted averaging process). (Or differential processing), the structure depending on the illumination direction can be accurately extracted while reducing the influence of shading. Therefore, it is possible to easily and accurately acquire information related to the structure of the surface of the object to be inspected without performing processing with a high calculation load such as shading correction.

  Further, in the conventional inspection of scratches, it is necessary to observe each of a plurality of images having different illumination directions. According to the first embodiment, a composite image (extracted from the structure of the surface of the inspection object) Since only the weighted average image or the difference image) and the structure detection result need to be confirmed, it is possible to reduce the time and labor required for the inspection of the inspection object 10.

  In the first embodiment, the case where a scratch is detected as the structure of the surface of the object to be inspected has been described. However, the detection target is not limited to a scratch, and for example, a structure having different optical anisotropy or the like according to polarization is detected. Even in this case, the first embodiment can be applied.

  In the first embodiment, illumination is performed by providing four point light sources corresponding to the four sides to the visual field 13a of the imaging unit 13, but by appropriately changing the type, number, and arrangement of the light sources. It is also possible to further improve the detection accuracy. For example, as described above, a plurality of parallel light sources may be used instead of the light sources 11a to 11d, or a ring light source may be used. Further, instead of sequentially emitting the plurality of light sources, the illumination direction with respect to the visual field 13a may be switched by sequentially switching the filters.

  In the first embodiment, in consideration of shading characteristics, two region images are extracted from two images and combined for one region. However, the present invention is not limited to this. In addition, it is possible to extract three or more region images with a small difference in the influence of shading from three or more images and perform synthesis processing.

  In the first embodiment, the composition process and the output method are selected according to the preset contents (see steps S6 and S10), but the user inputs the instruction information using the input unit 101, so that the composition process is performed. You may make it designate the synthetic | combination process and output method which the part 105 performs.

  Further, the processing for each image described above may be executed by software or may be executed by hardware.

(Embodiment 2)
Next, a second embodiment of the present invention will be described.
FIG. 11 is a schematic diagram illustrating a configuration of an imaging apparatus according to Embodiment 2 of the present invention. As illustrated in FIG. 11, the imaging apparatus 2 according to the second embodiment includes an image processing apparatus 200 instead of the image processing apparatus 100 illustrated in FIG. 1. The configurations and operations of the divided light source unit 11, the divided light source control unit 12, and the imaging unit 13 are the same as those in the first embodiment.

  The image processing apparatus 200 includes a region image extraction unit 201 instead of the region image extraction unit 104 included in the image processing device 100 illustrated in FIG. FIG. 12 is a block diagram illustrating a detailed configuration of the region image extraction unit 201.

  As illustrated in FIG. 12, the region image extraction unit 201 includes a shading characteristic estimation unit 202, an image region division unit 111, and a region image selection unit 112. Among these, the operations of the image region dividing unit 111 and the region image selecting unit 112 are the same as those in the first embodiment.

  The shading characteristic estimation unit 202 estimates shading characteristics generated in each of a plurality of images having different illumination directions based on image data input from the imaging unit 13 to the image processing apparatus 200 and stored in the storage unit 103. . At this time, unlike the first embodiment, the shading characteristic estimation unit 202 estimates the shading characteristic using the luminance information of the image itself without using the light source position information from the divided light source control unit 12.

  Next, an imaging method according to Embodiment 2 will be described. FIG. 13 is a flowchart illustrating an imaging method according to the second embodiment. Note that steps S1 to S3 shown in FIG. 13 are common to the first embodiment (see FIG. 3).

  In step S20 following step S3, the shading characteristic estimation unit 202 estimates the shading characteristics of the image based on the luminance information of each image in the anisotropic illumination image group. Here, shading can be regarded as low-frequency luminance fluctuations in an image. Therefore, shading characteristics can be extracted by performing low-pass filter processing on a luminance image obtained by converting the pixel value of each pixel in the image into a luminance value. Alternatively, the shading characteristics may be extracted by a known technique such as top hat processing or bottom hat processing by morphology.

In subsequent step S21, the region image selection unit 112 is estimated in step S20 from the region images 21a to 21d, 22a to 22d, 23a to 23d, and 24a to 24d (see FIG. 7) obtained by the region division in step S3. Based on the shading characteristics, a group of region images to be subjected to the subsequent synthesis processing is selected. The region image group selection method is the same as in the first embodiment (see step S5).
The subsequent steps S6 to S13 are the same as those in the first embodiment.

  FIGS. 14A to 14D are schematic diagrams illustrating images obtained by capturing images by sequentially illuminating the visual field 13a (see FIG. 4) of the imaging unit 13 from the directions of the light sources 11a to 11d. FIG. 14 shows an example in which each image is divided into eight in step S3.

  When the imaging device 2 actually performs imaging, shading that appears in the image changes due to, for example, the effects of aging and directivity of the light sources 11a to 11d. For example, in FIG. 14A, since the light of the light source 11a that illuminates the visual field 13a (see FIG. 4) from the upper side of the drawing does not reach the lower side enough, the image 31 is shaded as a whole. Yes. In FIG. 14B, since the light from the light source 11b that illuminates the visual field 13a from the lower side of the drawing reaches a wide range, the image 32 is shaded as a whole. In such a case, in the shading characteristic estimation method in which the shading characteristic acquired in advance is read based on the light source position information, an error from the actually generated shading characteristic becomes large. Therefore, in order to effectively reduce the influence of shading in the subsequent synthesis process (steps S7 and S9), it is preferable to directly obtain the shading characteristics from the images 31 to 34 that are the processing targets.

  As described above, according to the second embodiment, since the shading characteristics are estimated based on the luminance information of the images 31 to 34 to be processed, the light amounts and directivities of the light sources 11a to 11d are changed. Even so, it is possible to select an optimal pair of region images (for example, the region image 32a and the region image 34a) to be subjected to the subsequent synthesis process. Therefore, it is possible to acquire information regarding the structure of the surface of the inspection object 10 with high accuracy.

  The present invention is not limited to the above-described first and second embodiments and modifications, but by appropriately combining a plurality of constituent elements disclosed in the first and second embodiments and modifications. Various inventions can be formed. For example, some components may be excluded from all the components shown in the first and second embodiments and the modified examples. Or you may form combining the component shown in different embodiment suitably.

DESCRIPTION OF SYMBOLS 1, 2 Imaging device 10 Inspected object 10a Structure 11 Division | segmentation light source part 11a-11d Light source 12 Division | segmentation light source control part 13 Imaging part 13a Field of view 21-24, 31-34 Image 21a-21d, 22a-22d, 23a-23d, 24a -24d, 32a, 34a
Area image 100, 200 Image processing apparatus 101 Input unit 102 External interface (I / F) unit 103 Storage unit 104, 201 Area image extraction unit 105 Composition processing unit 106 Output unit 107 Control unit 110, 202 Shading characteristic estimation unit 111 Image region Division unit 112 Region image selection unit

Claims (11)

An illumination unit that sequentially irradiates light from a plurality of different directions on the subject;
An imaging unit that images the subject so that at least a part of an imaging field of view is common each time the light is irradiated on the subject, and generates image information of a plurality of images respectively corresponding to a plurality of illumination directions;
A partial image in which a common area of the subject is captured from at least two of the plurality of images based on information on the illumination direction of the light when each of the plurality of images is captured. An area image extraction unit for extracting at least two area images,
A synthesis processing unit for synthesizing the extracted region images;
An imaging apparatus comprising: The plurality of illumination directions includes at least two directions in an orthogonal relationship,
The imaging apparatus according to claim 1. The region image extraction unit
A shading characteristic estimation unit for estimating a shading characteristic generated in each of the plurality of images;
Extracting the at least two region images based on the shading characteristics estimated by the shading characteristic estimation unit;
The imaging apparatus according to claim 1. The illumination unit is disposed at different positions with respect to the imaging field of view, each having a plurality of light sources or a plurality of light source groups that emit light that irradiates the subject,
The shading characteristic estimation unit is configured to determine the shading characteristic based on a position and / or orientation of a light source or a light source group that is emitted when each of the plurality of images is captured among the plurality of light sources or the plurality of light source groups. The imaging apparatus according to claim 3, wherein:   The imaging apparatus according to claim 3, wherein the shading characteristic estimation unit estimates the shading characteristic based on luminance information of each of the plurality of images.   The imaging apparatus according to claim 1, wherein the synthesis processing unit acquires information regarding a structure depending on each of the plurality of illumination directions by synthesizing the at least two region images.   The imaging apparatus according to claim 1, wherein the synthesis processing unit synthesizes the at least two area images by a weighted average process.   The imaging apparatus according to claim 1, wherein the synthesis processing unit synthesizes the at least two region images by difference processing. The region image extraction unit
An image region dividing unit that sets a plurality of region images by dividing each of the plurality of images into a plurality of regions according to the plurality of illumination directions,
The imaging apparatus according to claim 1, wherein the at least two area images are extracted by selecting a corresponding area image between the at least two images. Image information generated by sequentially irradiating a subject with light from a plurality of different directions and imaging the subject so that at least a part of the imaging field of view is common each time the subject is irradiated with the light. In the image processing apparatus for processing a plurality of images respectively corresponding to a plurality of illumination directions,
A partial image in which a common area of the subject is captured from at least two of the plurality of images based on information on the illumination direction of the light when each of the plurality of images is captured. An area image extraction unit for extracting at least two area images,
A synthesis processing unit for synthesizing the extracted region images;
An image processing apparatus comprising: Image information generated by sequentially irradiating a subject with light from a plurality of different directions and imaging the subject so that at least a part of the imaging field of view is common each time the subject is irradiated with the light. In the image processing method for processing a plurality of images respectively corresponding to a plurality of illumination directions,
A partial image in which a common area of the subject is captured from at least two of the plurality of images based on information on the illumination direction of the light when each of the plurality of images is captured. A region image extraction step for extracting at least two region images respectively;
A synthesis processing step of synthesizing the extracted region images;
An image processing method comprising: