JPWO2020241648A1 - Image acquisition device and image acquisition method - Google Patents

Abstract

The image acquisition device (1) includes a model target registration unit (141) for registering a model target area, a model reference registration unit (142) for registering a model reference area as a reference for alignment, and an image pickup device (11). , A moving device (12) that moves the image pickup device in the optical axis direction, and a focusing position acquisition unit (131) that acquires the focusing position of the image having the highest focusing degree among a plurality of work images having different focal points. , The alignment unit (132) that aligns the work image at the in-focus position with the model reference area, and the attitude estimation unit (151) that estimates the posture of the work at the time of imaging based on the alignment result. And a focused area output unit (152) that outputs the position of the area corresponding to the model target area in the work image by using the posture of the work.

Description

The present disclosure relates to a technique for capturing a focused image of a work to be imaged.

Generally, when an imaging device (camera) using a high-magnification lens captures an imaged object, the depth of field (the range that appears to be in focus in the image) becomes shallow. Depending on the unevenness of the work to be imaged, the evaluation target portion in the work may not be focused, which tends to adversely affect the evaluation (inspection, positioning, etc.) of the work.

As a countermeasure, there is known a method of creating an omnifocal image of a work to be imaged and evaluating the work based on the omnifocal image (see, for example, Japanese Patent Application Laid-Open No. 2016-21115). A omnifocal image is an image in which multiple images with different focal points are taken while changing the distance between the work and the image pickup device in the direction of the optical axis of the image pickup device, and the degree of focus (degree of focus) is determined for each local area. The whole image is reconstructed by evaluating and combining local images with a high degree of focus. The omnifocal image created in this way is almost in focus at all pixels.

Japanese Unexamined Patent Publication No. 2016-21115

In the method disclosed in Japanese Patent Application Laid-Open No. 2016-21115, it is necessary to generate a omnifocal image every time the workpiece to be evaluated is imaged. When the unevenness of the work is large with respect to the depth of field, omnifocal images are generated for a large number of images, and the amount of calculation of image processing becomes large. Therefore, there is a problem that the omnifocal image cannot be generated at high speed and it takes time to evaluate the work.

The present disclosure has been made to solve the above-mentioned problems, and an object of the present disclosure is to enable high-speed acquisition of a focused image of a work to be evaluated.

The image acquisition device according to the present disclosure acquires an image of the work. This image acquisition device aligns with a first registration unit that registers a first region to be acquired by using a model image that is an image of a work to be model-registered, and a model image. A second registration unit for registering a second region used for First acquisition of the moving device configured in the above and the depth position of the image with the highest degree of focus among a plurality of work images with different focal points captured by the imaging device while operating the moving device as the focusing position. The alignment unit that identifies the region corresponding to the second region in the work image by aligning the portion and the work image at the focused position with the second region, and the second region and the work in the model image. A posture estimation unit configured to be able to acquire information indicating the posture of the work at the time of imaging based on the positional relationship with the region corresponding to the second region in the image, and a second position in the work image using information indicating the posture of the work. It includes an area output unit that outputs the position of the area corresponding to one area.

According to the present disclosure, it is possible to acquire a focused image of a work to be evaluated at high speed.

It is a figure (the 1) which shows the structure of the image acquisition system schematicly. It is a flowchart (the 1) which shows an example of the registration procedure of a model area. It is a flowchart (the 1) which shows an example of the calculation procedure of a work target area. It is a figure (the 2) which shows the structure of the image acquisition system schematicly. It is a flowchart (2) which shows an example of the registration procedure of a model area. It is a flowchart (2) which shows an example of the calculation procedure of a work target area. It is a figure (the 3) which shows the structure of the image acquisition system schematicly. It is a flowchart (3) which shows an example of the registration procedure of a model area. It is a flowchart (3) which shows an example of the calculation procedure of a work target area. It is a figure (the 4) which shows the structure of the image acquisition system schematicly. It is a flowchart (4) which shows an example of the registration procedure of a model area. It is a flowchart (4) which shows an example of the calculation procedure of a work target area. FIG. 5 is a diagram (No. 5) schematically showing the configuration of an image acquisition system. It is a figure which shows an example of the model of a neural network. It is a flowchart (No. 5) which shows an example of the registration procedure of a model area. It is a flowchart which shows an example of the learning procedure of a model area.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The same or corresponding parts in the drawings are designated by the same reference numerals, and the description thereof will not be repeated.

Embodiment 1.
FIG. 1 is a diagram schematically showing a configuration of an image acquisition system including an image acquisition device 1 according to the present embodiment. This image acquisition system includes an image acquisition device 1 and a pedestal 3 on which a work 2 to be imaged is installed.

The image acquisition device 1 includes an image pickup device 11, a moving device 12, an image processing unit 13, a model registration unit 14, a posture estimation unit 15, and a focused image acquisition unit 16.

The image pickup apparatus 11 includes a camera 111 and a trigger generator 112. The camera 111 photographs the work 2 using an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor). The trigger generator 112 outputs a trigger input signal Trig indicating the imaging timing of the camera 111 to the camera 111. The camera 111 photographs the work 2 based on the trigger input signal Trig from the trigger generator 112.

The moving device 12 includes an actuator 121. The actuator 121 is configured to be able to adjust the relative distance between the work 2 and the image pickup device 11 in the optical axis direction by moving the image pickup device 11 (camera 111) in the optical axis direction. In the following, as shown in FIG. 1, the direction along the optical axis direction of the image pickup apparatus 11 is the "Z-axis direction", and the directions perpendicular to the optical axis direction and perpendicular to each other are the "X-axis direction" and "Y", respectively. Also called "axial direction".

The moving device 12 has a function of moving the work 2 in the X-axis direction and the Y-axis direction (planar direction) by moving the pedestal 3 on which the work 2 is installed along the X-axis direction and the Y-axis direction. You may have. In the present embodiment, it is assumed that the XY coordinate positions (positions in the X-axis direction and the Y-axis direction) of the pedestal 3 are fixed when the work 2 is imaged.

Further, the moving device 12 may have a function of adjusting the relative distance between the work 2 and the imaging device 11 in the Z-axis direction by moving the pedestal 3 on which the work 2 is installed along the Z-axis direction. .. In the following, for convenience of explanation, it is assumed that the Z coordinate position (position in the Z axis direction) of the pedestal 3 is fixed at the time of imaging of the work 2.

The moving device 12 may have a function of outputting information indicating the XYZ coordinate position of the imaging device 11 to the trigger generator 112 as the moving unit position information Enc. In this case, the trigger generator 112 may output the trigger input signal Trig to the camera 111 based on the moving unit position information Enc. For example, when imaging a work 2 while moving the imaging device 11 in the Z-axis direction, the trigger generator 112 determines that the Z coordinate position of the imaging device 11 has changed by a predetermined amount based on the moving unit position information Enc. The trigger input signal Trig may be output to the camera 111 each time. In this way, a plurality of images can be captured while moving the image pickup device 11 in the Z-axis direction.

When a plurality of images are captured while the image pickup device 11 is moved in the Z-axis direction, the image pickup device 11 registers the captured plurality of images (hereinafter, also referred to as “image stack Stack”) with the image processing unit 13 and the model. Output to unit 14.

The image processing unit 13 includes a focusing position acquisition unit (first acquisition unit) 131 and an alignment unit 132. The focusing position acquisition unit 131 calculates a focusing index for each of a plurality of images included in the image stack Stack, and based on the calculated size of the focusing index, the focusing position (focus position of the image stack Stack) ( Depth position) PosZ is calculated. The region of each image for which the focus index is calculated may be the entire region of each image or a predetermined local region.

The "focused position PosZ" can be represented by, for example, the Z coordinate position of the image pickup apparatus 11 when the most focused image among the plurality of images included in the image stack Stack is captured. Therefore, the "focus index" used in the calculation of the focus position PosZ can be calculated based on the image of the work 2 captured by the image pickup device 11, and the relative distance between the image pickup device 11 and the work 2 in the Z-axis direction. It is desirable that the index changes according to the above and takes the maximum or minimum when the degree of focus is the highest.

For example, the in-focus position acquisition unit 131 can calculate the sum of the elements of the image obtained by convolving the matrices shown in the following equations (1) and (2) for each pixel of the image as the in-focus index. ..

When calculating the focus index by the above method, the focus position acquisition unit 131 selects the image having the maximum focus index among the plurality of images included in the image stack Stack, and the selected image is selected. The Z coordinate position of the imaging device 11 at the time of imaging is calculated as the in-focus position PosZ. The in-focus position acquisition unit 131 outputs the calculated in-focus position PosZ to the alignment unit 132.

The alignment unit 132 selects an image corresponding to the in-focus position PosZ from the image stack Stack acquired from the image pickup apparatus 11, and calculates the work reference region WRoiPos in the selected image.

The “work reference area WRoiPos” is an area corresponding to the model reference area MRoiPos in the image stack Stack, and is specified by the XYZ coordinate position. The "model reference area MRoiPos" is an area including a model reference image (for example, a boundary of a pattern on the work 2) as a reference for alignment, and is specified by an XYZ coordinate position. The model reference area MRoiPos is registered in advance by the model reference registration unit 142 described later.

For example, the alignment unit 132 sequentially calculates the difference between each region of the image stack Stack and the image in the model reference region MRoiPos while shifting the image in the model reference region MRoiPos with respect to the image of the image stack Stack in the XY axis direction. Then, a process of specifying the XY coordinate position of the region where the calculated difference is the minimum as the XY coordinate position of the work reference region WRoiPos (hereinafter, also referred to as “plane alignment”) is performed. Then, the alignment unit 132 indicates a signal obtained by combining the focus position PosZ (Z coordinate position) with the XY coordinate position of the work reference region WRoiPos specified by the plane alignment, and indicates the work reference region WRoiPos (XYZ coordinate position). It is output to the attitude estimation unit 15 as a signal.

The model registration unit 14 includes a model target registration unit (first registration unit) 141 and a model reference registration unit (second registration unit) 142. The model target registration unit 141 and the model reference registration unit 142 are provided, for example, in a computer (not shown).

The model target registration unit 141 registers the XYZ coordinate position of the region designated as the region on which the operator wants to acquire the focused image I Focus for the evaluation of the work 2 from the regions on the work 2 as the “model target region MRoiPat”. do. The operator can specify the model target area MRoiPat by operating the computer on the image displayed on the display and performing an operation such as surrounding the area to be specified.

The model reference registration unit 142 registers the XYZ coordinate position of the region designated as the region including the model reference image from the regions on the work 2 as the above-mentioned "model reference region MRoiPos". The operator can specify the model reference area MRoiPos by operating the computer on the image displayed on the display and performing an operation such as surrounding the area to be specified.

The image displayed on the display when the model target area MRoiPat and the model reference area MRoiPos are specified may be an image selected by the operator from a plurality of images included in the image stack Stack. , The image may be automatically selected by the computer.

The posture estimation unit 15 is the posture of the work 2 imaged as an evaluation target based on the work reference area WRoiPos calculated by the alignment unit 132 and the model reference area MRoiPos registered in advance by the model reference registration unit 142. To estimate.

In order to estimate the posture of the work 2 three-dimensionally, it is desirable that at least three or more work reference regions WRoiPos exist. Therefore, in the present embodiment, three or more model reference regions MRoiPos are registered in advance by the model reference registration unit 142. Then, the above-mentioned alignment unit 132 outputs three or more work reference regions WRoiPos corresponding to each of the three or more model reference regions MRoiPos to the posture estimation unit 15.

The posture estimation unit 15 includes a work posture estimation unit 151 and a focus region output unit 152. The work posture estimation unit 151 estimates the posture of the imaged work 2 based on the work reference region WRoiPos and the model reference region MRoiPos.

When the XYZ coordinate positions of three or more regions of the work 2 are known, the following equation (3) holds.

In the equation (3), each component (a11, a12, ..., A34) of the matrix A on the right side is an arbitrary constant. In the equation (3), when x, y, z represent the XYZ coordinate position of the arbitrary model reference region MRoiPos, and x', y', z'represent the XYZ coordinate position of the work reference region WRoiPos, the above equation ( The matrix A on the right side of 3) represents the three-dimensional posture P of the work 2. Therefore, the work is obtained by finding each component of the matrix A that can be approximated that the right side and the left side of the equation (3) are equal based on the combination of the positional relationships between the three or more model reference regions MRoiPos and the work reference region WRoiPos. The posture P of 2 can be estimated three-dimensionally. The above estimation method is merely an example, and the method for estimating the posture of the work 2 is not limited to the above estimation method.

The focus region output unit 152 is a model target in the image stack Stack based on the posture P of the work 2 estimated by the work posture estimation unit 151 and the model target area MRoiPat registered in advance by the model target registration unit 141. The XYZ coordinate position of the region corresponding to the region MRoiPat is calculated as the "work target region WRoiPat". The work target area WRoiPat is a partial area in which the focused image IFocus of the entire image of the work 2 is desired to be acquired, and is an area to be evaluated by the work 2.

The focused area output unit 152 outputs the calculated work target area WRoiPat to the outside. The work target area WRoiPat output to the outside is used, for example, for positioning the work 2. Further, the focused area output unit 152 also outputs the calculated work target area WRoiPat to the focused image acquisition unit 16.

The focused image acquisition unit 16 identifies and identifies the image most focused on the work target area WRoiPat from the plurality of images included in the image stack Stack by the same method as the image processing unit 13. The image in the work target area WRoiPat of the image is acquired as a focused image IFocus and output to the outside. The focused image I Focus is used, for example, for inspection of work 2. The focused image acquisition unit 16 may be provided outside the image acquisition device 1. Further, when the focused image I Focus is not used for the evaluation of the work 2, the focused image acquisition unit 16 can be omitted.

<Procedure for registering model area>
FIG. 2 is a flowchart showing an example of a procedure for registering a model area (model target area MRoiPat and model reference area MRoiPos). The flowchart of FIG. 2 is started in a state where the work 2 to be model-registered is set in the image acquisition device 1.

First, a process of acquiring an image stack Stack (hereinafter, also referred to as “model image stack MS Stack”) of the work 2 to be model-registered is executed (step S10). For example, the model image stack MS stack is acquired by capturing a plurality of images with the image pickup device 11 while moving the image pickup device 11 in the Z-axis direction using the moving device 12.

Next, the process of registering the model target area MRoiPat is executed (step S20). For example, the model target area MRoiPat is registered by the operator using the model target registration unit 141 to perform an operation such as surrounding the area to be registered as the model target area MRoiPat on the model image stack MS stack.

Next, the process of registering the model reference region MRoiPos is executed (step S30). For example, the model reference area MRoiPos is registered by the operator using the model reference registration unit 142 to perform an operation such as surrounding the area to be registered as the model reference area MRoiPos on the model image stack MS stack.

<Calculation procedure of work target area WRoiPat>
FIG. 3 is a flowchart showing an example of a calculation procedure of the work target area WRoiPat (a partial area in which the in-focus image IFocus of the entire image of the work 2 is desired to be acquired). The flowchart of FIG. 3 is started with the work 2 to be evaluated set in the image acquisition device 1.

First, a process of acquiring the image stack Stack of the work 2 to be evaluated (hereinafter, also referred to as “work image stack WS Stack”) is executed (step S11). For example, the work image stack WStack is acquired by capturing a plurality of images with the image pickup device 11 while moving the image pickup device 11 in the Z-axis direction using the moving device 12.

Next, the in-focus position acquisition unit 131 executes a process of calculating the in-focus position PosZ of the work image stack WStack (step S40). In this process, for example, in the image in the work image stack WStack, three or more calculation areas including three or more model reference areas MRoiPos registered in advance by the model reference registration unit 142 are extracted and extracted. The focal position PosZ is calculated for each of the three or more calculation areas. As a result, the same number (three or more) of focus positions PosZ as the number of model reference region MRoiPos is calculated.

In each calculation area extracted in step S40, for example, assuming that the set position of the work 2 at the time of model registration and the set position of the work 2 to be evaluated are not significantly different, each model reference. Each model reference region MRoiPos is set to a region expanded by a predetermined amount centering on the region MRoiPos. By doing so, each model reference region MRoiPos can be included in each calculation region.

Next, the alignment unit 132 performs a process of calculating the work reference region WRoiPos (step S50). In this process, the work reference region WRoiPos is calculated for each focus position PosZ calculated in step S40. In each process, the above-mentioned plane alignment is performed on the calculation area of the focus position PosZ to specify the XY coordinate position of the work reference area WRoiPos, and the focus position PosZ is set to the XY coordinate position of the specified work reference area WRoiPos. The signal in which the above is combined is defined as the work reference region WRoiPos (XYZ coordinate position). By sequentially performing the above processing for each focus position PosZ, the same number (three or more) of work reference regions WRoiPos as the number of model reference region MRoiPos is calculated.

Next, the work posture estimation unit 151 performs a process of estimating the posture of the imaged work 2 (step S60). In this process, for example, as described above, each of the matrices A in the above equation (3) is based on the combination of the positional relationships between the three or more model reference regions MRoiPos and the three or more work reference regions WRoiPos. By obtaining the components, the posture P of the work 2 is estimated three-dimensionally.

Next, the focused area output unit 152 performs a process of calculating the work target area WRoiPat (step S70). In this process, the area corresponding to the model target area MRoiPat in the work image stack WStuck is calculated as the work target area WRoiPat in consideration of the posture P of the work 2 estimated in step S60. When the work target area WRoiPat is not included in the work image stack WStack, the operator may be notified by using a buzzer or the like (not shown).

Finally, the in-focus image acquisition unit 16 performs a process of acquiring the image most focused on the work target area WRoiPat as the in-focus image IFocus (step S71).

As described above, according to the image acquisition device 1 according to the present embodiment, the work target is estimated by estimating the focus position PosZ, the work reference area WRoiPos, and the work posture P only for the model reference area MRoiPos registered in advance. The region WRoiPat and its focused image IFocus can be acquired. That is, the work target area WRoiPat and its in-focus image IFocus can be acquired without generating a omnifocal image. Therefore, the work target area WRoiPat and its in-focus image IFocus can be acquired with a smaller amount of calculation than in the case of generating a omnifocal image.

If a omnifocal image is generated, it is necessary to perform positioning in the plane direction and the height direction for each image acquisition area, and the amount of calculation increases linearly according to the size of the evaluation area. On the other hand, according to the image acquisition device 1 according to the present embodiment, since the omnifocal image is not generated, the amount of calculation is significantly reduced. As a result, the focused image I Focus of the work 2 to be evaluated can be acquired at high speed.

Embodiment 2.
In the above-described first embodiment, it is assumed that the set position of the work 2 at the time of model registration and the set position of the work 2 to be evaluated are not significantly different. However, in reality, it can be assumed that the set position of the work 2 at the time of model registration and the set position of the work 2 to be evaluated are significantly different.

Therefore, in the second embodiment, a reference area for alignment in the imaging field of view (hereinafter, also referred to as “field of view alignment area”) is registered in advance, and alignment (coarse) with respect to the field of view alignment area is performed. (Alignment) is performed, and then the same processing as in the first embodiment described above is performed. As a result, it is possible to cope with a case where the set position of the work 2 at the time of model registration and the set position of the work 2 to be evaluated are significantly different.

FIG. 4 is a diagram schematically showing the configuration of an image acquisition system including the image acquisition device 1A according to the second embodiment. The image acquisition device 1A adds a model field of view registration unit (third registration unit) 143 inside the model registration unit 14 and adds a field of view image processing unit 17 to the image acquisition device 1 shown in FIG. It was done. Since other structures, functions, and processes are the same as those of the image acquisition device 1 shown in FIG. 1 described above, the detailed description here will not be repeated.

The model field of view registration unit 143 is provided in, for example, a computer (not shown) and registers the field of view alignment area. The operator can register the field-of-view alignment area by operating a computer on the image displayed on the display and performing an operation such as surrounding an area to be specified. The field of view alignment area includes a calculation area MRoiV Focus of the field of view focus position at the time of model registration and a model field of view alignment area MRoiVPat. The model field of view registration unit 143 outputs the registered field of view alignment area to the field of view image processing unit 17.

The visual field image processing unit 17 includes a visual field focusing position acquisition unit (second acquisition unit) 171 and a visual field alignment unit 172. The visual field focusing position acquisition unit 171 calculates the visual field focusing position VFocus and outputs it to the visual field alignment unit 172 in the same manner as the visual field focusing position acquisition unit 131. The field of view alignment unit 172 outputs the field of view position VPos (field of view position deviation amount OffsetXYZ described later) in the same manner as the alignment unit 132.

FIG. 5 is a flowchart showing an example of the procedure for registering the model area according to the second embodiment. The flowchart of FIG. 5 is obtained by adding steps S80 and S81 to the flowchart of FIG. Since the other steps (steps having the same numbers as the steps shown in FIG. 2 above) have already been described, detailed description will not be repeated here.

After the acquisition of the model image stack MS stack (step S10), the registration of the model target area MRoiPat (step S20), and the registration of the model reference area MRoiPos (step S30), the field of view alignment area is set by the model field of view registration unit 143. The process of registering is performed (step S80). In this process, the calculation area MRoiVFocus of the field of view focus position at the time of model registration and the model field of view alignment area MRoiVPat are registered.

Next, a process of registering the model visual field focusing position MVFocus is performed (step S81). In this process, the focusing index is calculated using the field focusing focus position acquisition unit 171 for the calculation area MRoiV Focus of each image in the model image stack MStack, and the Z coordinate position of the image having the maximum focusing index is the model field of view. It is registered as the in-focus position MVFocus.

FIG. 6 is a flowchart showing an example of the calculation procedure of the work target area WRoiPat according to the second embodiment. In the flowchart of FIG. 6, steps S40 and S50 of FIG. 3 are changed to steps S41 and S51, respectively, and steps S90 and S100 are added. Since the other steps (steps having the same numbers as the steps shown in FIG. 3 above) have already been described, detailed description will not be repeated here.

After the work image stack WStack is acquired (step S11), the visual field focusing position acquisition unit 171 performs a process of calculating the work visual field focusing position WVFocus (step S90). In this process, the focus index is calculated for the calculation area MRoiVFocus of each image in the work image stack WStack, and the Z coordinate position of the image determined to have the highest focus based on the focus index is the work field focus. Calculated as position WVFocus.

Next, the field of view alignment unit 172 performs the field of view alignment (step S100). In this process, the work field of view alignment area WRoiVPat corresponding to the model field of view alignment area MRoiVPat is calculated by adding the depth position deviation amount OffsetZ of the work field of view focusing position WVFocus with respect to the model field of view focusing position MVFocus, and the work field of view is calculated. The amount of misalignment between the alignment region WRoiVPat and the model visual field alignment region MRoiVPat is calculated as the visual field misalignment amount OffsetXYZ. This field of view position shift amount OffsetXYZ corresponds to a three-dimensional position shift amount between the set position of the work 2 at the time of model registration and the set position of the work 2 to be evaluated. The above-mentioned visual field position VPos is synonymous with the visual field position deviation amount OffsetXYZ calculated in step S100.

Next, the in-focus position acquisition unit 131 executes a process of calculating the in-focus position PosZ of the work image stack WStack (step S41). In this process, the work reference area WRoiPos corresponding to the model reference area MRoiPos is calculated in consideration of the visual field position deviation amount OffsetXYZ calculated in step S100, and for each work reference area WRoiPos as in step S40 of FIG. 3 described above. The focal position PosZ of is calculated.

Next, the alignment unit 132 performs a process of calculating the work reference region WRoiPos (step S51). In this process, the work reference region WRoiPos is calculated by adding the visual field position deviation amount OffsetXYZ calculated in step S100, and the displacement amount for each work reference region WRoiPos is calculated in the same manner as in step S50 of FIG. 3 described above. ..

In the following, the process of estimating the posture of the work 2 described with reference to FIG. 3 (step S60), the process of calculating the work target area WRoiPat (step S70), and the process of acquiring the focused image IFocus of the work target area WRoiPat. (Step S71) is performed.

By doing so, in the image acquisition device 1A according to the second embodiment, the work target area WRoiPat of the work 2 to be evaluated and its focused image IFocus can be performed at high speed as in the above-described first embodiment. Can be obtained in. Further, the work target area WRoiPat can be calculated by adding the visual field position deviation amount OffsetXYZ between the set position of the work 2 at the time of model registration and the set position of the work 2 to be evaluated. Therefore, even when the set position of the work 2 at the time of model registration and the set position of the work 2 to be evaluated are significantly different, the work target area WRoiPat and its focused image IFocus can be acquired.

Embodiment 3.
In the above-described first and second embodiments, it is assumed that there is a symbol that can be registered as a reference for alignment in the imaging field of view when the imaging device 11 or the pedestal 3 is moved in the Z-axis direction. There is. However, in reality, since the size of the work 2 in the XY-axis direction is large, it can be registered as a reference for alignment in the imaging field of view when the image pickup device 11 or the pedestal 3 is moved in the Z-axis direction. It can be assumed that there is no design.

In view of the above points, in the third embodiment, the work 2 has a plurality of imaging fields of view slid in the XY-axis direction by moving the image pickup device 11 or the pedestal 3 in the Z-axis direction as well as the XY-axis direction. Is imaged, and a reference for alignment is registered in the image of one of a plurality of imaging fields of view.

FIG. 7 is a diagram schematically showing the configuration of an image acquisition system including the image acquisition device 1B according to the third embodiment. The image acquisition device 1B is an image acquisition device 1A shown in FIG. 4 with an imaging position recording unit 122 added. Since other structures, functions, and processes are the same as those of the image acquisition device 1A described above, the detailed description here will not be repeated.

The imaging position recording unit 122 records the moving unit position information Enc when the work 2 is imaged as an imaging position signal VPos, and outputs the recorded imaging position signal VPos to the work posture estimation unit 151. In the third embodiment, as described above, by moving the image pickup device 11 or the pedestal 3 in the Z-axis direction and also in the XY-axis direction, a plurality of imaging fields of view slid in the XY-axis direction can be obtained. It is assumed that the work 2 is imaged.

The work posture estimation unit 151 has a function of receiving the image pickup position signal VPos from the image pickup position recording unit 122 in addition to the functions described in the first and second embodiments.

FIG. 8 is a flowchart showing an example of the procedure for registering the model area according to the third embodiment. The flowchart of FIG. 8 is obtained by adding steps S110 and S120 to the flowchart of FIG. Since the other steps (steps having the same numbers as the steps shown in FIG. 5 above) have already been described, detailed description will not be repeated here.

First, it is determined whether or not the imaging of all the visual fields in the XY axis directions is completed (step S110). When the imaging of all the visual fields in the XY-axis direction is not completed (NO in step S110), the imaging position recording unit 122 performs a process of recording the current imaging position signal VPos (step S120). After that, the processes of steps S10 to S81 shown in FIG. 5 described above are performed. After that, the process is returned to step S110, and the processes of steps S110, S120, and S10 to S81 are repeated for each field of view in the XY axis direction until the imaging of all the fields of view in the XY axis direction is completed. Then, when the imaging of all the visual fields in the XY axis directions is completed (YES in step S110), the model registration process is completed.

FIG. 9 is a flowchart showing an example of the calculation procedure of the work target area WRoiPat according to the third embodiment. Of the steps shown in FIG. 9, the steps having the same numbers as the steps shown in FIG. 6 described above have already been described, and detailed description thereof will not be repeated here.

First, it is determined whether or not the imaging of all the visual fields in the XY axis directions is completed (step S111). When the imaging of all the visual fields in the XY-axis direction is not completed (NO in step S111), the imaging position recording unit 122 performs a process of recording the current imaging position signal VPos (step S121). After that, the processes of steps S11 to S51 shown in FIG. 6 described above are performed. After that, the process is returned to step S111, and the processes of steps S111, S121, and S11 to S51 are repeated for each field of view until the imaging of all the fields of view in the XY axis directions is completed.

Then, when the imaging of all the visual fields in the XY axis directions is completed (YES in step S111), the process of estimating the posture of the work 2 is performed (step S61). In this process, the focus position PosZ of each field of view calculated in step S41, the work reference region WRoiPos of each field of view calculated in step S51, and the imaging position signal VPos of each field of view recorded in step S121 are used. The posture of the work 2 is estimated.

After that, as in the first and second embodiments, the process of calculating the work target area WRoiPat (step S70) and the process of acquiring the focused image IFocus (step S71) are performed.

By doing so, in the image acquisition device 1B according to the third embodiment, the work target area WRoiPat of the work 2 to be evaluated and its focused image IFocus are similarly to the above-described first and second embodiments. Can be obtained at high speed. Further, the image acquisition device 1B according to the third embodiment captures the work 2 in a plurality of imaging fields of view slid in the XY axis directions, and aligns the work 2 with an image of any of the plurality of fields of view. I try to register the standard. Therefore, even when there is no structure that can be registered as a reference for alignment in one imaging field of view, the work target area WRoiPat and its focused image IFocus can be acquired.

Embodiment 4.
In the first, second, and third embodiments, it is assumed that there is no positional deviation between the alignment reference of the work 2 and the image pickup region by the image pickup apparatus 11. However, in the case where the work 2 is a module composed of a plurality of parts assembled, it is assumed that a slight misalignment may occur between the alignment reference and the imaging region due to the influence of the assembly error of the work 2. obtain.

In view of the above points, in the fourth embodiment, the alignment reference for correcting the misalignment with respect to the alignment reference is registered for each imaging region.

FIG. 10 is a diagram schematically showing the configuration of an image acquisition system including the image acquisition device 1C according to the fourth embodiment. The image acquisition device 1C is an image acquisition device 1 shown in FIG. 1 described by adding a site reference registration unit 144 and a site image processing unit 18 inside the model registration unit 14. Since other structures, functions, and processes are the same as those of the image acquisition device 1 shown in FIG. 1 described above, the detailed description here will not be repeated.

The site reference registration unit 144 is provided, for example, in a computer (not shown), and registers a "site alignment area" that serves as a reference for alignment of each part of the work 2. It is assumed that the operator registers the site alignment area by operating a computer on the image displayed on the display and performing an operation such as surrounding the area to be specified. The site alignment area includes the calculation area MRoiP Focus of the site focus position at the time of model registration and the model site alignment area MRoiPPat. The site reference registration unit 144 outputs the registered site alignment area to the site image processing unit 18.

The site image processing unit 18 includes a site focusing position acquisition unit 181 and a site alignment unit 182.

The site focusing position acquisition unit 181 calculates the site focusing position PFOcus and outputs it to the site alignment unit 182 by the same method as the site focusing position acquisition unit 131. The site alignment unit 182 calculates the site position Ppos (site position deviation amount POffsetXYZ, which will be described later) in the same manner as the alignment unit 132, and outputs it to the in-focus region output unit 152.

The focused area output unit 152 according to the fourth embodiment has a function of receiving the site misalignment amount POffsetXYZ in addition to the functions of the focused area output unit 152 of the first to third embodiments described above.

FIG. 11 is a flowchart showing an example of the procedure for registering the model area according to the fourth embodiment. The flowchart of FIG. 11 is obtained by adding steps S130 and S131 to the flowchart of FIG. Since the other steps (steps having the same numbers as the steps shown in FIG. 2 above) have already been described, detailed description will not be repeated here.

In the registration of the model area according to the fourth embodiment, after the acquisition of the model image stack MS stack (step S10), the registration of the model target area MRoiPat (step S20), and the registration of the model reference area MRoiPos (step S30) are performed. The process of registering the site alignment area is performed (step S130). In this process, the site reference registration unit 144 registers the calculation area MRoiPFocus of the site focus position at the time of model registration and the model site alignment area MRoiPPat.

Next, a process of registering the model site focusing position MPFocus is performed (step S131). In this process, the focusing index is calculated by using the part focusing position acquisition unit 181 for the part focus position calculation area MRoiPFocus of each image in the model image stack MStack, and the calculation area of each image in the model image stack MStack. For MRoiV Focus, the focus index is calculated using the visual field focus position acquisition unit 171 and the obtained focus position (Z coordinate position of the image with the maximum focus index) is registered as the model site focus position MPFocus. NS.

FIG. 12 is a flowchart showing an example of the calculation procedure of the work target area WRoiPat according to the fourth embodiment. The flowchart of FIG. 12 is obtained by adding steps S140 and S150 to the flowchart of FIG. 3 described above. Since the other steps (steps having the same numbers as the steps shown in FIG. 3 above) have already been described, detailed description will not be repeated here.

After the work image stack WStack is acquired (step S11), a process of calculating the work site focusing position WP Focus is performed (step S140). In this process, the site focus position acquisition unit 181 calculates the work site focus position WP Focus for the area registered as the site focus position calculation area MRoiP Focus in the site reference registration unit 144.

After the execution of step 140, the process of step 150 is performed. In step S150, for the region registered as the model site alignment region MRoiPPat by the site reference registration unit 144, the depth position in consideration of the site depth position shift amount POffsetZ of the work site focus position WP Focus with respect to the model site focus position MPFocus. The work site alignment area WRoiPPat is calculated based on the above, and the amount of misalignment between the work site alignment area WRoiPPat and the model site alignment area MRoiPPat is calculated as the site misalignment amount POffsetXYZ.

Next, the process of calculating the in-focus position PosZ (step S40), the process of calculating the work reference region WRoiPos (step S50), and the process of estimating the posture P of the work 2 (step S60) described with reference to FIG. 3 above. Is performed.

In the process of calculating the work target area WRoiPat (step S70) in the fourth embodiment, the posture P of the work 2 estimated in step S60, the position of the model target area MRoiPat, and the site position calculated in step 150. The work target area WRoiPat in the work image stack WStack is calculated based on the deviation amount POffsetXYZ.

Finally, a process (step S71) of acquiring the focused image I Focus of the work target area WRoiPat described with reference to FIG. 3 is performed.

By doing so, in the image acquisition device 1C according to the fourth embodiment, the work target area WRoiPat of the work 2 to be evaluated and its focusing focus are the same as those of the above-described first, second, and third embodiments. The image I Focus can be acquired at high speed. Further, the image acquisition device 1C according to the fourth embodiment registers the alignment reference for correcting the positional deviation for each imaging region. Therefore, when an image is taken of a module or the like formed by assembling a plurality of parts, it is possible to absorb the assembly error and acquire a focused image without being affected by the positional deviation of each part.

Embodiment 5.
In the first, second, third, and fourth embodiments, it is assumed that the operator performs the work of registering the model area for each area for which the in-focus image is desired to be acquired. However, when there are a large number of areas for which a focused image is desired to be acquired for the work 2, there is a concern that the registration work will take an enormous amount of time.

In view of the above points, in the fifth embodiment, the model area is based on the image area candidates by using the machine learning function that creates the image area candidates based on the images of the image area registered in the past. To register. As a result, the model area can be efficiently registered even when there are a large number of areas for which the focused image is to be acquired.

FIG. 13 is a diagram schematically showing the configuration of an image acquisition system including the image acquisition device 1D according to the fifth embodiment. The image acquisition device 1D is obtained by adding the learning unit 19 to the image acquisition device 1 shown in FIG. 1 described above. Since other structures, functions, and processes are the same as those of the image acquisition device 1 shown in FIG. 1 described above, the detailed description here will not be repeated.

The learning unit 19 includes a sample image generation unit 191, a machine learning unit 192, a learning data recording unit 193, an image cutting unit 194, and a determination unit 195. The learning unit 19 is not necessarily limited to being built in the image acquisition device 1D, and may exist outside the image acquisition device 1D (for example, a cloud server).

Based on the image stack Stack and the model target area MRoiPat, the sample image generation unit 191 determines whether the sample image MRoiPatImg, which is an image of the model target area MRoiPat in the image stack Stack, and the sample image MRoiPatImg are the model target area MRoiPat. Outputs the combination with the truth value isMRoiPat, which is the truth value of. Sample image when the boolean value isMRoiPat is false MRoiPatImg outputs a region having the same size and not including the model target region MRoiPat based on the model target region MRoiPat.

The machine learning unit 192 learns the probability MRoiPatP indicating the probability of being a model target region according to the combination of the sample image MRoiPatImg and its true / false value isMRoiPat, and records the learning result as learning data C in the learning data recording unit 193. Further, the machine learning unit 192 inputs the learning data C recorded in the learning data recording unit 193 and the partial image data RoiImg from the image cutting unit 194, and determines the probability that the partial image data RoiImg is the model target area. It functions as an inference means for outputting the indicated probability MRoiPatP. The machine learning unit 192 is, for example, a device that performs so-called supervised learning according to a neural network model.

Here, supervised learning is a model in which a large amount of data sets of a certain input and output results (labels) are given to a learning device, features in those data sets are learned, and the output results are estimated from the inputs. To say. A neural network is composed of an input layer composed of a plurality of neurons, an intermediate layer (hidden layer) composed of a plurality of neurons, and an output layer composed of a plurality of neurons. The intermediate layer may be one layer or two or more layers.

FIG. 14 is a diagram showing an example of a neural network model. In the case of a three-layer neural network as shown in FIG. 14, when a plurality of inputs are input to the input layers (X1 to X3), the result of multiplying the values by the weights W1 (w11 to w16) is the intermediate layer (the intermediate layer (X1 to X3). It is input to Y1, Y2), and the result of multiplying the result by the weight W2 (w21 to w26) is output from the output layer (Z1 to Z3). This output result depends on the values of the weights W1 and W2.

In the present application, the neural network learns the above-mentioned probability MRoiPatP by so-called supervised learning according to the data set created based on the combination of the sample image MRoiPatImg and the boolean value isMRoiPat, and learns the learning result as learning data C. Record in the data recording unit 193.

Returning to FIG. 13, the learning data recording unit 193 has a function of storing the learning data C which is the learning result of the machine learning unit 192.

Further, the learning of the machine learning unit 192 may be performed according to the data sets created for the plurality of types of work 2. The machine learning unit 192 may acquire data sets from a plurality of workpieces 2 used at the same site, or may use data sets collected from a plurality of machine tools operating independently at different sites. Then, the probability MRoiPatP may be learned. Further, the work 2 for collecting the data set can be added to the target on the way, or conversely, can be removed from the target.

The image cutting unit 194 cuts out the partial image data RoiImg based on the image stack Stack and outputs it to the machine learning unit 192. The method of cutting out the partial image data RoiImg may be a method of cutting out images at equal pitches and equal sizes in the vertical and horizontal directions with respect to each image constituting the image stack Stack.

The determination unit 195 determines whether or not the partial image data RoiImg is a candidate for the model target area based on the probability MRoiPatP from the machine learning unit 192, and the partial image data RoiImg determined to be a candidate for the model target area. Is output as the estimated model target area MRoiPatE. For example, the estimated model target region MRoiPatE may be output as a result of comparing the probability MRoiPatP with the preset threshold value Th.

The model target registration unit 141 according to the fifth embodiment receives the image stack Stack and the estimated model target area MRoiPatE as inputs, and outputs the model target area MRoiPat.

The model target registration unit 141 is provided in a computer (not shown). If there is an estimated model target area MRoiPatE, the estimated model target area MRoiPatE is presented to the operator. The operator operates a computer to register the estimated model target area MRoiPatE as the model target area MRoiPat. Alternatively, it is assumed that the estimated model target area MRoiPatE is modified on the screen on which the image is displayed and registered as the model target area MRoiPat.

FIG. 15 is a flowchart showing an example of the procedure for registering the model area according to the fifth embodiment. The flowchart of FIG. 15 is obtained by adding steps S160, S170, and S180 to the flowchart of FIG. Since the other steps (steps having the same numbers as the steps shown in FIG. 2 above) have already been described, detailed description will not be repeated here.

In the model registration according to the fifth embodiment, step S160 is performed after step S10. In step S160, it is confirmed whether or not the learning data C that can be used by the machine learning unit 192 is in the learning data recording unit 193. If there is no training data C (NO in step S160), the model area is registered in step S20, that is, in the same manner as the procedure for registering the model area in the first embodiment.

If there is learning data C (YES in step S160), step 170 is performed. In step S170, the partial image data RoiImg is cut out by the image cutting unit 194 with respect to the image stack Stack acquired in step S10, and is input to the machine learning unit 192 together with the learning data C. The machine learning unit 192 calculates the probability MRoiPatP in which the partial image data RoiImg is the model target region. Based on this probability MRoiPatP, the determination unit 195 determines whether or not the partial image data RoiImg is a candidate for the model target area, and estimates the partial image data RoiImg determined to be a candidate for the model target area MRoiPatE. Is output to the model target registration unit 141.

In step S20 according to the fifth embodiment, the operator registers the region for which the focused image is to be acquired as the model target region MRoiPat while referring to the estimated model target region MRoiPatE.

Further, in the fifth embodiment, after the completion of step S30, step S180 is carried out. In step S180, in order to carry out the learning procedure of the model area shown in FIG. 16 described later, the combination of the model image stack MS stack and the model target area MRoiPat is recorded in the sample image generation unit 191.

FIG. 16 is a flowchart showing an example of the learning procedure of the model area in the fifth embodiment. In the learning of the model area, first, step S200 is carried out. In step S200, based on the combination of the model image stack MS stack and the model target area MRoiPat recorded in advance in step S180 of FIG. 15, the sample image generation unit 191 sets the sample image MRoiPatImg and the boolean value isMRoiPat. Output the combination of. Next, the machine learning unit 192 learns a data set consisting of a group of combinations of the sample image MRoiPatImg and the boolean value isMRoiPat, and infers the probability MRoiPatP which is the probability that the given image is the model target region. To learn.

Subsequently, step S201 is carried out. In step S201, the learning data C is recorded in the learning data recording unit 193 so that the learning data C, which is the learning result of the machine learning unit 192, can be used in the model area registration.

By doing so, in the image acquisition device 1D according to the fifth embodiment, the work target area WRoiPat of the work 2 to be evaluated and its focused image as in the above-described first, second, and third embodiments. IFocus can be acquired at high speed. Further, the image acquisition device 1D according to the fifth embodiment uses a machine learning function that creates a candidate for the imaging region based on the image of the imaging region registered in the past, and uses the machine learning function to create the region based on the candidate for the imaging region. to register. As a result, even when there are a large number of areas for which the focused image is to be acquired, the registration work can be performed without hassle.

It should be considered that the embodiments disclosed this time are exemplary in all respects and not restrictive. The scope of the present disclosure is indicated by the scope of claims rather than the above description, and is intended to include all modifications within the meaning and scope of the claims.

1,1A, 1B, 1C, 1D image acquisition device, 2 works, 3 pedestals, 11 imaging device, 12 moving device, 13 image processing unit, 14 model registration unit, 15 posture estimation unit, 16 focused image acquisition unit, 17 Field image processing unit, 18 site image processing unit, 19 learning unit, 122 imaging position recording unit, 131 focus position acquisition unit, 132 alignment unit, 141 model target registration unit, 142 model reference registration unit, 143 model field image registration unit 144 part reference registration part, 151 work posture estimation part, 152 focus area calculation part, 171 field focus position acquisition part, 172 field alignment part, 181 part focus position acquisition part, 182 part alignment part, 191 sample Image generation unit, 192 machine learning unit, 193 learning data recording unit, 194 image cutting unit, 195 judgment unit.

Claims (10)

It is an image acquisition device that acquires an image of a work.
A first registration unit that registers a first area to be acquired as a focused image using a model image that is an image of a work that is a target of model registration, and a first registration unit.
A second registration unit that registers a second region used for alignment using the model image, and a second registration unit.
An imaging device that captures a work image, which is an image of the work to be evaluated,
A moving device configured so that at least one of the imaging device and the work can be moved in the optical axis direction of the imaging device.
A first acquisition unit that acquires the depth position of the image having the highest degree of focus among a plurality of work images having different focal points captured by the imaging device while operating the moving device as the in-focus position.
An alignment unit that identifies a region corresponding to the second region in the work image by performing alignment with the second region on the work image at the in-focus position.
A posture estimation unit configured to be able to acquire information indicating the posture of the work at the time of imaging based on the positional relationship between the second region in the model image and the region corresponding to the second region in the work image.
A region output unit that outputs the position of a region corresponding to the first region in the work image using information indicating the posture of the work, and a region output unit.
An image acquisition device. A third registration unit that registers a third region used for positioning the imaging field of view using the model image, and a third registration unit.
A second acquisition unit that acquires the depth position of the image having the highest degree of focus in the third region of the plurality of work images having different focal points captured by the imaging device as the visual field focusing position.
By aligning the region corresponding to the third region in the work image at the in-focus position with the third region in the model image, the region corresponding to the third region in the work image and the region can be described. Further provided with a visual field alignment unit for acquiring the amount of misalignment with the position of the third region in the model image.
The image acquisition device according to claim 1, wherein the first acquisition unit acquires the in-focus position in consideration of the amount of misalignment. A position recording unit for recording the imaging position in the optical axis direction of the imaging device for each field of view is further provided.
The image acquisition device according to claim 1 or 2, wherein the posture estimation unit acquires information indicating the posture of the work based on the imaging position. A site area registration unit for registering a site area used for positioning each part of the model image, and a site area registration unit.
A site focus position acquisition unit that acquires a site focus position, which is a site focus position for each part of the model image, and a site focus position acquisition unit.
Any one of claims 1 to 3, further comprising a region corresponding to the site region in the work image of the site focusing position and a site alignment portion for performing alignment with the site region in the model image. The image acquisition device described in the section. Based on a plurality of combinations of the first region registered by the first registration unit and the image data captured by the imaging device, the correspondence between the first region and the image data is machine-learned, and the result of machine learning is obtained. The image acquisition device according to any one of claims 1 to 4, further comprising a machine learning unit for determining whether or not the given image data is a candidate for the first region. This is an image acquisition method for acquiring an image of a work.
The step of registering the first region to be the target for acquiring the in-focus image using the model image which is the image of the work to be the target of model registration, and
A step of registering a second region used for alignment using the model image, and
The step of taking a work image, which is an image of the work to be evaluated, with an imaging device,
A step of acquiring the depth position of the image having the highest degree of focus among a plurality of work images having different focal points captured by the imaging device as the in-focus position, and
A step of identifying a region corresponding to the second region in the work image by performing alignment with the second region on the work image at the focused position, and a step of identifying the region corresponding to the second region in the work image.
A step of acquiring information indicating the posture of the work at the time of imaging based on the positional relationship between the second region in the model image and the region corresponding to the second region in the work image, and
A step of outputting the position of a region corresponding to the first region in the work image using information indicating the posture of the work, and a step of outputting the position of the region corresponding to the first region.
Image acquisition methods, including. A step of registering a third region used for positioning the imaging field of view using the model image, and
A step of acquiring the depth position of the image having the highest degree of focusing in the third region among the plurality of work images having different focal points captured by the imaging device as the visual field focusing position.
Further including a step of acquiring the amount of misalignment between the region corresponding to the third region in the work image at the in-focus position and the third region in the model image.
The image acquisition method according to claim 6, wherein the step of acquiring the in-focus position includes a step of acquiring the in-focus position in consideration of the amount of misalignment. Further including a step of recording the imaging position in the optical axis direction of the imaging device for each field of view.
The image acquisition method according to claim 6 or 7, wherein the step of acquiring the information indicating the posture of the work is to acquire the information indicating the posture of the work based on the imaging position. A step of registering a part region used for alignment of each part of the model image, and
The step of acquiring the site focusing position, which is the focus position for each part of the model image, the position of the region corresponding to the site region in the work image of the site focus position, and the position of the site region in the model image. The image acquisition method according to any one of claims 6 to 8, further comprising a step of performing matching. A step of machine learning the correspondence between the first region and the image data based on a plurality of combinations of the registered first region and the image data captured by the imaging device.
The image acquisition method according to any one of claims 6 to 9, further comprising a step of determining whether or not the given image data is a candidate for the first region using the result of machine learning.

Images