JP7077485B2 - Image acquisition device and image acquisition method - Google Patents

Description

The present disclosure relates to a technique for capturing a focused image of a work to be imaged.

Generally, when an image pickup device (camera) using a high-magnification lens captures an image pickup target, the depth of field (the range that appears to be in focus in the image) becomes shallow. Depending on the unevenness of the work to be imaged, the evaluation target portion in the work may not be focused, which tends to adversely affect the evaluation (inspection, positioning, etc.) of the work.

As a countermeasure, there is known a method of creating an omnifocal image of a work to be imaged and performing an evaluation during the work based on the omnifocal image (see, for example, Japanese Patent Application Laid-Open No. 2016-21115). A omnifocal image is an image in which multiple images with different focal points are taken while changing the distance between the work and the image pickup device in the optical axis direction of the image pickup device, and the degree of focus (degree of focus) is determined for each local area. The whole image is reconstructed by evaluating and combining local images with a high degree of focus. The omnifocal image created in this way is almost in focus at all pixels.

Japanese Unexamined Patent Publication No. 2016-21115

In the method disclosed in Japanese Patent Application Laid-Open No. 2016-21115, it is necessary to generate a omnifocal image every time the work to be evaluated is imaged. When the unevenness of the work is large with respect to the depth of field, a omnifocal image is generated for a large number of images, and the amount of calculation of image processing becomes large. Therefore, there is a problem that the omnifocal image cannot be generated at high speed and it takes time to evaluate the work.

The present disclosure has been made to solve the above-mentioned problems, and an object thereof is to enable high-speed acquisition of a focused image of a work to be evaluated.

The image acquisition device according to the present disclosure acquires an image of the work. This image acquisition device aligns with the first registration unit that registers the first region to acquire the in-focus image using the model image that is the image of the work to be model-registered, and the model image. A second registration unit for registering a second region used for First acquisition to acquire the depth position of the image with the highest degree of focus among a plurality of work images with different focal points captured by the image pickup device while operating the moving device configured in The alignment part that specifies the area corresponding to the second area in the work image by performing the alignment with the second area with respect to the work image at the in-focus position, and the second area and the work in the model image. A posture estimation unit configured to be able to acquire information indicating the posture of the work at the time of imaging based on the positional relationship with the region corresponding to the second region in the image, and a second position in the work image using information indicating the posture of the work. It includes an area output unit that outputs the position of the area corresponding to one area.

According to the present disclosure, it is possible to acquire a focused image of the work to be evaluated at high speed.

It is a figure (the 1) which shows the structure of the image acquisition system roughly. It is a flowchart (the 1) which shows an example of the registration procedure of a model area. It is a flowchart (the 1) which shows an example of the calculation procedure of a work target area. It is a figure (the 2) which shows the structure of an image acquisition system. It is a flowchart (2) which shows an example of the registration procedure of a model area. It is a flowchart (2) which shows an example of the calculation procedure of a work target area. It is a figure (the 3) which shows the structure of an image acquisition system schematically. It is a flowchart (3) which shows an example of the registration procedure of a model area. It is a flowchart (3) which shows an example of the calculation procedure of a work target area. It is a figure (4) which shows the structure of an image acquisition system schematicly. It is a flowchart (4) which shows an example of the registration procedure of a model area. It is a flowchart (4) which shows an example of the calculation procedure of a work target area. It is a figure (the 5) which shows the structure of an image acquisition system. It is a figure which shows an example of the model of a neural network. It is a flowchart (the 5) which shows an example of the registration procedure of a model area. It is a flowchart which shows an example of the learning procedure of a model area.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The same or corresponding parts in the drawings are designated by the same reference numerals and the description thereof will not be repeated.

Embodiment 1.
FIG. 1 is a diagram schematically showing a configuration of an image acquisition system including an image acquisition device 1 according to the present embodiment. This image acquisition system includes an image acquisition device 1 and a pedestal 3 on which a work 2 to be imaged is installed.

The image acquisition device 1 includes an image pickup device 11, a moving device 12, an image processing unit 13, a model registration unit 14, a posture estimation unit 15, and a focused image acquisition unit 16.

The image pickup device 11 includes a camera 111 and a trigger generator 112. The camera 111 photographs the work 2 using an image pickup device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor). The trigger generator 112 outputs a trigger input signal Trig indicating the imaging timing of the camera 111 to the camera 111. The camera 111 photographs the work 2 based on the trigger input signal Trig from the trigger generator 112.

The moving device 12 includes an actuator 121. The actuator 121 is configured to be able to adjust the relative distance between the work 2 and the image pickup device 11 in the optical axis direction by moving the image pickup device 11 (camera 111) in the optical axis direction. In the following, as shown in FIG. 1, the direction along the optical axis direction of the image pickup apparatus 11 is "Z-axis direction", and the directions perpendicular to the optical axis direction and perpendicular to each other are "X-axis direction" and "Y", respectively. Also called "axial direction".

The moving device 12 has a function of moving the work 2 in the X-axis direction and the Y-axis direction (planar direction) by moving the pedestal 3 on which the work 2 is installed along the X-axis direction and the Y-axis direction. You may have. In this embodiment, it is assumed that the XY coordinate positions (positions in the X-axis direction and the Y-axis direction) of the pedestal 3 are fixed at the time of imaging of the work 2.

Further, the moving device 12 may have a function of adjusting the relative distance between the work 2 and the image pickup device 11 in the Z-axis direction by moving the pedestal 3 on which the work 2 is installed along the Z-axis direction. .. In the following, for convenience of explanation, it is assumed that the Z coordinate position (position in the Z axis direction) of the pedestal 3 is fixed at the time of imaging of the work 2.

The moving device 12 may have a function of outputting information indicating the XYZ coordinate position of the image pickup device 11 to the trigger generation device 112 as the moving unit position information Enc. In this case, the trigger generator 112 may output the trigger input signal Trig to the camera 111 based on the moving unit position information Enc. For example, when an image is taken on a work 2 while the image pickup device 11 is moved in the Z-axis direction, the trigger generator 112 determines that the Z coordinate position of the image pickup device 11 has changed by a predetermined amount based on the moving portion position information Enc. The trigger input signal Trig may be output to the camera 111 each time. By doing so, it is possible to capture a plurality of images while moving the image pickup device 11 in the Z-axis direction.

When a plurality of images are captured while the image pickup device 11 is moved in the Z-axis direction, the image pickup device 11 registers the captured plurality of images (hereinafter, also referred to as “image stack Stack”) with the image processing unit 13 and a model. Output to unit 14.

The image processing unit 13 includes a focusing position acquisition unit (first acquisition unit) 131 and an alignment unit 132. The in-focus position acquisition unit 131 calculates an in-focus index for each of a plurality of images included in the image stack-sack, and based on the calculated magnitude of the in-focus index, the in-focus position of the image stack-sack ( Depth position) PosZ is calculated. The area of each image for which the focus index is calculated may be the entire area of each image or a predetermined local area.

The "focused position PosZ" can be represented by, for example, the Z coordinate position of the image pickup apparatus 11 when the most focused image is captured among the plurality of images included in the image stack Stack. Therefore, the "focus index" used in the calculation of the focus position PosZ can be calculated based on the image obtained by capturing the work 2 with the image pickup device 11, and the relative distance between the image pickup device 11 and the work 2 in the Z-axis direction. It is desirable that it is an index that changes according to the above and takes the maximum or minimum when the degree of focus is the highest.

For example, the in-focus position acquisition unit 131 can calculate the sum of the elements of the image obtained by convolving the matrices shown in the following equations (1) and (2) for each pixel of the image as the in-focus index. ..

When the focus index is calculated by the above method, the focus position acquisition unit 131 selects the image having the maximum focus index among the plurality of images included in the image stack Stack, and the selected image is selected. The Z coordinate position of the image pickup apparatus 11 at the time of image pickup is calculated as the in-focus position PosZ. The in-focus position acquisition unit 131 outputs the calculated in-focus position PosZ to the in-focus position 132.

The alignment unit 132 selects an image corresponding to the in-focus position PosZ from the image stack Stack acquired from the image pickup apparatus 11, and calculates the work reference region WRoiPos in the selected image.

The "work reference area WRoiPos" is an area corresponding to the model reference area MRoiPos in the image stack Stack, and is specified by the XYZ coordinate position. The "model reference area MRoiPos" is an area including a model reference image (for example, a boundary of a pattern on the work 2) as a reference for alignment, and is specified by an XYZ coordinate position. The model reference region MRoiPos is pre-registered by the model reference registration unit 142 described later.

For example, the alignment unit 132 sequentially calculates the difference between each region of the image stack Stack and the image in the model reference region MRoiPos while shifting the image in the model reference region MRoiPos with respect to the image of the image stack Stack in the XY axis direction. Then, a process of specifying the XY coordinate position of the area where the calculated difference is the minimum as the XY coordinate position of the work reference area WRoiPos (hereinafter, also referred to as “planar alignment”) is performed. Then, the alignment unit 132 indicates a signal obtained by combining the focus position PosZ (Z coordinate position) with the XY coordinate position of the work reference region WRoiPos specified by the plane alignment, and indicates the work reference region WRoiPos (XYZ coordinate position). It is output to the attitude estimation unit 15 as a signal.

The model registration unit 14 includes a model target registration unit (first registration unit) 141 and a model reference registration unit (second registration unit) 142. The model target registration unit 141 and the model reference registration unit 142 are provided, for example, in a computer (not shown).

The model target registration unit 141 registers the XYZ coordinate position of the area designated by the operator as the area on the work 2 from which the focused image I Focus is to be acquired for the evaluation of the work 2 as the “model target area MRoiPat”. do. The operator can specify the model target area MRoiPat by operating the computer on the image displayed on the display and performing an operation such as surrounding the area to be specified.

The model reference registration unit 142 registers the XYZ coordinate position of the area on the work 2 designated as the area including the model reference image as the above-mentioned "model reference area MRoiPos". The operator can specify the model reference area MRoiPos by operating the computer on the image displayed on the display and performing an operation such as surrounding the area to be specified.

The image displayed on the display when the model target area MRoiPat and the model reference area MRoiPos are specified may be an image selected by the operator from a plurality of images included in the image stack Stack. , The image may be automatically selected by the computer.

The posture estimation unit 15 is the posture of the work 2 imaged as an evaluation target based on the work reference area WRoiPos calculated by the alignment unit 132 and the model reference area MRoiPos registered in advance by the model reference registration unit 142. To estimate.

In order to estimate the posture of the work 2 three-dimensionally, it is desirable that at least three work reference regions WRoiPos exist. Therefore, in the present embodiment, three or more model reference regions MRoiPos are registered in advance by the model reference registration unit 142. Then, the above-mentioned alignment unit 132 outputs three or more work reference regions WRoiPos corresponding to each of the three or more model reference regions MRoiPos to the posture estimation unit 15.

The posture estimation unit 15 includes a work posture estimation unit 151 and a focal point region output unit 152. The work posture estimation unit 151 estimates the posture of the imaged work 2 based on the work reference region WRoiPos and the model reference region MRoiPos.

When the XYZ coordinate positions of three or more regions of the work 2 are known, the following equation (3) holds.

In the equation (3), each component (a11, a12, ..., A34) of the matrix A on the right side is an arbitrary constant. In the equation (3), when x, y, z represent the XYZ coordinate position of the arbitrary model reference region MRoiPos, and x', y', z'represent the XYZ coordinate position of the work reference region WRoiPos, the above equation ( The matrix A on the right side of 3) represents the three-dimensional posture P of the work 2. Therefore, the work is obtained by finding each component of the matrix A that can be approximated to have the right side and the left side of the equation (3) equal to each other based on the combination of the positional relationship between the three or more model reference regions MRoiPos and the work reference region WRoiPos. The posture P of 2 can be estimated three-dimensionally. The above estimation method is merely an example, and the method for estimating the posture of the work 2 is not limited to the above estimation method.

The focused area output unit 152 is a model target in the image stack Stack based on the posture P of the work 2 estimated by the work posture estimation unit 151 and the model target area MRoiPat registered in advance by the model target registration unit 141. The XYZ coordinate position of the region corresponding to the region MRoiPat is calculated as the "work target region WRoiPat". The work target area WRoiPat is a partial area in which the focused image I Focus of the entire image of the work 2 is desired to be acquired, and is an area to be evaluated by the work 2.

The in-focus area output unit 152 outputs the calculated work target area WRoiPat to the outside. The work target area WRoiPat output to the outside is used, for example, for positioning the work 2. Further, the in-focus area output unit 152 also outputs the calculated work target area WRoiPat to the in-focus image acquisition unit 16.

The focused image acquisition unit 16 identifies and identifies the image most focused on the work target area WRoiPat from among the plurality of images included in the image stack Stack by the same method as the image processing unit 13. The image in the work target area WRoiPat of the image is acquired as a focused image IFocus and output to the outside. The in-focus image I Focus is used, for example, for inspection of work 2. The focused image acquisition unit 16 may be provided outside the image acquisition device 1. Further, when the focused image I Focus is not used for the evaluation of the work 2, the focused image acquisition unit 16 can be omitted.

<Procedure for registering model area>
FIG. 2 is a flowchart showing an example of a procedure for registering a model area (model target area MRoiPat and model reference area MRoiPos). The flowchart of FIG. 2 is started with the work 2 to be model registered set in the image acquisition device 1.

First, a process of acquiring an image stack Stack (hereinafter, also referred to as “model image stack MS Stack”) of the work 2 that is the target of model registration is executed (step S10). For example, the model image stack MS stack is acquired by taking a plurality of images with the image pickup device 11 while moving the image pickup device 11 in the Z-axis direction using the moving device 12.

Next, the process of registering the model target area MRoiPat is executed (step S20). For example, the model target area MRoiPat is registered by the operator using the model target registration unit 141 to perform an operation such as surrounding the area to be registered as the model target area MRoiPat on the model image stack MS stack.

Next, the process of registering the model reference region MRoiPos is executed (step S30). For example, the model reference area MRoiPos is registered by the operator using the model reference registration unit 142 to perform an operation such as surrounding the area to be registered as the model reference area MRoiPos in the model image stack MS stack.

<Calculation procedure of work target area WRoiPat>
FIG. 3 is a flowchart showing an example of a calculation procedure of the work target area WRoiPat (a partial area in which the in-focus image I Focus is desired to be acquired in the entire image of the work 2). The flowchart of FIG. 3 is started with the work 2 to be evaluated set in the image acquisition device 1.

First, a process of acquiring the image stack Stack of the work 2 to be evaluated (hereinafter, also referred to as “work image stack WS Stack”) is executed (step S11). For example, the work image stack WStack is acquired by taking a plurality of images with the image pickup device 11 while moving the image pickup device 11 in the Z-axis direction using the moving device 12.

Next, the in-focus position acquisition unit 131 executes a process of calculating the in-focus position PosZ of the work image stack WStack (step S40). In this process, for example, in the image in the work image stack WStack, three or more calculation areas including three or more model reference areas MRoiPos registered in advance by the model reference registration unit 142 are extracted and extracted. The focal position PosZ is calculated for each of the three or more calculation areas. As a result, the same number (three or more) of focus positions PosZ as the number of model reference region MRoiPos is calculated.

It should be noted that each calculation area extracted in step S40 is based on each model, assuming that, for example, the set position of the work 2 at the time of model registration and the set position of the work 2 to be evaluated are not significantly different. Each model reference region MRoiPos is set to a region expanded by a predetermined amount centering on the region MRoiPos. By doing so, each model reference region MRoiPos can be included in each calculation region.

Next, the alignment unit 132 performs a process of calculating the work reference region WRoiPos (step S50). In this process, the work reference region WRoiPos is calculated for each focal position PosZ calculated in step S40. In each process, the above-mentioned plane alignment is performed with respect to the calculation area of the focus position PosZ to specify the XY coordinate position of the work reference area WRoiPos, and the focus position PosZ is set to the XY coordinate position of the specified work reference area WRoiPos. The signal in which the above is combined is defined as the work reference region WRoiPos (XYZ coordinate position). By sequentially performing the above processing for each focus position PosZ, the same number (three or more) of work reference regions WRoiPos as the number of model reference regions MRoiPos is calculated.

Next, the work posture estimation unit 151 performs a process of estimating the posture of the imaged work 2 (step S60). In this process, for example, as described above, each of the matrix A of the above equation (3) is based on the combination of the positional relationship between the three or more model reference regions MRoiPos and the three or more work reference regions WRoiPos. By obtaining the components, the posture P of the work 2 is estimated three-dimensionally.

Next, the focused area output unit 152 performs a process of calculating the work target area WRoiPat (step S70). In this process, the area corresponding to the model target area MRoiPat in the work image stack WStack is calculated as the work target area WRoiPat in consideration of the posture P of the work 2 estimated in step S60. When the work target area WRoiPat is not included in the work image stack WStack, the worker may be notified by using a buzzer or the like (not shown).

Finally, the focused image acquisition unit 16 performs a process of acquiring the image most focused on the work target area WRoiPat as the focused image I Focus (step S71).

As described above, according to the image acquisition device 1 according to the present embodiment, the work target is estimated by estimating the in-focus position PosZ, the work reference area WRoiPos, and the work posture P only for the model reference area MRoiPos registered in advance. The region WRoiPat and its focused image IFocus can be acquired. That is, the work target area WRoiPat and its in-focus image IFocus can be acquired without generating a full-focus image. Therefore, the work target area WRoiPat and its in-focus image IFocus can be acquired with a small amount of calculation as compared with the case of generating a full-focus image.

If a omnifocal image is to be generated, it is necessary to perform positioning in the plane direction and the height direction for each image acquisition area, and the amount of calculation increases linearly according to the size of the evaluation area. On the other hand, according to the image acquisition device 1 according to the present embodiment, since the omnifocal image is not generated, the amount of calculation is greatly reduced. As a result, the focused image I Focus of the work 2 to be evaluated can be acquired at high speed.

Embodiment 2.
In the above-described first embodiment, it is assumed that the set position of the work 2 at the time of model registration and the set position of the work 2 to be evaluated are not significantly different. However, in reality, it can be assumed that the set position of the work 2 at the time of model registration and the set position of the work 2 to be evaluated are significantly different.

Therefore, in the second embodiment, a region (hereinafter, also referred to as “field of view alignment region”) as a reference for alignment in the imaging field of view is registered in advance, and alignment (coarse) with respect to the field of view alignment region is performed. After performing alignment), the same processing as in the first embodiment described above is performed. This makes it possible to deal with a case where the set position of the work 2 at the time of model registration and the set position of the work 2 to be evaluated are significantly different.

FIG. 4 is a diagram schematically showing the configuration of an image acquisition system including the image acquisition device 1A according to the second embodiment. The image acquisition device 1A adds a model field of view registration unit (third registration unit) 143 inside the model registration unit 14 and also adds a field of view image processing unit 17 to the image acquisition device 1 shown in FIG. It was done. Since other structures, functions, and processes are the same as those of the image acquisition device 1 shown in FIG. 1, the detailed description here will not be repeated.

The model field of view registration unit 143 is provided in, for example, a computer (not shown) and registers the field of view alignment area. The operator can register the field of view alignment area by operating a computer on the image displayed on the display and performing an operation such as surrounding the area to be specified. The field of view alignment area includes the field of view focus position calculation area MRoiV Focus at the time of model registration and the model field of view alignment area MRoiVPat. The model field of view registration unit 143 outputs the registered field of view alignment area to the field of view image processing unit 17.

The visual field image processing unit 17 includes a visual field focusing position acquisition unit (second acquisition unit) 171 and a visual field alignment unit 172. The visual field focusing position acquisition unit 171 calculates the visual field focusing position VFocus and outputs it to the visual field alignment unit 172 by the same method as the visual field focusing position acquisition unit 131. The field of view alignment unit 172 outputs the field of view position VPos (field of view position deviation amount OffsetXYZ, which will be described later) in the same manner as the field of view alignment unit 132.

FIG. 5 is a flowchart showing an example of the procedure for registering the model area according to the second embodiment. The flowchart of FIG. 5 is obtained by adding steps S80 and S81 to the flowchart of FIG. Since the other steps (steps with the same numbers as the steps shown in FIG. 2 above) have already been described, detailed description thereof will not be repeated here.

After the acquisition of the model image stack MS stack (step S10), the registration of the model target area MRoiPat (step S20), and the registration of the model reference area MRoiPos (step S30), the field of view alignment area is set by the model field of view registration unit 143. The process of registering is performed (step S80). In this process, the calculation area MRoiVFocus of the visual field focus position at the time of model registration and the model visual field alignment area MRoiVPat are registered.

Next, a process of registering the model visual field focusing position MVFocus is performed (step S81). In this process, the focusing index is calculated for the calculation area MRoiV Focus of each image in the model image stack MStack using the field focusing position acquisition unit 171, and the Z coordinate position of the image having the maximum focusing index is the model field. It is registered as the in-focus position MVFocus.

FIG. 6 is a flowchart showing an example of the calculation procedure of the work target area WRoiPat according to the second embodiment. In the flowchart of FIG. 6, steps S40 and S50 of FIG. 3 are changed to steps S41 and S51, respectively, and steps S90 and S100 are added. Since the other steps (steps with the same numbers as the steps shown in FIG. 3 above) have already been described, detailed description thereof will not be repeated here.

After the work image stack WStack is acquired (step S11), the visual field focusing position acquisition unit 171 performs a process of calculating the work visual field focusing position WV Focus (step S90). In this process, the focus index is calculated for the calculation area MRoiV Focus of each image in the work image stack WStack, and the Z coordinate position of the image determined to have the highest focus based on the focus index is the work field focus. Calculated as position WVFocus.

Next, the field of view alignment is performed by the field of view alignment unit 172 (step S100). In this process, the work field of view alignment area WRoiVPat corresponding to the model field of view alignment area MRoiVPat is calculated by adding the depth position deviation amount OffsetZ of the work field of view focusing position WVFocus with respect to the model field of view focusing position MVFocus, and the work field of view is calculated. The amount of misalignment between the alignment region WRoiVPat and the model visual field alignment region MRoiVPat is calculated as the visual field misalignment amount OffsetXYZ. This field of view position deviation amount OffsetXYZ corresponds to the three-dimensional position deviation amount between the set position of the work 2 at the time of model registration and the set position of the work 2 to be evaluated. The above-mentioned visual field position VPos is synonymous with the visual field position deviation amount OffsetXYZ calculated in step S100.

Next, the in-focus position acquisition unit 131 executes a process of calculating the in-focus position PosZ of the work image stack WStack (step S41). In this process, the work reference area WRoiPos corresponding to the model reference area MRoiPos is calculated by adding the field position deviation amount OffsetXYZ calculated in step S100, and for each work reference area WRoiPos as in step S40 of FIG. 3 described above. The focal position of PosZ is calculated.

Next, the alignment unit 132 performs a process of calculating the work reference region WRoiPos (step S51). In this process, the work reference region WRoiPos is calculated by adding the visual field position shift amount OffsetXYZ calculated in step S100, and the position shift amount for each work reference region WRoiPos is calculated in the same manner as in step S50 of FIG. 3 described above. ..

In the following, the process of estimating the posture of the work 2 described with reference to FIG. 3 (step S60), the process of calculating the work target area WRoiPat (step S70), and the process of acquiring the focused image IFocus of the work target area WRoiPat will be described later. (Step S71) is performed.

By doing so, in the image acquisition device 1A according to the second embodiment, the work target area WRoiPat of the work 2 to be evaluated and its focused image I Focus can be displayed at high speed, as in the above-described first embodiment. Can be obtained in. Further, the work target area WRoiPat can be calculated by adding the visual field position deviation amount OffsetXYZ between the set position of the work 2 at the time of model registration and the set position of the work 2 to be evaluated. Therefore, even when the set position of the work 2 at the time of model registration and the set position of the work 2 to be evaluated are significantly different, the work target area WRoiPat and its focused image I Focus can be acquired.

Embodiment 3.
In the above-described first and second embodiments, it is assumed that there is a symbol that can be registered as a reference for alignment in the imaging field of view when the imaging device 11 or the pedestal 3 is moved in the Z-axis direction. There is. However, in reality, since the size of the work 2 in the XY-axis direction is large, it can be registered as a reference for alignment in the imaging field of view when the image pickup device 11 or the pedestal 3 is moved in the Z-axis direction. It can be assumed that there is no design.

In view of the above points, in the third embodiment, the work 2 has a plurality of imaging fields of view slid in the XY-axis direction by moving the image pickup device 11 or the pedestal 3 in the Z-axis direction and also in the XY-axis direction. Is imaged, and the reference for alignment is registered in the image of one of the plurality of imaged fields of view.

FIG. 7 is a diagram schematically showing the configuration of an image acquisition system including the image acquisition device 1B according to the third embodiment. The image acquisition device 1B is an image acquisition device 1A shown in FIG. 4 with an image pickup position recording unit 122 added. Since other structures, functions, and processes are the same as those of the above-mentioned image acquisition device 1A, the detailed description here will not be repeated.

The image pickup position recording unit 122 records the moving unit position information Enc when the work 2 is imaged as an image pickup position signal VPos, and outputs the recorded image pickup position signal VPos to the work posture estimation unit 151. In the third embodiment, as described above, by moving the image pickup apparatus 11 or the pedestal 3 in the Z-axis direction and also in the XY-axis direction, a plurality of imaging fields of view slid in the XY-axis direction are used. It is assumed that the work 2 is imaged.

The work posture estimation unit 151 has a function of receiving an image pickup position signal VPos from the image pickup position recording unit 122 in addition to the functions described in the first and second embodiments.

FIG. 8 is a flowchart showing an example of the procedure for registering the model area according to the third embodiment. The flowchart of FIG. 8 is obtained by adding steps S110 and S120 to the flowchart of FIG. Since the other steps (steps with the same numbers as the steps shown in FIG. 5 above) have already been described, detailed description thereof will not be repeated here.

First, it is determined whether or not the imaging of all the visual fields in the XY-axis direction is completed (step S110). When the imaging of all the visual fields in the XY-axis direction is not completed (NO in step S110), the imaging position recording unit 122 performs a process of recording the current imaging position signal VPos (step S120). After that, the processes of steps S10 to S81 shown in FIG. 5 described above are performed. After that, the process is returned to step S110, and the processes of steps S110, S120, S10 to S81 are repeated for each field of view in the XY axis direction until the imaging of all the fields of view in the XY axis direction is completed. Then, when the imaging of all the visual fields in the XY-axis direction is completed (YES in step S110), the model registration process is terminated.

FIG. 9 is a flowchart showing an example of the calculation procedure of the work target area WRoiPat according to the third embodiment. Of the steps shown in FIG. 9, the steps having the same numbers as the steps shown in FIG. 6 described above have already been described, and detailed description thereof will not be repeated here.

First, it is determined whether or not the imaging of all the visual fields in the XY-axis direction is completed (step S111). When the imaging of all the visual fields in the XY-axis direction is not completed (NO in step S111), the imaging position recording unit 122 performs a process of recording the current imaging position signal VPos (step S121). After that, the processes of steps S11 to S51 shown in FIG. 6 described above are performed. After that, the process is returned to step S111, and the processes of steps S111, S121, S11 to S51 are repeated for each field of view until the imaging of all the fields of view in the XY axis direction is completed.

Then, when the imaging of all the visual fields in the XY-axis direction is completed (YES in step S111), the process of estimating the posture of the work 2 is performed (step S61). In this process, it is based on the in-focus position PosZ of each field of view calculated in step S41, the work reference region WRoiPos of each field of view calculated in step S51, and the image pickup position signal VPos of each field of view recorded in step S121. The posture of the work 2 is estimated.

After that, as in the first and second embodiments, the process of calculating the work target area WRoiPat (step S70) and the process of acquiring the focused image IFocus (step S71) are performed.

By doing so, in the image acquisition device 1B according to the third embodiment, the work target area WRoiPat of the work 2 to be evaluated and its focused image I Focus are similarly to the above-described first and second embodiments. Can be acquired at high speed. Further, the image acquisition device 1B according to the third embodiment captures the work 2 in a plurality of imaging fields of view slid in the XY axis direction, and aligns the work 2 with an image of any of the plurality of fields of view. I try to register the standard. Therefore, even when there is no structure that can be registered as a reference for alignment in one imaging field of view, the work target area WRoiPat and its in-focus image IFocus can be acquired.

Embodiment 4.
In the first, second, and third embodiments, it is assumed that there is no positional deviation between the alignment reference of the work 2 and the image pickup region by the image pickup apparatus 11. However, in the case of a module or the like in which the work 2 is composed by assembling a plurality of parts, it is assumed that a slight misalignment may occur between the alignment reference and the imaging region due to the influence of the assembly error of the work 2. obtain.

In view of the above points, in the fourth embodiment, the alignment reference for correcting the misalignment with respect to the alignment reference is registered for each imaging region.

FIG. 10 is a diagram schematically showing the configuration of an image acquisition system including the image acquisition device 1C according to the fourth embodiment. The image acquisition device 1C is an image acquisition device 1 shown in FIG. 1 with a site reference registration unit 144 and a site image processing unit 18 added inside the model registration unit 14. Since other structures, functions, and processes are the same as those of the image acquisition device 1 shown in FIG. 1, the detailed description here will not be repeated.

The site reference registration unit 144 is provided, for example, in a computer (not shown), and registers a "site alignment area" that serves as a reference for alignment of each part of the work 2. It is assumed that the operator registers the site alignment area by operating a computer on the image displayed on the display to surround the area to be specified. The site alignment area includes the calculation area MRoiP Focus of the site focus position at the time of model registration and the model site alignment area MRoiPPat. The site reference registration unit 144 outputs the registered site alignment area to the site image processing unit 18.

The site image processing unit 18 includes a site focusing position acquisition unit 181 and a site alignment unit 182.

The site focusing position acquisition unit 181 calculates the site focusing position PFocus by the same method as the site focusing position acquisition unit 131, and outputs the calculation to the site alignment unit 182. The site alignment unit 182 calculates the site position Ppos (site position shift amount POffsetXYZ described later) by the same method as the alignment unit 132, and outputs the calculation to the in-focus region output unit 152.

The focused area output unit 152 according to the fourth embodiment has a function of receiving the site misalignment amount POffsetXYZ in addition to the functions of the focused area output unit 152 according to the first to third embodiments described above.

FIG. 11 is a flowchart showing an example of the procedure for registering the model area according to the fourth embodiment. The flowchart of FIG. 11 is obtained by adding steps S130 and S131 to the flowchart of FIG. Since the other steps (steps with the same numbers as the steps shown in FIG. 2 above) have already been described, detailed description thereof will not be repeated here.

In the registration of the model area according to the fourth embodiment, after the acquisition of the model image stack MS stack (step S10), the registration of the model target area MRoiPat (step S20), and the registration of the model reference area MRoiPos (step S30) are performed. The process of registering the site alignment area is performed (step S130). In this process, the site reference registration unit 144 registers the calculation area MRoiP Focus of the site focus position at the time of model registration and the model site alignment area MRoiPPat.

Next, a process of registering the model site focused position MPFocus is performed (step S131). In this process, the focus index is calculated using the site focus position acquisition unit 181 for the region focus position calculation area MRoiP Focus of each image in the model image stack MS tack, and the calculation region of each image in the model image stack MS rack. For MRoiV Focus, the focus index is calculated using the visual field focus position acquisition unit 171 and the obtained focus position (Z coordinate position of the image with the maximum focus index) is registered as the model site focus position MPFocus. To.

FIG. 12 is a flowchart showing an example of the calculation procedure of the work target area WRoiPat according to the fourth embodiment. The flowchart of FIG. 12 is obtained by adding steps S140 and S150 to the flowchart of FIG. 3 described above. Since the other steps (steps with the same numbers as the steps shown in FIG. 3 above) have already been described, detailed description thereof will not be repeated here.

After the work image stack WStack is acquired (step S11), a process of calculating the work site in-focus position WP Focus is performed (step S140). In this process, the site focus position acquisition unit 181 calculates the work site focus position WP Focus for the area registered as the site focus position calculation area MRoiP Focus in the site reference registration unit 144.

After performing step 140, the process of step 150 is performed. In step S150, for the region registered as the model site alignment region MRoiPPat by the site reference registration unit 144, the depth position in consideration of the site depth position shift amount POffsetZ of the work site focus position WP Focus with respect to the model site focus position MPFocus. The work site alignment region WRoiPPat is calculated based on the above, and the amount of misalignment between the work site alignment region WRoiPPat and the model site alignment region MRoiPPat is calculated as the site misalignment amount POffsetXYZ.

Next, the process of calculating the in-focus position PosZ (step S40), the process of calculating the work reference region WRoiPos (step S50), and the process of estimating the posture P of the work 2 (step S60) described with reference to FIG. 3 above. Is done.

In the process of calculating the work target area WRoiPat (step S70) in the fourth embodiment, the posture P of the work 2 estimated in step S60, the position of the model target area MRoiPat, and the site position calculated in step 150. The work target area WRoiPat in the work image stack WStack is calculated based on the deviation amount POffsetXYZ.

Finally, a process (step S71) for acquiring the focused image I Focus of the work target area WRoiPat described with reference to FIG. 3 is performed.

By doing so, in the image acquisition device 1C according to the fourth embodiment, the work target area WRoiPat of the work 2 to be evaluated and its in-focus point are similarly to the above-described first, second, and third embodiments. Image I Focus can be acquired at high speed. Further, the image acquisition device 1C according to the fourth embodiment registers the alignment reference for correcting the misalignment for each imaging region. Therefore, when an image is taken of a module or the like composed by assembling a plurality of parts, it is possible to absorb the assembly error and acquire a focused image without being affected by the positional deviation of each part.

Embodiment 5.
In the first, second, third, and fourth embodiments, it is assumed that the operator performs the work of registering the model area for each area for which the in-focus image is desired to be acquired. However, when there are a large number of areas for which a focused image is desired to be acquired for the work 2, there is a concern that the registration work will take an enormous amount of time.

In view of the above points, in the fifth embodiment, the machine learning function for creating a candidate for the imaging region based on the image of the imaging region registered in the past is used, and the model region is based on the candidate for the imaging region. To register. As a result, even when there are a large number of areas for which a focused image is desired to be acquired, the model area can be efficiently registered.

FIG. 13 is a diagram schematically showing the configuration of an image acquisition system including the image acquisition device 1D according to the fifth embodiment. The image acquisition device 1D is an image acquisition device 1 shown in FIG. 1 with a learning unit 19 added. Since other structures, functions, and processes are the same as those of the image acquisition device 1 shown in FIG. 1, the detailed description here will not be repeated.

The learning unit 19 includes a sample image generation unit 191, a machine learning unit 192, a learning data recording unit 193, an image cutting unit 194, and a determination unit 195. The learning unit 19 is not necessarily limited to being built in the image acquisition device 1D, and may exist outside the image acquisition device 1D (for example, a cloud server).

Based on the image stack Stack and the model target area MRoiPat, the sample image generation unit 191 determines whether the sample image MRoiPatImg, which is an image of the model target area MRoiPat in the image stack Stack, and the sample image MRoiPatImg are the model target area MRoiPat. The combination with the truth value isMRoiPat, which is the truth value of, is output. Sample image when the boolean value isMRoiPat is false MRoiPatImg outputs a region having the same size and not including the model target region MRoiPat based on the model target region MRoiPat.

The machine learning unit 192 learns the probability MRoiPatP indicating the probability of being the model target region according to the combination of the sample image MRoiPatImg and its boolean value isMRoiPat, and records the learning result as learning data C in the learning data recording unit 193. Further, the machine learning unit 192 inputs the learning data C recorded in the learning data recording unit 193 and the partial image data RoiImg from the image cutting unit 194, and determines the probability that the partial image data RoiImg is the model target area. It functions as an inference means for outputting the indicated probability MRoiPatP. The machine learning unit 192 is, for example, a device that performs so-called supervised learning according to a neural network model.

Here, supervised learning is a model in which a large amount of data sets of a certain input and output results (labels) are given to a learning device to learn the characteristics of those data sets and estimate the output results from the inputs. To say. A neural network is composed of an input layer composed of a plurality of neurons, an intermediate layer (hidden layer) composed of a plurality of neurons, and an output layer composed of a plurality of neurons. The intermediate layer may be one layer or two or more layers.

FIG. 14 is a diagram showing an example of a model of a neural network. In the case of a three-layer neural network as shown in FIG. 14, when a plurality of inputs are input to the input layers (X1 to X3), the result of multiplying the values by the weights W1 (w11 to w16) is the intermediate layer (the intermediate layer (X1 to X3). It is input to Y1 and Y2), and the result of further multiplying the result by the weight W2 (w21 to w26) is output from the output layer (Z1 to Z3). This output result depends on the values of the weights W1 and W2.

In the present application, the neural network learns the above-mentioned probability MRoiPatP by so-called supervised learning according to a data set created based on a combination of a sample image MRoiPatImg and a boolean value isMRoiPat, and learns the learning result as learning data C. Record in the data recording unit 193.

Returning to FIG. 13, the learning data recording unit 193 has a function of storing the learning data C which is the learning result of the machine learning unit 192.

Further, the learning of the machine learning unit 192 may be performed according to the data set created for the plurality of types of work 2. The machine learning unit 192 may acquire data sets from a plurality of workpieces 2 used at the same site, or may use data sets collected from a plurality of machine tools operating independently at different sites. Then, the probability MRoiPatP may be learned. Further, the work 2 for collecting the data set can be added to the target on the way, or conversely, can be removed from the target.

The image cutting unit 194 cuts out the partial image data RoiImg based on the image stack Stack and outputs it to the machine learning unit 192. The method of cutting out the partial image data RoiImg may be a method of cutting out an image having the same pitch and the same size in the vertical direction and the horizontal direction for each image constituting the image stack Stack.

The determination unit 195 determines whether or not the partial image data RoiImg is a candidate for the model target area based on the probability MRoiPatP from the machine learning unit 192, and the determination unit 195 determines whether or not the partial image data RoiImg is a candidate for the model target area. Is output as the estimation model target area MRoiPatE. For example, the estimated model target region MRoiPatE may be output as a result of comparing the probability MRoiPatP with the preset threshold value Th.

The model target registration unit 141 according to the fifth embodiment receives the image stack Stack and the estimated model target area MRoiPatE as inputs, and outputs the model target area MRoiPat.

The model target registration unit 141 is provided in a computer (not shown). If there is an estimated model target area MRoiPatE, the estimated model target area MRoiPatE is presented to the operator. The operator operates a computer to register the estimated model target area MRoiPatE as the model target area MRoiPat. Alternatively, it is assumed that the estimated model target area MRoiPatE is registered as the model target area MRoiPat by performing an operation such as modifying the estimated model target area MRoiPatE on the screen on which the image is displayed.

FIG. 15 is a flowchart showing an example of the procedure for registering the model area according to the fifth embodiment. The flowchart of FIG. 15 is obtained by adding steps S160, S170, and S180 to the flowchart of FIG. Since the other steps (steps with the same numbers as the steps shown in FIG. 2 above) have already been described, detailed description thereof will not be repeated here.

In the model registration according to the fifth embodiment, step S160 is performed after step S10. In step S160, it is confirmed whether or not the learning data C that can be used by the machine learning unit 192 is in the learning data recording unit 193. If there is no training data C (NO in step S160), the model area is registered in step S20, that is, in the same manner as the procedure for registering the model area in the first embodiment.

If there is learning data C (YES in step S160), step 170 is performed. In step S170, the partial image data RoiImg is cut out by the image cutting unit 194 with respect to the image stack Stack acquired in step S10, and is input to the machine learning unit 192 together with the learning data C. The machine learning unit 192 calculates the probability MRoiPatP in which the partial image data RoiImg is the model target region. Based on this probability MRoiPatP, the determination unit 195 determines whether or not the partial image data RoiImg is a candidate for the model target area, and estimates the partial image data RoiImg that is determined to be a candidate for the model target area MRoiPatE. Is output to the model target registration unit 141.

In step S20 according to the fifth embodiment, the operator registers the region for which the focused image is to be acquired as the model target region MRoiPat while referring to the estimated model target region MRoiPatE.

Further, in the fifth embodiment, after the completion of step S30, step S180 is carried out. In step S180, in order to carry out the learning procedure of the model region shown in FIG. 16 described later, the combination of the model image stack MS stack and the model target region MRoiPat is recorded in the sample image generation unit 191.

FIG. 16 is a flowchart showing an example of the learning procedure of the model area in the fifth embodiment. In the learning of the model area, first, step S200 is carried out. In step S200, the sample image generation unit 191 sets the sample image MRoiPatImg and the boolean value isMRoiPat based on the combination of the model image stack MS stack and the model target area MRoiPat recorded in advance in step S180 of FIG. Output the combination of. Next, the machine learning unit 192 learns a data set consisting of a group of combinations of a sample image MRoiPatImg and a boolean value isMRoiPat, and infers a probability MRoiPatP which is a probability that a given image is a model target region. To learn.

Subsequently, step S201 is carried out. In step S201, the learning data C is recorded in the learning data recording unit 193 so that the learning data C, which is the learning result of the machine learning unit 192, can be used in the model area registration.

By doing so, in the image acquisition device 1D according to the fifth embodiment, the work target area WRoiPat of the work 2 to be evaluated and its focused image as in the above-described first, second, and third embodiments. IFocus can be acquired at high speed. Further, the image acquisition device 1D according to the fifth embodiment uses a machine learning function for creating a candidate for an imaging region based on an image of an imaging region registered in the past, and obtains an region based on the candidate for the imaging region. sign up. As a result, even when there are a large number of areas for which a focused image is desired to be acquired, the registration work can be performed without hassle.

It should be considered that the embodiments disclosed this time are exemplary in all respects and not restrictive. The scope of this disclosure is set forth by the claims rather than the description above and is intended to include all modifications within the meaning and scope of the claims.

1,1A, 1B, 1C, 1D image acquisition device, 2 works, 3 pedestals, 11 image pickup device, 12 mobile device, 13 image processing unit, 14 model registration unit, 15 attitude estimation unit, 16 in-focus image acquisition unit, 17 Field image processing unit, 18 site image processing unit, 19 learning unit, 122 imaging position recording unit, 131 in-focus position acquisition unit, 132 alignment unit, 141 model target registration unit, 142 model reference registration unit, 143 model field image registration unit. 144 part reference registration part, 151 work posture estimation part, 152 focus area calculation part, 171 field focus position acquisition part, 172 field alignment part, 181 part focus position acquisition part, 182 part alignment part, 191 sample Image generation unit, 192 machine learning unit, 193 learning data recording unit, 194 image cutting unit, 195 judgment unit.

Claims (10)

It is an image acquisition device that acquires an image of a work, and is an image acquisition device.
A first registration unit that registers a first area to be acquired as a focused image using a model image that is an image of a work to be model registered, and a first registration unit.
A second registration unit that registers a second region used for alignment using the model image, and a second registration unit.
An image pickup device that captures a work image, which is an image of the work to be evaluated,
A moving device configured to move at least one of the image pickup device and the work in the optical axis direction of the image pickup device, and a moving device.
A first acquisition unit that acquires the depth position of the image having the highest degree of focus among a plurality of work images having different focal points captured by the image pickup apparatus as the in-focus position while operating the moving device.
An alignment unit that specifies a region corresponding to the second region in the work image by performing alignment with the second region on the work image at the in-focus position.
A posture estimation unit configured to be able to acquire information indicating the posture of the work at the time of imaging based on the positional relationship between the second region in the model image and the region corresponding to the second region in the work image.
A region output unit that outputs the position of a region corresponding to the first region in the work image using information indicating the posture of the work, and a region output unit.
An image acquisition device. A third registration unit that registers a third region used for positioning the imaging field of view using the model image, and a third registration unit.
A second acquisition unit that acquires the depth position of the image having the highest degree of focus in the third region among the plurality of work images having different focal points captured by the image pickup apparatus as the field-of-focus focus position.
By aligning the region corresponding to the third region in the work image at the in-focus position with the third region in the model image, the region corresponding to the third region in the work image and the region are described. Further, a visual field alignment unit for acquiring the amount of misalignment with the position of the third region in the model image is provided.
The image acquisition device according to claim 1, wherein the first acquisition unit acquires the in-focus position in consideration of the amount of misalignment. Further, a position recording unit for recording the image pickup position in the optical axis direction of the image pickup device for each field of view is provided.
The image acquisition device according to claim 1 or 2, wherein the posture estimation unit acquires information indicating the posture of the work based on the image pickup position. A site area registration unit for registering a site area used for alignment of each part of the model image, and a site area registration unit.
A site focus position acquisition unit that acquires a site focus position, which is a site focus position for each site in the model image, and a site focus position acquisition unit.
Any one of claims 1 to 3, further comprising a region corresponding to the site region in the work image of the site focus position and a site alignment portion for performing alignment with the site region in the model image. The image acquisition device described in the section. Machine learning is performed on the correspondence between the first region and the image data based on a plurality of combinations of the first region registered by the first registration unit and the image data captured by the image pickup device, and the result of the machine learning is obtained. The image acquisition device according to any one of claims 1 to 4, further comprising a machine learning unit for determining whether or not the given image data is a candidate for the first region. It is an image acquisition method to acquire an image of a work.
The step of registering the first area to be the target for acquiring the in-focus image using the model image which is the image of the work to be the target of model registration, and
A step of registering a second region used for alignment using the model image, and
The step of taking a work image, which is an image of the work to be evaluated, with an image pickup device,
A step of acquiring the depth position of the image having the highest degree of focus among a plurality of work images having different focal points captured by the image pickup apparatus as the in-focus position.
A step of specifying a region corresponding to the second region in the work image by performing alignment with the second region on the work image at the focused position, and a step of specifying the region corresponding to the second region in the work image.
A step of acquiring information indicating the posture of the work at the time of imaging based on the positional relationship between the second region in the model image and the region corresponding to the second region in the work image, and
A step of outputting the position of a region corresponding to the first region in the work image using information indicating the posture of the work, and a step of outputting the position of the region corresponding to the first region.
Image acquisition methods, including. A step of registering a third region used for positioning the imaging field of view using the model image, and
A step of acquiring the depth position of the image having the highest degree of focusing in the third region among the plurality of work images having different focal points captured by the imaging device as the visual field focusing position.
Further including a step of acquiring a position deviation amount between the region corresponding to the third region in the work image at the in-focus position and the third region in the model image.
The image acquisition method according to claim 6, wherein the step of acquiring the in-focus position includes a step of acquiring the in-focus position in consideration of the amount of misalignment. Further including a step of recording the image pickup position in the optical axis direction of the image pickup device for each field of view.
The image acquisition method according to claim 6 or 7, wherein the step of acquiring the information indicating the posture of the work is to acquire the information indicating the posture of the work based on the imaging position. The step of registering the part area used for the alignment of each part of the model image, and
The step of acquiring the site focus position, which is the focus position for each part of the model image, the position of the region corresponding to the site region in the work image of the site focus position, and the position of the site region in the model image. The image acquisition method according to any one of claims 6 to 8, further comprising a step of performing matching. A step of machine learning the correspondence between the first region and the image data based on a plurality of combinations of the registered first region and the image data captured by the image pickup device.
The image acquisition method according to any one of claims 6 to 9, further comprising a step of determining whether or not the given image data is a candidate for the first region using the result of machine learning.

Images