WO2020039922A1

WO2020039922A1 - Image processing system, image processing method and program

Info

Publication number: WO2020039922A1
Application number: PCT/JP2019/031042
Authority: WO
Inventors: 加藤　豊
Original assignee: オムロン株式会社
Priority date: 2018-08-23
Filing date: 2019-08-07
Publication date: 2020-02-27
Also published as: JP7087823B2; JP2020030339A

Abstract

This image processing system is provided with: an imaging device (10) which images an object and generates an image including the object; and an image processing device (20) which acquires the image from the imaging device and executes image pattern matching on the image. The image pattern is a pattern that is a reference for the position orientation of the object in the image. The imaging device (10) can change a focal position. The image processing device (20) searches the focal position and the position orientation and specifies the position orientation and the focal position by executing image pattern matching while narrowing the range of the focal position.

Description

Image processing system, image processing method and program

The present invention relates to an image processing system, an image processing method, and a program.

Generally, in the field of cameras, an autofocus function for determining a focus state and moving a focus lens of a lens unit so as to focus on a subject is known. For example, Japanese Patent Laying-Open No. 2018-84701 (Patent Document 1) discloses a focus adjustment device that determines focus within a limited range using a contrast detection method or a phase difference detection method.

JP 2018-84701 A

When capturing an image of an object placed at an unknown distance from the imaging device, it may be necessary to specify the position and orientation of the object in the image in addition to the focus position (focusing position) of the object. . As a method for this, for example, the following two methods can be considered.

One method is to search for an image pattern from a plurality of images having different focus positions. The search for the image pattern is executed by, for example, an image processing device. In this method, an image pattern must be searched for each of a plurality of images. Therefore, there is a problem that the processing time becomes long. For example, in a visual inspection of a product flowing on a manufacturing line, a process in a short time is required. A long processing time causes a reduction in inspection throughput.

Another method is to search for an image after determining the focus position. For example, the imaging device images the target object while changing the focus position. The image processing device evaluates the degree of focusing by image processing and determines an optimal focus position. The imaging device searches for an image pattern using an image obtained by imaging the target at the optimal focus position. In this method, in order to obtain an optimal focus position, the image processing apparatus needs to know in advance where the image of the target object is located in the image. However, since the position of the target object in the image cannot be determined in advance, it is not easy to find the optimum focus position.

An object of the present invention is to enable a focus position and a position and orientation in an image to be specified in a short time.

According to an example of the present disclosure, an image processing system captures an image of an object, generates an image including the object, acquires an image from the image capturing unit, and performs image pattern matching on the image. And an image processing unit to execute. The image pattern is a pattern serving as a reference for the position and orientation in the image of the target object. The imaging unit can change the focus position. The image processing unit searches for the focus position and the position and orientation by executing image pattern matching while narrowing the range of the focus position, and specifies the position and orientation and the focus position.

According to this disclosure, the focus position and the position and orientation in the image can be specified in a short time. The image processing system searches for a focus position and a position and orientation in an image from a search space having two degrees of freedom, a focus position and a position and orientation in an image. By executing image pattern matching while narrowing the range of the focus position, the degree of matching in the matching process is improved. By obtaining the focus position in a state where the matching degree in the matching is high, the accuracy of the focus position is further improved. Therefore, the focus position and the position and orientation in the image can be specified in a short time. Further, the focus position and the position and orientation in the image can be specified more accurately.

In the above disclosure, the image processing unit shifts the search for the focus position and the position and orientation from the coarse search to the fine search.

According to this disclosure, the focus position and the position and orientation can be specified in a short time. The transition from the coarse (course) search to the fine (fine) search is characterized in that the information amount is gradually increased and the search step width is gradually reduced.

In the above disclosure, the image processing unit sets a search range for matching in the image, and resets the search range based on the matching degree obtained from the matching result.

According to this disclosure, the focus position and the posture position in the image can be specified in a short time by resetting the search range based on the matching degree.

In the above disclosure, the image processing unit obtains at least one candidate point from the degree of coincidence, and resets the search range so that at least one candidate point is included in the search range and the search range is narrowed.

According to this disclosure, the focus position and the posture position in the image can be specified in a short time by gradually narrowing the search range for matching.

In the above disclosure, in correcting the focus position, the image processing unit performs matching on a plurality of images having different focus positions, and based on a result of the matching, an image having the highest focus degree from the plurality of images. Select The image processing unit reduces a change in the focus position between a plurality of images as the number of corrections of the focus position increases.

According to this disclosure, the focus position and the posture position in the image can be specified in a short time by gradually narrowing the search range of the focus position.

In the above disclosure, the image processing unit selects an image used for correcting the focus position by thinning out a plurality of images captured by the imaging unit while changing the focus position.

According to this disclosure, it is possible to execute a coarse search and a fine search based on a plurality of images acquired in advance.

According to an example of the present disclosure, an image processing method is an image processing method by an image processing system including an imaging unit that captures an object to generate an image including the object, and an image processing unit. The imaging unit can change the focus position when capturing the target object. The image processing method performs matching of an image pattern serving as a reference of the position and orientation in the image of the target object with respect to the image including the target object generated by the imaging unit while narrowing the range of the focus position, The method includes a step of searching for the focus position and the position and orientation, and a step of specifying the position and orientation and the focus position based on the search result of the searching step.

According to this disclosure, the focus position and the position and orientation in the image can be specified in a short time.

According to an example of the present disclosure, a program is a program for processing an image obtained by capturing an image of an object by an imaging device. The imaging device can change the focus position when capturing an object. The program causes the computer to execute matching of an image pattern serving as a reference of the position and orientation in the image of the target object with respect to the image including the target object generated by the imaging device while narrowing the range of the focus position. A step of searching for the focus position and the position and orientation, and a step of specifying the position and orientation and the focus position based on the result of the search by the searching step.

According to the present invention, the focus position and the position and orientation in the image can be specified in a short time.

1 is a schematic diagram illustrating an outline of a visual inspection system that is one application example of an image processing system according to the present embodiment. FIG. 3 is a diagram illustrating an example of an internal configuration of the imaging device. FIG. 3 is a schematic diagram for explaining adjustment of a focus position. FIG. 3 is a diagram illustrating an example of a configuration of a lens. It is a figure which shows another example of a lens for focus position adjustment. FIG. 3 is a diagram schematically illustrating an image obtained by imaging by an imaging device. FIG. 4 is a schematic diagram illustrating a first method for specifying a focus position and a position and orientation of a work. FIG. 9 is a schematic diagram illustrating a second method for specifying a focus position and a position and orientation of a work. FIG. 5 is a schematic diagram showing an area in an image designated to execute the search method according to the present embodiment. It is a schematic diagram for explaining the search method according to the present embodiment. FIG. 4 is a diagram showing a search route by a search method according to the present embodiment. 5 is a flowchart showing a flow of a pre-setting for a search method according to the present embodiment. 5 is a flowchart illustrating a search flow. FIG. 10 is a diagram showing a first applied example of search according to the present embodiment. FIG. 3 is a schematic diagram illustrating an inspection area on a work surface. FIG. 5 is a diagram illustrating an example of setting an inspection area and a reference area. FIG. 14 is a diagram illustrating a second applied example of the search according to the present embodiment. FIG. 14 is a diagram illustrating a third applied example of the search according to the present embodiment.

§1 Application Example First, an example of a scene to which the present invention is applied will be described with reference to FIG. FIG. 1 is a schematic diagram showing one application example of the image processing system according to the present embodiment.

As shown in FIG. 1, the image processing system 1 according to the present embodiment is realized, for example, as a visual inspection system. The appearance inspection system images a plurality of inspection target positions on the work W placed on the stage 90 in, for example, a production line of an industrial product, and performs an appearance inspection of the work W using the obtained images. . In the appearance inspection, the work W is inspected for scratches, dirt, presence or absence of foreign matter, dimensions, and the like.

When the appearance inspection of the work W placed on the stage 90 is completed, the next work (not shown) is transported onto the stage 90. At the time of imaging the work W, the work W may be stationary at a predetermined position on the stage 90 in a predetermined posture. Alternatively, the work W may be imaged while the work W moves on the stage 90.

As shown in FIG. 1, the image processing system 1 includes an imaging device 10 and an image processing device 20 as basic components. In this embodiment, the image processing system 1 further includes a PLC (Programmable Logic Controller) 50 and an input / display device 60.

The imaging device 10 is connected to the image processing device 20. The imaging device 10 captures an image of a subject (work W) present in an imaging field of view in accordance with a command from the image processing device 20, and generates image data including an image of the work W. In the present embodiment, the imaging device 10 is an imaging system capable of changing a focus position. The imaging device 10 and the image processing device 20 may be integrated.

The imaging device 10 includes a lens module with a variable focus position. The focus position means a position where an image of the work W is formed. The focus position of the lens module is changed according to the distance between the imaging device 10 and the work W. Accordingly, an image in which the work W is focused can be captured. Note that the imaging device 10 has an autofocus function, and can automatically focus on the work W.

The image processing device 20 acquires an image of the work W from the imaging device 10. The image processing device 20 performs a predetermined process on the image. The image processing device 20 includes a determination unit 21, an output unit 22, a storage unit 23, and a command generation unit 24.

The determination unit 21 performs a predetermined process on the image data generated by the imaging device 10 to determine whether the appearance of the work W is good. The output unit 22 outputs the result of the determination by the determination unit 21. For example, the output unit 22 causes the input / display device 60 to display the determination result.

The storage unit 23 stores various data, programs, and the like. For example, the storage unit 23 stores the image data acquired from the imaging device 10 and the image data subjected to a predetermined process. The storage unit 23 may store the determination result by the determination unit 21. Further, the storage unit 23 stores a program for causing the image processing device 20 to execute various processes.

The command generation unit 24 receives a control command from the PLC 50 and outputs an imaging command (imaging trigger) to the imaging device 10.

The image processing device 20 is connected to the PLC 50. The PLC 50 controls the image processing device 20. For example, the PLC 50 controls the timing at which the image processing device 20 outputs an imaging command (imaging trigger) to the imaging device 10.

The input / display device 60 is connected to the image processing device 20. The input / display device 60 receives a user input regarding various settings of the image processing system 1. Further, the input / display device 60 displays information on the settings of the image processing system 1 and a result of the image processing of the work W by the image processing device 20 (for example, a result of determining whether the appearance of the product is good or bad).

In the present embodiment, the image processing device 20 performs image pattern matching on an image acquired from the imaging device 10. This image pattern is a pattern serving as a reference for the position and orientation of the work W in the image. The image processing device 20 searches the focus position and the position and orientation by executing image pattern matching while narrowing the range of the focus position, and specifies the position and orientation and the focus position. The image processing system searches for a focus position and a position and orientation in an image from a search space having two degrees of freedom, a focus position and a position and orientation in an image. By executing image pattern matching while narrowing the range of the focus position, the degree of matching in the matching process is improved. By obtaining the focus position in a state where the matching degree in the matching is high, the accuracy of the focus position is further improved. Therefore, the focus position and the position and orientation in the image can be specified in a short time. Further, the focus position and the position and orientation in the image can be specified more accurately.

FIG. 2 is a diagram illustrating an example of the internal configuration of the imaging device 10. As shown in FIG. 2, the imaging device 10 includes an illumination unit 11, a lens module 12, an imaging element 13, an imaging element control unit 14, a lens control unit 16, registers 15, 17, a communication I / O F section 18.

The illumination unit 11 irradiates the work W with light. Light emitted from the illumination unit 11 is reflected on the surface of the work W and enters the lens module 12. The lighting unit 11 may be omitted.

The lens module 12 forms the reflected light from the work W on the imaging surface 13 a of the imaging device 13. The lens module 12 has a lens 12a, a lens group 12b, a lens 12c, a movable unit 12d, and a focus adjustment unit 12e. The lens 12a is a lens for mainly changing a focus position. The focus adjustment unit 12e controls the lens 12a to change the focus position.

The lens group 12b is a lens group for changing the focal length. The zoom magnification is controlled by changing the focal length. The lens group 12b is provided on the movable part 12d and is movable along the optical axis direction. The lens 12c is a lens fixed at a predetermined position in the imaging device 10.

The imaging device 13 is a photoelectric conversion device such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor, for example, and converts light from an imaging visual field into an image signal.

Upon receiving an imaging command from the image processing device 20 via the communication I / F unit 18, the imaging device control unit 14 opens a shutter (not shown) to perform exposure, and generates image data based on an image signal from the imaging device 13. I do. At this time, the imaging element control unit 14 opens and closes the shutter so as to have a shutter speed (exposure time) corresponding to the imaging position, and generates image data of a preset resolution. Information indicating the shutter speed and the resolution corresponding to the imaging position is stored in the register 15 in advance. The imaging device control unit 14 outputs the generated image data to the image processing device 20 via the communication I / F unit 18.

The lens controller 16 adjusts the focus of the imaging device 10 according to the command stored in the register 17. Specifically, the lens control unit 16 controls the focus adjustment unit 12e so that the focus position changes according to the region of the work W where the work W is imaged. The focus adjustment unit 12e adjusts the position of the lens 12a under the control of the lens control unit 16. That is, the lens control unit 16 controls the lens 12a so that the focus is on the imaging target area of the work W. “Focused” means that an image of the imaging target area of the work W is formed on the imaging surface 13 a of the imaging element 13. The lens 12a will be described later in detail.

The lens control unit 16 may control the movable unit 12d to adjust the position of the lens group 12b so that the size of the region included in the imaging field of view of the work W is substantially constant. In other words, the lens control unit 16 can control the movable unit 12d such that the size of a region included in the imaging field of view of the workpiece W is within a predetermined range. The lens controller 16 may adjust the position of the lens group 12b according to the distance between the imaging position and the work W. Note that in this embodiment, zoom adjustment is not essential.

§2 Specific example <A. Configuration example for focus adjustment>
FIG. 3 is a schematic diagram for explaining the adjustment of the focus position. For simplicity, FIG. 3 shows only one lens (lens 12a).

As shown in FIG. 3, the distance from the principal point O of the lens 12a to the target surface (the surface of the workpiece W) is a, the distance from the principal point O of the lens 12a to the imaging surface 13a is b, and the distance from the principal point O of the lens 12a is The distance (focal length) from the point O to the focal point F of the lens 12a is defined as f. When the position of the image of the work W is equal to the position of the imaging surface 13a, the following expression (1) is established.

1 / a + 1 / b = 1 / f (1)
The working distance (WD) can change according to the height of the surface of the work W. The working distance is defined as a distance from the surface of the lens 12a on the work W side to the work W. When Expression (1) holds, it is possible to capture an image in a state where the surface of the work W is in focus. For example, the focus can be adjusted by changing the distance b.

変化 The change amount of the distance b can be represented as an offset from a reference distance. In the present embodiment, this offset is referred to as “lens extension amount”. By relating the distance from the surface of the lens 12a on the work W side to the surface of the work W to the amount of lens movement, the amount of lens movement for obtaining an image focused on the surface of the work W can be determined. . Note that the reference value of the distance b can be arbitrarily determined. For example, the reference value of the distance b may be set as the value of the focal length f.

撮像 As described above, the imaging device 10 has an autofocus function. Therefore, the imaging device 10 determines the degree of focusing from the image of the work W and adjusts the focus position. The configuration of the lens 12a for adjusting the focus position is not particularly limited. Hereinafter, an example of the configuration of the lens 12a will be described.

FIG. 4 is a diagram showing an example of the configuration of the lens 12a. In the configuration shown in FIG. 4, the focus adjustment unit 12e moves the lens 12a along the optical axis direction. By changing the position of the lens 12a, the extension amount of the lens 12a changes. Therefore, the lens 12a moves so that an image of the work W is formed on the imaging surface 13a according to a change in the working distance WD.

FIGS. 3 and 4 show an example of one lens. Usually, an FA lens is often composed of a plurality of grouped lenses. However, also in a group lens, the combined focal length f and the position of the lens principal point can be obtained using the focal length f of each lens and the positional relationship between the lenses. WD can be calculated by using the characteristic values.

FIG. 4 shows an example in which the focus position is adjusted by a mechanical method. However, the method of adjusting the focus position is not limited to a mechanical method. FIG. 5 is a diagram illustrating another example of the focus position adjusting lens.

例示 A liquid lens is illustrated in FIG. The lens 12a includes a translucent container 70,

electrodes

73a, 73b, 74a, 74b,

insulators

75a, 75b, and insulating

layers

76a, 76b.

密閉 The sealed space in the translucent container 70 is filled with a conductive liquid 71 such as water and an insulating liquid 72 such as oil. The conductive liquid 71 and the insulating liquid 72 are not mixed and have different refractive indexes.

The electrodes 73a and 73b are fixed between the

insulators

75a and 75b and the translucent container 70, respectively, and are located in the conductive liquid 71.

The

electrodes

74a and 74b are arranged near the end of the interface between the conductive liquid 71 and the insulating liquid 72. An insulating layer 76a is interposed between the electrode 74a and the conductive liquid 71 and the insulating liquid 72. An insulating layer 76b is interposed between the electrode 74b and the conductive liquid 71 and the insulating liquid 72. The electrode 74a and the electrode 74b are arranged at positions symmetric with respect to the optical axis of the lens 12a.

において In the configuration shown in FIG. 5, the focus adjustment unit 12e includes a voltage source 12e1 and a voltage source 12e2. The voltage source 12e1 applies a voltage Va between the electrode 74a and the electrode 73a. Voltage source 12e2 applies voltage Vb between electrode 74b and electrode 73b.

When a voltage Va is applied between the electrode 74a and the electrode 73a, the conductive liquid 71 is pulled by the electrode 74a. Similarly, when a voltage Vb is applied between the electrode 74b and the electrode 73b, the conductive liquid 71 is pulled by the electrode 74b. Thereby, the curvature of the interface between the conductive liquid 71 and the insulating liquid 72 changes. Since the refractive index of the conductive liquid 71 and the refractive index of the insulating liquid 72 are different, the curvature of the interface between the conductive liquid 71 and the insulating liquid 72 changes, thereby changing the focal length of the lens 12a (the focal length f shown in FIG. 3). Changes). The curvature of the interface depends on the magnitude of the voltages Va and Vb. Therefore, by changing the magnitudes of the voltages Va and Vb, the focus position of the lens 12a can be adjusted so that an image is formed on the imaging surface 13a even when the working distance WD changes.

Normally, voltage Va and voltage Vb are controlled to the same value. Thereby, the interface between the conductive liquid 71 and the insulating liquid 72 changes symmetrically with respect to the optical axis. However, the voltage Va and the voltage Vb may be controlled to different values. Thereby, the interface between the conductive liquid 71 and the insulating liquid 72 becomes asymmetric with respect to the optical axis, and the direction of the imaging visual field of the imaging device 10 can be changed.

Furthermore, a liquid lens and a solid lens may be combined. In this case, the position of the principal point of the lens changes in addition to the focal length f. Accordingly, since the distance b changes, the focus adjustment may be performed in consideration of the change.

<B. Image processing device>
The image processing device 20 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), an auxiliary storage device, a communication I / F, and performs information processing. The auxiliary storage device includes, for example, a hard disk drive, a solid state drive, and the like, and stores a program executed by the CPU.

The determination unit 21 of the image processing device 20 binarizes the difference image between the image of the non-defective work and the image of the non-defective work stored in the storage unit 23 in advance, and determines the number of pixels exceeding the threshold value and the reference value. The quality of the inspection target position may be determined by collation. The output unit 22 of the image processing device 20 may display the determination result on the input / display device 60.

<C. Challenges of Searching Work Focus Position and Position>
FIG. 6 is a diagram schematically illustrating an image obtained by imaging by the imaging device. As shown in FIG. 6, the image 100 includes an image of the work W. Since the distance (work distance) from the lens of the imaging device 10 to the work may differ for each work, in order to obtain an image of the work W in focus, it is necessary to specify an optimum focus position for each work W. is necessary.

Furthermore, in order to image the work W, the work W is originally placed at a predetermined position on the stage 90 in a predetermined posture. However, the work W moves below the imaging device 10 by moving the stage 90. Due to the movement of the work W, there is a possibility that the position of the work W is deviated from the original shooting position, or there is a possibility that the work W is rotating on the stage 90. The degree of displacement of the position and orientation may also differ for each work.

Therefore, when imaging the work W, it is required to specify the position and orientation of the work W in the image 100 in addition to the focus position of the work W. As a method for this, for example, the following two methods can be considered.

FIG. 7 is a schematic diagram illustrating a first method for specifying the focus position and the position and orientation of the work W. As illustrated in FIG. 7, the imaging device 10 captures an image of the work W while changing the focus position, and acquires a plurality (M) of images. The image processing apparatus 20 searches for an image pattern of the work W on each of the M sheets. Then, the image processing device 20 specifies one image that is most focused on the workpiece W from the M images.

When the above method is executed, there is a problem that the processing time becomes long. For example, in a visual inspection of a product flowing on a manufacturing line, a process in a short time is required. A long processing time causes a reduction in inspection throughput.

FIG. 8 is a schematic diagram illustrating a second method for specifying the focus position and the position and orientation of the work W. In the second method, an image is searched after a focus position is determined. As in the first method, the imaging device 10 images the work W while changing the focus position, and acquires M images. The image processing device 20 evaluates the degree of focusing of the M images by image processing, and determines an optimal focus position. The imaging device 10 selects an image obtained by imaging the work W at the optimum focus position from among the M images, and searches for an image pattern in the image.

According to this method, in order to obtain the optimum focus position, the image processing device 20 needs to know the position of the image of the work W in the image and the posture of the work W in advance. However, as described above, since the position and orientation of the work W can be different for each work W, the position and orientation of the work W cannot be characterized in advance. Therefore, the search route becomes longer. In other words, it takes time to specify the optimum focus position and the position and orientation of the work. Therefore, in the present embodiment, a search method for shortening the search route is used.

<D. Search method according to present embodiment>
FIG. 9 is a schematic diagram showing an area in an image designated to execute the search method according to the present embodiment. As shown in FIG. 9, a predetermined area in the image 100 is set as a reference area B1. The image pattern of the reference area B1 is registered in the image processing device 20 as a model. Next, a certain range in the image 100 including the reference area B1 is set as a search range A1. Further, the search range A1 and the search range of the focus position are registered in the image processing device 20.

FIG. 10 is a schematic diagram for explaining the search method according to the present embodiment. As shown in FIG. 10, in the present embodiment, first, a group of images having different focus positions is collectively photographed within the focus position search range. As a result, N images are obtained. The search range of the focus position is, for example, a range in which the focus of the imaging device 10 can be controlled. However, the focus position search range may be narrower than the focus controllable range.

Next, the image processing device 20 extracts M images by thinning out the N image groups. The M images are a so-called subset of a group of N images. Pattern matching is performed on the extracted M images using the registered image model. As a result, one image having the highest focusing degree and the highest degree of pattern matching is selected from the M images.

Subsequently, the search area is limited based on the result of the pattern matching, and a plurality of images having different focus positions are extracted. The plurality of images selected at this time include one image selected last time. Further, the degree of change of the focus position among the plurality of images is smaller than the degree of change of the focus position between the plurality of images selected in the previous search. That is, the focus position changes finely among a plurality of images. The image processing device 20 determines the degree of focus of a plurality of images and selects the image with the highest degree of focus. In the image, the image processing device 20 executes pattern matching by narrowing the search range. As described above, the pattern matching is repeatedly performed while the search range is narrowed while the degree of change in the focus position is reduced.

FIG. 11 is a diagram showing a search route by the search method according to the present embodiment. As shown in FIG. 11, the pattern matching is repeatedly performed while limiting the search range while reducing the degree of change in the focus position. The search for the focus position and the position and orientation is shifted from the coarse search to the fine search. The search route can be shortened for each search. Therefore, according to the present embodiment, it is possible to specify the focus position and the position and orientation within the image in a shorter time.

<E. Flow of search method according to present embodiment>
The processing described below is executed by the determination unit 21 of the image processing device 20 reading out the program stored in the storage unit 23.

FIG. 12 is a flowchart showing a flow of the pre-setting for the search method according to the present embodiment. Referring to FIG. 12, first, in step S11, a sample of the inspection object (work) is prepared. In step S12, the sample is placed at a specified position (in front of the lens of the imaging device 10). In step S13, the lens of the imaging device 10 is controlled to focus on an object (sample).

In step S14, the user specifies a reference area in the image. For example, a sample image 100 is displayed on the display of the input / display device 60. When the user operates an input device such as a pointing device, the designation of a reference region in the image is accepted, and the region is set as the reference region B1. In step S15, the image processing device 20 stores the image pattern of the reference area B1 in the storage unit 23. As a result, the image pattern is registered in the image processing device 20 as a model.

In step S16, the user specifies a search range in the image. As in the process of step S14, the sample image 100 is displayed on the display of the input / display device 60, for example. When the user operates an input device such as a pointing device, the designation of a search range in the image is accepted, and the area is set as the search range A1. The image processing device 20 stores the information of the search range A1 in the storage unit 23. As a result, the specified search range is registered in the image processing apparatus 20.

In step S17, the user specifies a search range of the focus position. The input / display device 60 receives designation of a search range of a focus position. The image processing device 20 stores information on the search range of the focus position in the storage unit 23. Thereby, the search range of the designated focus position is registered in the image processing device 20.

FIG. 13 is a flowchart showing a search flow. Referring to FIG. 13, in step S21, the imaging device 10 collectively captures a group of images having different focus positions within the focus position search range. As a result, N (N is an integer of 2 or more) images are obtained. It is desirable that the focus position of each image is arranged so that the depth of field does not leak.

In step S22, the image processing apparatus 20 extracts M images (subsets) by thinning out the N image groups. Here, M is an integer satisfying 1 <M <N. The extraction method is not particularly limited. For example, images may be extracted at equal intervals (for each predetermined number of images).

In step S23, the image processing device 20 performs pattern matching on the M images using the model registered in step S15 (see FIG. 12). Here, the degree of freedom of pattern matching is the degree of freedom regarding X, Y, and θ. X and Y respectively represent two orthogonal axes in the image. θ represents the rotation angle in the image.

In step S24, the image processing device 20 obtains a plurality of candidate points that give a high correlation value (degree of coincidence). The attributes of the candidate points are correlation values, X, Y, and θ.

As a calculation method of the correlation value, various known methods can be applied. As an example, a normalized cross-correlation method can be used. The normalized cross-correlation method is widely used as a method of searching for an image pattern, and is a method of statistically processing the grayscale information of an image showing an object (for example, the “normalized cross-correlation method” (image Analysis Handbook, edited by Mikio Takagi and Hirohisa Shimoda, University of Tokyo Press). Alternatively, a correlation method as disclosed in JP-A-7-78257 may be used. In this method, a grayscale image is captured by a camera, and an evaluation value at each pixel position of the grayscale image is calculated. Then, the maximum evaluation value in the point sequence in which the evaluation value continuously exceeds the threshold value in the scanning line direction is detected.

エッジ As a correlation method, an edge-based correlation method may be used. The edge-based correlation method is a method of calculating a degree of coincidence using characteristics of an edge (boundary) portion of an image showing an object. The degree of coincidence reflects edge strength or edge gradient. For example, a method as disclosed in JP-A-10-171989 may be used. In this method, a density gradient direction at each pixel of the input image is calculated, and a value for evaluating a difference in the density gradient direction between the density gradient direction of the input image and the density gradient direction of each pixel of a predetermined pattern is calculated. Desired.

In step S25, the image processing device 20 resets the search range. As one embodiment, the image processing apparatus 20 determines the degree of focus of a plurality of images having different focus positions by limiting the search range to an area near the candidate point position. That is, the search range is narrowed to a range including the candidate points. The image processing device 20 selects an image with the highest focus degree from among the plurality of images. For example, an edge is extracted from a region near a candidate point position of each image. The degree of focus is determined based on the edge.

In step S26, the image processing device 20 performs pattern matching on the selected image. At this time, the search range may be limited by resetting the search range. Representative parameters of pattern matching include “search increments” and “model thinning”. “Search step” means a search step (resolution), and “model thinning” does not express an image pattern to be a model with all pixels, but approximately expresses it with thinned pixels. Things. In the coarse search, which is an initial stage of the search, the values of these parameters are increased, and the values of these parameters are gradually reduced as the search proceeds to the detailed search (dense search). Such a method is a conventional method in pattern matching. However, conventionally, it has been used closed for searching for one image. In the present embodiment, the above method is used without closing within an image. This can be expected to improve the search efficiency (speed up).

In step S27, the image processing device 20 obtains a candidate point that gives a high correlation value. As described above, the attributes of the candidate point are the correlation value, X, Y, and θ.

In step S28, the image processing device 20 determines whether the termination condition is satisfied. Note that the termination condition is not particularly limited. For example, the resolution of the final search may be defined in advance, and the entire processing may be terminated when the resolution is reached. For example, for the position in the xy direction, the final resolution may be one pixel. As for the angle, the final resolution may be 1 deg. Regarding the focus position, the final resolution of the lens extension amount may be 1 mm.

(5) After the search is completed, estimation with a precision smaller than the step size (sub-pixel estimation) may be performed by various methods (for example, parabola fitting or the like) using correlation values.

If the end condition is not satisfied, the process returns to step S25, and the processes of steps S25 to S28 are repeated. Thereby, the accuracy of the focus position and the position and orientation in the image can be improved. When the termination condition is satisfied (YES in S28), the search processing ends. Therefore, the focus position and the position and orientation of the work in the image are optimized.

In the present embodiment, a coarse search is performed at the beginning of the iteration, and a finer search is performed as the number of iterations increases. This makes it possible to speed up the search process without lowering the accuracy. When there are a plurality of candidate points, only one point may be finally left using the correlation value, or a plurality of points may be output.

<F. Effect of the present embodiment>
According to the present embodiment, a short-time search can be performed. Such an effect will be described with a specific example. The preconditions are as follows.

・ Image size: 1600 × 1200 pixels ・ Reference pattern size: 300 × 300 pixels ・ Search range: 1000 × 1000 pixels ・ WD search range: 100 mm to 500 mm
Time required to search for a pattern from one image: 20 ms (1) (generally, the sum of the times (2) and (3) below)
Time required to roughly search for a pattern from one image: 10 ms (2) (candidate points are obtained)
Time required to search for a pattern in detail around a candidate point of an image: 10 ms (3)
-Time required for one image shooting by changing the focus position from the current position: 10 ms
-Number of variations of the focus position in the WD search range: 20-Time required to calculate the evaluation value of the focal point: 5 ms
(In case of full search)
Time required for image capturing with a changed focus position: 10 ms × 20 times = 200 ms
-Image pattern search time: 20 ms x 20 sheets = 400 ms
・ Total time: 600 ms (in the case of search according to the present embodiment)
Time required for image shooting with the focus position changed (decimated): 10 ms x 5 times = 50 ms
Time to roughly search for an image pattern from a thinned image set: 10 ms x 5 images = 50 ms
・ Time required for image capturing around the found candidate: 10 ms × 4 times = 40 ms
Time required for searching for a focus position: 5 ms × 4 times = 20 ms
・ Time to search for an image pattern finely around one candidate point: 10 ms × 1 time = 10 ms
・ Total time: 170ms
As described above, the search according to the present embodiment is about three to four times faster than the conventional full search. According to the above example, the number of captured images is nine (= 5 + 4). However, as in the case of full search, 20 images were captured by changing the focus position. Also in this case, the search according to the present embodiment is faster than the full search. The time required for image capturing is 10 ms × 20 times = 200 ms, but the time required for other processing is the same, so the total time is 320 ms. Therefore, the search according to the present embodiment is about twice as fast as the full search.

<G. Application example of search according to present embodiment>
FIG. 14 is a diagram showing a first applied example of the search according to the present embodiment. As shown in FIG. 14, the search according to the present embodiment can be used for the appearance inspection of the work W. In the example shown in FIG. 14, the work W has an area W1 and an area W2. The region W1 is, for example, the surface of a transparent body (such as glass). The region W2 is a region surrounding the region W1, for example, a surface of a housing of the electronic device. Examples of such a work W include an electronic device having a display (for example, a smartphone or a tablet). That is, the area W1 can be a display screen.

FIG. 15 is a schematic diagram showing an inspection area on the work surface. Referring to FIG. 15, inspection area A2 is set in area W1 of work W. In the appearance inspection, an area including the inspection area A2 is imaged by the imaging device 10, and the inspection area is inspected for the presence or absence of a flaw, dirt, or foreign matter.

In this example, an area (reference area B1) serving as a focus reference may be set separately from the inspection area. FIG. 16 is a diagram showing a setting example of the inspection area A2 and the reference area B1. Referring to FIG. 16, reference region B1 is a region including a clear pattern. According to the example shown in FIG. 16, the reference area B1 includes a part of the area W2. For example, the reference area is an area including an end of the area W2 and includes an outline of the work W. Therefore, the reference area B1 includes a boundary between the work W and the periphery of the work W. Thereby, the reference area B1 can include a clear pattern. Thus, the image of the reference area B1 has such a contrast that the degree of focus can be determined. Therefore, the focus position and the position and orientation of the work W in the image can be obtained from the reference area B1.

フォーカス Set the focus position before the actual inspection. The sample work W is imaged by the imaging device 10 to obtain an image of the work W. The reference region B1 and the inspection region A2 are respectively specified from the image of the work W, and the images of the reference region B1 and the inspection region A2 are registered in the image processing device 20. Further, a focus position when the reference area B1 is in focus and a focus position when the inspection area A2 is in focus are registered in the image processing apparatus 20, respectively. The search according to the present embodiment can be used for searching for a focus position. Further, the search according to the present embodiment may be used to correct the deviation of the XY position of the workpiece W in the image.

FIG. 17 is a diagram showing a second applied example of the search according to the present embodiment. A second application is picking of a workpiece. As shown in FIG. 17, a work W (for example, a part) carried on the belt conveyor 91 is gripped by the robot 30 and transferred. The robot is, for example, a parallel link robot, but is not limited to this.

Conventionally, parts were recognized using a two-dimensional camera. However, when the parts lie down or overlap each other, the gripping may fail. In such a case, it is originally desirable to securely hold only the grippable parts while avoiding the non-grabable parts. If a three-dimensional sensor is used, it is possible to recognize the height and posture of the component, so that it is possible to cope with such a situation. However, three-dimensional sensors have a problem in that they are more expensive than two-dimensional cameras. According to the present embodiment, the position and height of the component can be recognized without using a special sensor. Therefore, it is possible to reliably grip only the grippable components with a low-cost system.

FIG. 18 is a diagram showing a third applied example of the search according to the present embodiment. A third application is reading of bar codes or characters on the surface of an article (not limited to a work). Referring to FIG. 18, imaging apparatus 10 captures an image of barcode 5 printed on work W (for example, a component or a packing box) carried on belt conveyor 91. The image processing device 20 acquires an image from the imaging device 10 and reads the barcode 5. The present embodiment can be used not only for barcodes but also for recognition of characters or two-dimensional codes.

場合 When products of different varieties flow on the belt conveyor 91, it is necessary to read the bar code 5 and the like for works of various heights. In a conventional imaging method, in order to obtain an in-focus image, a method is used in which the lens aperture is reduced (that is, the F-number is increased) and the depth of field is increased to perform reading. . However, reducing the lens aperture reduces the amount of light that enters the imaging device. Therefore, it was necessary to prepare a large-sized illumination in order to supplement the light quantity. As another method, there is a method of combining a height sensor and a variable focus lens. In that case, however, it is necessary to separately prepare a height sensor.

According to the present embodiment, the focus position and the image pattern can be specified in a short time and accurately without the need for an external sensor or large illumination. Bar codes or characters can be read accurately in a short time.

<H. Appendix>
As described above, the present embodiment includes the following disclosure.

(Configuration 1)
An imaging unit (10) configured to capture an image of the object (W) and generate an image including the object (W);
An image processing unit (20) that acquires the image from the imaging unit (10) and executes image pattern matching on the image.
The image pattern is a pattern serving as a reference for a position and orientation of the object (W) in the image,
The imaging unit (10) is capable of changing a focus position,
The image processing unit (20) searches for the focus position and the position / posture by executing the matching of the image pattern while narrowing the range of the focus position, and searches the position / posture and the focus position. An image processing system (1) for specifying

(Configuration 2)
The image processing system (1) according to Configuration 1, wherein the image processing unit (20) shifts the search for the focus position and the position and orientation from a coarse search to a fine search.

(Configuration 3)
The image processing unit (20), wherein a search range for the matching is set in the image, and the search range is reset based on the matching degree obtained from the matching result. 3. The image processing system (1) according to item 2.

(Configuration 4)
The image processing unit (20) obtains at least one candidate point from the matching degree, and re-determines the search range so that the at least one candidate point is included in the search range and the search range is narrowed. The image processing system (1) according to Configuration 3, which is set.

(Configuration 5)
In the correction of the focus position, the image processing unit (20) performs the matching on a plurality of images having different focus positions, and determines a degree of focus from the plurality of images based on a result of the matching. Select the image with the highest
The image processing according to any one of Configurations 1 to 4, wherein the image processing unit (20) reduces a change in the focus position between the plurality of images as the number of times of correction of the focus position increases. System (1).

(Configuration 6)
The image processing unit (20) according to configuration 5, wherein the image capturing unit (10) selects an image used for correcting the focus position by thinning out a plurality of images captured while changing the focus position. Image processing system (1).

(Configuration 7)
An image processing method by an image processing system (1) including an imaging unit (10) for imaging an object (W) to generate an image including the object (W), and an image processing unit (20). hand,
The imaging unit (10) can change a focus position when imaging the object (W),
The image processing method includes:
An image serving as a reference for the position and orientation of the object (W) in the image with respect to the image including the object (W) generated by the imaging unit (10) while narrowing the range of the focus position. Searching for the focus position and the position and orientation by executing pattern matching (S25-S28);
Specifying the position and orientation and the focus position from a result of the search performed by the searching step.

(Configuration 8)
A program for processing an image obtained by imaging an object (W) by an imaging device (10),
The imaging device (10) can change a focus position when imaging the object (W),
The program is stored in a computer,
While narrowing the range of the focus position, matching of an image including the object (W) generated by the imaging device with an image pattern serving as a reference of the position and orientation of the object (W) in the image (S25-S28) of searching for the focus position and the position and orientation by executing
A program for executing the step of specifying the position and orientation and the focus position from a result of the search in the searching step.

Although the embodiments of the present invention have been described, the embodiments disclosed this time are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the terms of the claims, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

1 image processing system, 5 barcode, 10 image pickup device, 11 illumination unit, 12 lens module, 12a, 12c lens, 12b lens group, 12d movable unit, 12e focus adjustment unit, 12e1, 12e2 voltage source, 13 image sensor, 13a Imaging surface, 14 image sensor control unit, 15, 17 register, 16 lens control unit, 18 communication I / F unit, 20 image processing device, 21 judgment unit, 22 output unit, 23 storage unit, 24 command generation unit, 30 robot , 60 display device, 70 translucent container, 71 conductive liquid, 72 insulating liquid, 73a, 73b, 74a, 74b electrode, 75a, 75b insulator, 76a, 76b insulating layer, 90 stage, 91 belt conveyor, 100 Image, A1 search range, A2 inspection area, B1 group Regions, F focus, O principal point, S11 ~ S17, S21 ~ S28 step, W workpiece, W1, W2 region.

Claims

An imaging unit configured to capture an image of an object and generate an image including the object;
An image processing unit that acquires the image from the imaging unit and performs image pattern matching on the image,
The image pattern is a pattern serving as a reference for the position and orientation of the object in the image,
The imaging unit is capable of changing a focus position,
The image processing unit searches the focus position and the position and orientation by executing the matching of the image pattern while narrowing the range of the focus position, and specifies the position and orientation and the focus position. , Image processing system.
The image processing system according to claim 1, wherein the image processing unit shifts the search for the focus position and the position and orientation from a coarse search to a fine search.
The said image processing part sets the search range for the said matching in the said image, and resets the said search range based on the matching degree calculated | required from the said matching result. An image processing system according to claim 1.
The image processing unit obtains at least one candidate point from the matching degree, and resets the search range so that the at least one candidate point is included in the search range and the search range is narrowed; The image processing system according to claim 3.
In the correction of the focus position, the image processing unit performs the matching on a plurality of images having different focus positions, and, based on a result of the matching, a highest degree of focus from the plurality of images. Select an image,
The image according to any one of claims 1 to 4, wherein the image processing unit reduces a change in the focus position between the plurality of images as the number of times of correction of the focus position increases. Processing system.
6. The image processing system according to claim 5, wherein the image processing unit selects an image used for correcting the focus position by thinning out a plurality of images captured by the imaging unit while changing the focus position. 7.
An imaging unit that captures an object to generate an image including the object, and an image processing method by an image processing system including an image processing unit,
The imaging unit is capable of changing a focus position when imaging the object,
The image processing method includes:
While narrowing the range of the focus position, by performing matching of an image pattern serving as a reference of the position and orientation of the target object in the image, for an image including the target object generated by the imaging unit, Searching for the focus position and the position and orientation;
Specifying the position and orientation and the focus position from a result of the search performed by the searching step.
A program for processing an image obtained by imaging an object by an imaging device,
The imaging device is capable of changing a focus position when imaging the object,
The program is stored in a computer,
While narrowing the range of the focus position, for an image including the target generated by the imaging device, by performing matching of an image pattern serving as a reference of the position and orientation of the target in the image, Searching for the focus position and the position and orientation;
A program for executing the step of specifying the position and orientation and the focus position from a result of the search in the searching step.