JP6852525B2 - Calibration equipment, calibration method, calibration program - Google Patents

Description

The present disclosure relates to calibration devices, calibration methods, and calibration programs.

A technique for calibrating an optical system is known based on an image obtained by imaging a calibration pattern such as a checker pattern with an optical system.

Japanese Unexamined Patent Publication No. 2012-7980 Japanese Unexamined Patent Publication No. 2012-2612 Japanese Unexamined Patent Publication No. 9-329418

However, in the conventional technique as described above, when proofreading is performed using a plurality of images having different positional relationships between the proofreading pattern at the time of imaging and the optical system, a common reference point for proofreading is used for each image. For example, it is difficult to accurately identify the intersection of checker patterns). For example, when the intersection of checker patterns is a reference point for calibration, there are many intersections of checker patterns. Therefore, it may be difficult to identify an intersection related to a common reference point from among a large number of intersections between images having different positional relationships between the calibration pattern at the time of imaging and the optical system. If the intersection of the common reference points is erroneously specified, the position of the reference point for calibration cannot be calculated accurately, and highly accurate calibration cannot be realized.

Therefore, on one aspect, it is an object of the present invention to realize highly accurate calibration by using a plurality of images in which the positional relationship between the calibration pattern and the optical system at the time of imaging is different from each other.

On one side, it is an image obtained by imaging a calibration pattern extending in a plane with two or more marks added to a regular pattern with an optical system, and the image obtained by imaging the calibration pattern when the image is taken. An image acquisition unit that acquires a plurality of images having different positional relationships with the optical system, and an image acquisition unit.
For each image, a mark recognition unit that recognizes the two or more marks in the image, and a mark recognition unit.
Based on the recognition result by the mark recognition unit, the reference approximate position calculation unit that calculates the approximate position of the reference point of the regular pattern in the image, and the reference approximate position calculation unit.
For each image, the position of the reference point of the regular pattern in the image is calculated based on the approximate position calculated by the reference approximate position calculation unit and the feature amount in the predetermined region of the image. Reference point calculation unit and
Provided is a calibration apparatus including a calibration parameter value deriving unit that derives a calibration parameter value of the optical system based on the position of the reference point for each image calculated by the reference point calculation unit.

On one aspect, according to the present invention, it is possible to realize highly accurate calibration by using a plurality of images in which the positional relationship between the calibration pattern and the optical system at the time of imaging is different from each other.

It is a figure which shows roughly the structure of the system including the control device. It is explanatory drawing of the calibration pattern. It is explanatory drawing of the principle of the calibration method used in a control device. It is a figure which shows an example of the hardware composition of a control device. It is a figure which shows an example of the functional structure of a control device. It is explanatory drawing of the calculation method of the inclination angle. It is explanatory drawing of the search range. It is explanatory drawing of the concrete example of the derivation method of the feature amount in a search range. It is explanatory drawing of the concrete example of the derivation method of the feature amount in a search range. It is explanatory drawing of the image of the calibration pattern which does not give a mark. It is explanatory drawing of the image of the calibration pattern which does not give a mark. It is a figure which shows the image of the calibration pattern which gave the mark. It is a figure which shows the image of the calibration pattern which gave the mark. It is a figure which shows the modification of the positional relationship between a mark and a reference point. It is a figure which shows the image in the state which the mark is in the search range. It is a figure which shows the image in the state which the mark is in the search range. It is a figure which shows the image which the mark was removed. It is a figure which shows the image which the mark was removed. It is a schematic flowchart which shows the operation example of the control device in a calibration phase. It is a schematic flowchart which shows the operation example of the control device in the production phase.

Hereinafter, each embodiment will be described in detail with reference to the accompanying drawings.

FIG. 1 is a diagram schematically showing a configuration of a system 1 including a control device 10. In FIG. 1, the X, Y, and Z axes are shown as three orthogonal axes.

The system 1 includes a sample stage device 2, an image pickup device 3 (an example of an optical system), a lighting device 4, an output device 5, a stage controller 6A, a lighting controller 6B, an imaging controller 6C, and a control device 10 (an example of an optical system). An example of a calibrator) and. The image pickup device 3 and the control device 10 form an example of a calibration system.

The system 1 may be, for example, a system for mounting electronic components by a robot. In this case, an object on which electronic components are mounted, such as a substrate, is placed as a sample on the stage 23 of the sample stage device 2. The system 1 may also be a system for component inspection, and its use is arbitrary.

The sample stage device 2 includes a drive mechanism 20, a fixing portion 22, and a stage 23.

The drive mechanism 20 changes the positions of the image pickup device 3 and the illumination device 4 with respect to the stage 23 in the X, Y, and Z axis directions. The drive mechanism 20 includes a mechanism 20X that changes the positions of the image pickup device 3 and the illumination device 4 with respect to the stage 23 in the X-axis direction (an example of the second direction), a mechanism 20Y in the Y-axis direction, and a Z-axis direction (first). Includes a mechanism 20Z (an example of direction). The fixed portion 22 is provided with a stage 23 via a mechanism 20Y. Therefore, the stage 23 is movably supported in the Y-axis direction with respect to the fixed portion 22 via the mechanism 20Y. In the example shown in FIG. 1, the image pickup device 3 and the lighting device 4 are movably supported in the Y-axis direction with respect to the fixed portion 22 via the mechanisms 20Y and 20Z. If the positional relationship between the stage 23 and the image pickup device 3 is variable in the three-axis directions, the arrangement of each mechanism of the drive mechanism 20 is arbitrary. For example, the mechanism 20Z is provided on the fixed portion 22 side. The mechanism 20Y may be provided on the image pickup apparatus 3 side. As described above, the stage 23 is a table on which the sample is placed. At the time of calibration, a calibration board, which will be described later, is placed on the stage 23 instead of the sample.

The image pickup device 3 is a monocular camera, and includes a lens (not shown) having perspective characteristics and an image pickup device. A lens having perspective characteristics means that it is a general lens and not a special lens such as a so-called telecentric lens. The image pickup device is arbitrary, and is a CCD (Charge Coupled Device), a CMOS (Complementary MOS: metal-oxide-semiconductor), or the like.

The lighting device 4 illuminates the stage 23. The lighting device 4 may include, for example, an LED (light emitting diode).

The output device 5 is, for example, a liquid crystal display, an organic EL (ElectroLuminescence) display, or the like. The stage controller 6A is a controller that controls the drive mechanism 20. The lighting controller 6B is a controller that controls the lighting device 4. The image pickup controller 6C is a controller that controls the image pickup device 3. The output device 5, the stage controller 6A, the lighting controller 6B, and the image pickup controller 6C are connected to the control device 10.

The control device 10 is a device that controls the system 1. The control device 10 also functions as an example of the calibration device. However, in the modified example, the control device 10 functions only as a calibration device, and a control device that performs other control may be separately provided. Hereinafter, the functions related to the calibration device will be mainly described with respect to the functions of the control device 10.

The control device 10 calibrates the image pickup device 3 by using parallax. Specifically, the control device 10 performs calibration using the parallax generated with respect to the same reference point on the calibration pattern by changing the position of the image pickup device 3 with respect to the calibration pattern (see FIG. 2). In this embodiment, as an example, the calibration parameter of the image pickup apparatus 3 is an f value.

FIG. 2 is an explanatory diagram of a calibration pattern (calibration pattern). The calibration pattern is an image-recognizable pattern applied to a calibration board (a plate material forming a flat surface) and is used for calibration. The calibration board is formed of, for example, a glass substrate. When the calibration board is placed on the stage 23, the calibration pattern extends in the plane (XY plane) of the stage 23.

In FIG. 2, the X1 and Y1 axes are shown as two orthogonal axes. The X1 and Y1 axes are axes defined by the calibration board, for example, the X1 axis is parallel to the longitudinal direction of the calibration board, and the Y1 axis is parallel to the lateral direction of the calibration board. As for the calibration pattern, the X1Y1 plane forms the same plane as the XY plane, but the X1 axis may be inclined with respect to the X axis depending on the mounting mode of the calibration board on the stage 23.

In the example shown in FIG. 2, the calibration pattern includes a checker pattern (an example of a regular pattern). The checker pattern is a regular pattern in which two types of rectangular portions having different colors (shades) are arranged alternately and continuously in two orthogonal directions (X1 direction and Y1 direction). The two types of rectangular portions having different colors are typically a white square rectangular portion and a black square rectangular portion. Therefore, in the checker pattern, the four rectangular portions forming the square include two white square rectangular portions and two black square rectangular portions, and the center of the four rectangular portions forming the square is. Form the intersection of checkered patterns. In other words, the intersections of two diagonally adjacent rectangular portions of the same color form the intersection of the checkered patterns.

The size of the calibration pattern corresponds to the size of the checker pattern, but the size of the checker pattern is arbitrary. For example, the checker pattern shown in FIG. 2 may be a part of the calibration pattern on the calibration board.

In this embodiment, the checker pattern is given three marks M1, M2, and M3 as the calibration pattern. The score of the mark may be 2 points or 4 points or more.

The marks M1, M2, and M3 have the same shape, the same color, and the same size, but may be different. However, the marks M1, M2, and M3 have a size that fits within one rectangular portion of the checker pattern, and are preferably arranged at the center of the rectangular portion. The marks M1, M2, and M3 have shapes that do not change their appearance even when the calibration board is rotated, and are circular in the example shown in FIG.

In the example shown in FIG. 2, the marks M1, M2, and M3 are arranged in the white rectangular portion and are black, but may be reversed. That is, the marks M1, M2, and M3 are arranged in the black rectangular portion, respectively, and may be white.

The positions of the marks M1, M2, and M3 are arbitrary, but they are arranged at a relatively long distance so that the inclination angle θ described later can be calculated accurately. For example, in the example shown in FIG. 2, the marks M1 and M2 are offset in the X1 direction by two pitches of the grid pitch. The grid pitch is the pitch of the checker pattern, which is twice the length of one side of the rectangular portion.

Further, when three or more marks are given as marks M1, M2, and M3 (an example of the first mark, the second mark, and the third mark), the marks of three points or more circumscribe each mark. The outer shapes formed by connecting them in this manner are arranged so as not to be rotationally symmetric. This is because, as will be described later, the inclination angle θ can be calculated based on the recognition results of the marks M1, M2, and M3. In the example shown in FIG. 2, the marks M1 and M3 are offset in the Y1 direction by one pitch of the grid pitch. Therefore, the length of the line segment connecting the marks M1 and M2 (an example of the first distance) is larger than the length of the line segment connecting the marks M1 and M3 (an example of the second distance). The angle between the line segment connecting the marks M1 and M2 and the line segment connecting the marks M1 and M3 is 90 degrees.

A reference point used for calibration is set in the checker pattern. The reference point may be set arbitrarily, but is typically set at the intersection of checker patterns. In this case, the reference point is formed by the four rectangular portions (that is, the centers of the four rectangular portions). The number of reference points may be one, but two or more may be set. The reference point is preferably set near the end of the calibration board in the X1 direction. This is, for example, the parallax related to the reference point generated when the image pickup apparatus 3 is moved in the X-axis direction when the calibration board is placed on the stage 23 with the center of the calibration board aligned with the center of the stage 23. This is because it can be made larger. In the example shown in FIG. 2, the reference point Ref is set at the intersection of the checker patterns near the end in the X1 direction of the calibration board.

Further, in this embodiment, as an example, the reference point Ref is formed by the rectangular portions to which the marks M1, M2, and M3 are not assigned, as shown in FIG. That is, in this embodiment, as an example, the marks M1, M2, and M3 are not given in the rectangular portion forming the reference point Ref. The technical significance of this point will be described later.

In the following, as an example, as shown in FIG. 2, it is assumed that the length of one side of each rectangular portion is “1”, and therefore the lattice pitch is “2”. Further, it is assumed that the origins related to the coordinate systems of the X1 axis and the Y1 axis are offset in the Y1 direction by one pitch of the lattice pitch with respect to the reference point Ref. In this case, the coordinates of the center positions of the marks M1, M2, and M3 are as follows.
Center position of mark M1 = (1.5,3.5)
Center position of mark M2 = (5.5,3.5)
Center position of mark M3 = (1.5,1.5)

FIG. 3 is an explanatory diagram of the principle of the calibration method used in the control device 10. FIG. 3 shows a state when the image pickup apparatus 3 is positioned at different positions P1 and P2 in the X-axis direction. Further, in FIG. 3, a part of the expansion of the field of view of the image pickup apparatus 3 at each position P1 and P2 is shown by the hatching ranges SL and SR. That is, at the position P1 and the position P2, the position of the calibration board with respect to the image pickup apparatus 3 in the X-axis direction changes by "b". The distance b is predetermined according to the field of view, and is, for example, about 5 mm. In FIG. 3, the points Ref1 and Ref2 represent the positions of the reference points Ref when the positions of the calibration boards with respect to the image pickup apparatus 3 in the Z-axis direction are different. That is, at the points Ref1 and Ref2, the position of the calibration board with respect to the image pickup apparatus 3 in the Z-axis direction changes by ΔZ. ΔZ corresponds to the amount of movement by the mechanism 20Z and becomes known. ΔZ is arbitrary, but in this embodiment, it is 20 mm as an example. Further, in FIG. 3, the f value, which is a calibration parameter of the image pickup apparatus 3, is indicated by “f”.

FIG. 3 shows a calibration method by the parallel stereo method. In the parallel stereo method, the four parameters xL ₁ , xL ₂ , xR ₁ and xR ₂ shown in FIG. 3 are calculated. The parameter xL ₁ is the position of the reference point Ref in the image obtained by the image pickup device 3 in a state where the image pickup device 3 is at the position P2 and the reference point Ref is located at the point Ref 1 as schematically shown in FIG. Corresponds to (distance from the left edge of the image). The parameter xL ₂ is the position of the reference point Ref in the image obtained by the image pickup device 3 in a state where the image pickup device 3 is at the position P2 and the reference point Ref is located at the point Ref2, as schematically shown in FIG. Corresponds to (distance from the left edge of the image). Similarly, the parameter xR ₁ is a reference point in the image obtained by the image pickup device 3 in a state where the image pickup device 3 is at the position P1 and the reference point Ref is located at the point Ref 1 as schematically shown in FIG. Corresponds to the position of Ref (distance from the left edge of the image). The parameter xR ₂ sets the position of the reference point Ref in the image obtained by the image pickup device 3 in a state where the image pickup device 3 is at the position P1 and the reference point Ref is located at the point Ref 2 as schematically shown in FIG. Corresponds to (distance from the left edge of the image).

At this time, each parameter satisfies the following relational expression.
Z _n = bf / | xL _n -xR _n | Equation (1)
Here, Z _n, xL _n, and the subscript in xR _n "n" is, in FIG. 3, a "1" or "2", specifically, as follows.
Z ₁ = bf / | xL ₁ -xR ₁ | Equation (2)
Z ₂ = bf / | xL ₂ -xR ₂ | Equation (3)
As _{shown in FIG. 3, Z 1} is the distance in the Z-axis direction between the point Ref 1 and the image pickup device 3 (an example of the third distance), and Z ₂ is the distance between the point Ref 2 and the image pickup device 3. It is a distance in the Z-axis direction (an example of a fourth distance) and has the following relationship.
Z ₁ = Z ₂ + ΔZ equation (4)
Here, since b and ΔZ are known, there are three _{unknowns, f, Z 1} , and Z _2. Therefore, if the values of the parameters xL ₁ , xL ₂ , xR ₁ , and xR ₂ are calculated, the f-number can be obtained using the equations (2) to (4). If each value of the parameters xL ₁ , xL ₂ , xR ₁ , and xR ₂ can be calculated with high accuracy, highly accurate calibration can be realized (that is, a highly accurate f value can be obtained).

FIG. 4 is a diagram showing an example of the hardware configuration of the control device 10. The peripheral device 8 is schematically shown in association with the hardware configuration of the control device 10. The peripheral device 8 includes an output device 5, a stage controller 6A, a lighting controller 6B, and an imaging controller 6C.

The control device 10 includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, an auxiliary storage device 14, a drive device 15, and a communication interface 17 connected by a bus 19. , A wired transmission / reception unit 25 and a wireless transmission / reception unit 26 connected to the communication interface 17 are included.

The auxiliary storage device 14 is, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like, and is a storage device that stores data related to application software or the like.

The wired transmission / reception unit 25 includes a transmission / reception unit capable of communicating using a wired network. A peripheral device 8 is connected to the wired transmission / reception unit 25. However, a part or all of the peripheral device 8 may be connected to the bus 19.

The wireless transmission / reception unit 26 is a transmission / reception unit capable of communicating using a wireless network. The wireless network may include a wireless communication network of a mobile phone, the Internet, the World Wide Web, a VPN (virtual private network), a WAN (Wide Area Network), and the like. Further, the wireless transmission / reception unit 26 may include a short-range wireless communication (NFC: Near Field Communication) unit, a Bluetooth (Bluetooth, registered trademark) communication unit, a Wi-Fi (Wireless-Fidelity) transmission / reception unit, an infrared transmission / reception unit, and the like. .. In the modified example, the wireless transmission / reception unit 26 may be omitted.

The control device 10 may be connectable to the recording medium 16. The recording medium 16 stores a predetermined program. The program stored in the recording medium 16 is installed in the auxiliary storage device 14 or the like of the control device 10 via the drive device 15. The installed predetermined program can be executed by the CPU 11 of the control device 10. For example, the recording medium 16 is a recording medium such as a CD (Compact Disc) -ROM, a flexible disk, a magneto-optical disk, or the like that optically, electrically, or magnetically records information, a ROM, a flash memory, or the like. It may be a semiconductor memory or the like that electrically records. The recording medium 16 does not include a carrier wave.

FIG. 5 is a diagram showing an example of the functional configuration of the control device 10.

The control device 10 includes an image acquisition unit 100, a mark recognition unit 101, an inclination calculation unit 102, a ratio calculation unit 103, a reference approximate position calculation unit 104, a reference point calculation unit 105, and a calibration parameter value derivation unit 106. And include. Further, the control device 10 includes a point of interest position derivation unit 107, a distance calculation unit 108, and a calibration parameter value storage unit 110. In the image acquisition unit 100, the mark recognition unit 101, the inclination calculation unit 102, the ratio calculation unit 103, the reference approximate position calculation unit 104, the reference point calculation unit 105, and the calibration parameter value derivation unit 106, the CPU 11 is stored in the ROM 13. This can be achieved by executing one or more programs. Further, the point of interest position derivation unit 107 and the distance calculation unit 108 can be realized by the CPU 11 executing one or more programs stored in the ROM 13. The one or more programs may be stored in a storage device other than the ROM 13 (for example, the auxiliary storage device 14). The calibration parameter value storage unit 110 can be realized by, for example, the auxiliary storage device 14.

The control device 10 performs different operations in the calibration phase and the actual operation (operation) phase. The calibration phase is a phase before the production phase, and shifts to the production phase through the calibration phase. The image acquisition unit 100, the mark recognition unit 101, the inclination calculation unit 102, the ratio calculation unit 103, the reference approximate position calculation unit 104, the reference point calculation unit 105, and the calibration parameter value derivation unit 106 function in the calibration phase. The image acquisition unit 100, the point of interest position derivation unit 107, and the distance calculation unit 108 function in the actual operation phase. In the modified example, the control device 10 may be provided with another control device that functions only in the calibration phase and functions in the actual operation phase.

In the calibration phase, the image acquisition unit 100 acquires an image obtained by imaging the calibration board mounted on the stage 23 with the image pickup device 3. In the calibration phase, the calibration board is placed on the stage 23 so that the surface on the side on which the calibration pattern is drawn faces the image pickup apparatus 3. In addition, the image acquisition unit 100 acquires an image obtained by imaging the sample placed on the stage 23 with the image pickup device 3 in the actual operation phase.

In the calibration phase, the mark recognition unit 101 recognizes the marks M1, M2, and M3 in the image based on the image acquired by the image acquisition unit 100. For the recognition of the marks M1, M2, and M3, a normal recognition method such as pattern matching may be used. At this time, the mark recognition unit 101 may detect the positions of the marks M1, M2, and M3 after identifying the marks M1, M2, and M3. The marks M1, M2, and M3 can be identified based on the geometrical relationship of the positions of the three recognizable marks. In this embodiment, as an example, the mark recognition unit 101 identifies the marks M1, M2, and M3, and then calculates the positions A to C of the marks M1, M2, and M3 in the image. In the following, it is assumed that the positions A to C of the marks M1, M2, and M3 in the image are calculated as follows in the xy coordinate system of the image. In the xy coordinate system of the image, the x-axis direction is the horizontal direction of the image. It is assumed that the horizontal direction of the image coincides with the X-axis direction (see FIG. 1). That is, it is assumed that the image pickup apparatus 3 is arranged so that the horizontal direction of the image coincides with the X-axis direction. Therefore, the vertical direction of the image coincides with the Y-axis direction (see FIG. 1).
Mark M1 position A = (xa, ya)
Mark M2 position B = (xb, yb)
Mark M3 position C = (xc, yc)

The inclination calculation unit 102 calculates the inclination angle θ, which is the inclination of the calibration pattern with respect to the reference line, in the calibration phase. The reference line may be in any direction in the plane on which the calibration board is placed (that is, the XY plane), but as an example, it is assumed to be the horizontal direction of the image. FIG. 6 is an explanatory diagram of a method of calculating the inclination angle θ. FIG. 6 shows an image obtained in a state where the calibration pattern with respect to the reference line is inclined. In FIG. 6, the vector x is a vector along the horizontal direction of the image. As shown in FIG. 6, the inclination angle θ is an angle formed by a vector connecting the position A of the mark M1 and the position B of the mark M2 (hereinafter referred to as “vector AB”) and the vector x. Therefore, the inclination angle θ is as follows.

例えば、図６に示す例では、ベクトルＡＢ=(10,1)であったとすると、傾斜角θ≒５．７度となる。

For example, in the example shown in FIG. 6, if the vector AB = (10,1), the inclination angle θ≈5.7 degrees.

The inclination calculation unit 102 calculates a similar inclination angle θ by using a vector (hereinafter, referred to as “vector AC”) connecting the position A of the mark M1 and the position C of the mark M3 instead of the vector AB. You may. However, since the vector AB is larger than the vector AC, it can be expected that the calculation accuracy will be higher when the vector AB is used.

In the calibration phase, the ratio calculation unit 103 calculates the scale s, which is the ratio of _{the actual distance D0 AB between the two marks M1 and M2 to the distance D1 AB} between the two marks M1 and M2 in the image. .. In this embodiment, as an example, the scale s is the distance _D0 is expressed as the ratio of the distance _{D1 AB (= D1 AB / D0} AB) against _AB, be reversed ratio ₍₌ D0 _AB / _D1 AB) Good. The distance D0 _AB is a known value determined at the stage when the marks M1 and M2 are added to the calibration board. For example, in the example shown in FIG. 2, the distance D0 _AB = 4. The distance D1 _AB is obtained by deriving the length of the vector AB. Therefore, the scale s is as follows.

In the calibration phase, the reference approximate position calculation unit 104 calculates the approximate position q of the reference point Ref based on the inclination angle θ and the scale s. The approximate position q (x1, y1) is geometrically calculated as follows using the position C (= (xc, yc)) of the mark M3. The "rough" position is set because the accuracy of the calculated value of the position C of the mark M3 is relatively inferior to the position calculated by the reference point calculation unit 105 described later. is there.

ここで、（L_0x,L_0y）は、実際のマークＭ３の位置から実際の基準点Ｒｅｆの位置までのベクトルであり、既知である。例えば、図２に示す例では、（L_0x,L_0y）=(-1.5,0.5)である。

Here, (L _0x , L _0y ) is a vector from the position of the actual mark M3 to the position of the actual reference point Ref, and is known. For example, in the example shown in FIG. _{2, (L 0x, L 0y} ) = - a (1.5, 0.5).

The reference point calculation unit 105 accurately calculates the position of the reference point Ref in the image in the calibration phase. Specifically, the reference point calculation unit 105 accurately calculates the position of the reference point Ref in the image based on the approximate position q and the feature amount in the predetermined region including the approximate position q. In this embodiment, as an example, a search range that defines a predetermined area is set. FIG. 7 is an explanatory diagram of the search range. FIG. 7 shows an image obtained in a state where the calibration pattern with respect to the reference line is inclined, as in FIG. 6 above. In FIG. 7, the outer frame of the search range 70 is shown by a dotted line in order to show the positional relationship with the image.

The search range 70 is preferably smaller than the four rectangular portions of the checker pattern. That is, when the outer frame of the search range 70 is square, the length of one side of the outer frame of the search range 70 is smaller than the lattice pitch, for example, in the range of 0.5 times or more and 0.7 times or less. In this embodiment, as an example, the length of one side of the outer frame of the search range 70 is 0.6 times the lattice pitch.

The reference point calculation unit 105 sets the search range in such a manner that the center of the search range is located at the approximate position q. The reference point calculation unit 105 positions the search range 70 with respect to the image so that the center of the outer frame of the search range 70 coincides with the approximate position q. At this time, the search range 70 may be set with respect to the image in a direction in which the horizontal direction and the vertical direction match the horizontal direction and the vertical direction of the image, respectively. Then, the reference point calculation unit 105 accurately calculates the position of the reference point Ref in the image by deriving the feature amount within the search range 70. For the precise calculation of the position of the reference point Ref in the image, for example, the methods shown in FIGS. 8A and 8B are used.

8A and 8B are explanatory views of a specific example of a method for deriving a feature amount within the search range 70. 8A and 8B show four rectangular portions (four rectangular portions including the search range 70) of the checker pattern.

In the examples shown in FIGS. 8A and 8B, as feature quantities, a vector connecting the first position r1 and the second position r2 in the search range (hereinafter referred to as “vector r1r2”) and a gradient at the second position r2. An inner product with the vector ∇I (r2) (hereinafter, referred to as “inner product <∇I (r2), r1-r2>”) is used. The initial value of the first position r1 is set to the approximate position q. The second position r2 is an arbitrary position within the search range, and is moved to take a plurality of positions. The second position r2 is, for example, each pixel position within the search range. The direction of the gradient vector at the second position r2 corresponds to the direction in which the change gradient of the shade at the second position r2 is maximized. Here, as shown in FIG. 8A, when the second position r2 is located in the rectangular portion, the gradient vector ∇I (r2) is 0. This is because the shading change is uniform in the rectangular portion. Therefore, in this case, since ∇I (r2) = 0, the inner product <∇I (r2), r1-r2> = 0. Further, when the vector r1r2 is along the edge of the rectangular portion as shown in FIG. 8B, the gradient vector ∇I (r2) is perpendicular to the vector r1r2. Therefore, also in this case, the inner product <∇I (r2), r1-r2> = 0. Using this point, the reference point calculation unit 105 obtains the first position r1 in which the inner product <∇I (r2), r1-r2> = 0 for each second position r2, and determines this value as the first position. The above calculation is performed by setting it as a new initial value of the position r1. This operation is repeated repeatedly until a predetermined accuracy reference is reached, and the first position r1 is searched. Then, the reference point calculation unit 105 sets the first position r1 that has reached a predetermined accuracy reference as the position (precise solution) of the reference point Ref. For the precise solution search method described in FIGS. 8A and 8B, "Detailed OpenCV by Gary Bradski and Adrian Kaebler, O'Reilly, pp.325-326, 2011" is a reference. The precision of the position of the reference point calculated by the reference point calculation unit 105 may be, for example, on the order of 1/10 to 1/30 for one pixel.

In the calibration phase, the calibration parameter value derivation unit 106 derives the calibration parameter value (that is, the f value) of the image pickup apparatus 3 based on the position of the reference point calculated by the reference point calculation unit 105. The method for deriving the f value is as described above with reference to FIG. The values of the parameters xL ₁ , xL ₂ , xR ₁ , and xR ₂ are set at each position of the reference point calculated by the reference point calculation unit 105 (for example, the value of the x coordinate in the xy coordinate system of the image). Correspond. When the calibration parameter value derivation unit 106 derives the f value, the derived f value is stored in the calibration parameter value storage unit 110.

In the actual operation phase, the attention point position deriving unit 107 derives the position of the attention point (position in the image) of the sample placed on the stage 23 based on the image from the image acquisition unit 100. The points of interest of the sample are predetermined in an image-recognizable manner. The point of interest of the sample may be a characteristic portion of the sample itself, a marker given to the sample, or the like. The attention point position derivation unit 107 refers to the image obtained when the image pickup device 3 is at the position P2 and the image obtained when the image pickup device 3 is at the position P1, respectively. Derive the position.

In the actual working phase, the distance calculation unit 108 samples from the image pickup apparatus 3 based on the position of the point of interest of the sample calculated by the point of interest position derivation unit 107 and the f value stored in the calibration parameter value storage unit 110. The distance to the point of interest (distance in the Z-axis direction) is calculated. The distance from the image pickup apparatus 3 to the point of interest of the sample can be calculated using the equation (1).

By the way, in the calibration phase, the exact optical magnification of the image pickup apparatus 3 is unknown. Since the calibration pattern has a checker pattern (regular pattern), it is difficult to calculate even the approximate position of the reference point Ref for calibration in a situation where the exact optical magnification of the image pickup apparatus 3 is unknown. In some cases.

Further, in the calibration phase, the calibration pattern may be inclined with respect to the reference line (horizontal direction of the image). That is, the direction of the X1 axis (see FIG. 2) of the calibration pattern in the image may be inclined with respect to the horizontal direction of the image. When such an inclination occurs, it may be difficult to calculate even the approximate position of the reference point Ref for calibration. It can be difficult to arrange the calibration board so that the calibration pattern does not tilt with respect to the reference line.

9A and 9B are diagrams showing images of a calibration pattern in which the marks M1, M2, and M3 are not added to the checker pattern. FIG. 9A shows an image when ΔZ = 0, that is, when the reference point Ref is located at the point Ref1 (see FIG. 3), and FIG. 9B shows an image when ΔZ = 20, that is, the reference point Ref is the point Ref2 (FIG. 3). 3) shows the image when it is located.

In FIGS. 9A and 9B, the approximate positions of the reference points Ref are indicated by arrows R1 and R2, respectively. From FIGS. 9A and 9B, when the reference point Ref for calibration is set at the intersection of the checker patterns, it is difficult to specify the intersection related to the reference point from among the many intersections of the checker patterns in the image. You can see that.

In this regard, according to the present embodiment, as described above, the mark recognition unit 101, the inclination calculation unit 102, the ratio calculation unit 103, and the reference approximate position calculation unit 104 are provided. As a result, when the reference point Ref for calibration is set at the intersection of the checker patterns, the intersection related to the reference point Ref can be accurately specified from among the many intersections of the checker patterns in the image.

10A and 10B are diagrams showing images of calibration patterns in which marks M1, M2, and M3 are added to checker patterns. FIG. 10A shows an image when ΔZ = 0, that is, when the reference point Ref is located at the point Ref1 (see FIG. 3), and FIG. 10B shows an image when ΔZ = 20, that is, the reference point Ref is the point Ref2 (FIG. 3). 3) shows the image when it is located.

In FIGS. 10A and 10B, the approximate positions of the reference points Ref are indicated by arrows R3 and R4, respectively. As can be seen from FIGS. 10A and 10B, by adding marks M1, M2, and M3 to the checker pattern, it is possible to accurately identify the intersection related to the reference point Ref from among the many intersections of the checker pattern in the image. .. That is, the inconvenience that the approximate position q of the reference point Ref calculated by the reference approximate position calculation unit 104 is closer to another intersection than the intersection related to the reference point Ref (that is, the outline relating to an erroneous intersection different from the reference point Ref). The inconvenience of calculating the position q) can be reduced. As a result, according to this embodiment, highly accurate calibration can be realized.

Further, according to the present embodiment, as described above, the reference point Ref is formed by the rectangular portions to which the marks M1, M2, and M3 are not assigned, as shown in FIG. That is, the marks M1, M2, and M3 are not given in the rectangular portion forming the reference point Ref. As a result, it is possible to prevent the marks M1, M2, and M3 from affecting the precise calculation of the position of the reference point Ref in the image by the reference point calculation unit 105. Specifically, when a mark such as the mark M1 exists in the search range 70, the pixel value related to the mark may affect the feature amount calculated by the reference point calculation unit 105. In such a case, the precise calculation of the position of the reference point Ref in the image by the reference point calculation unit 105 is hindered due to the pixel value related to the mark. In this regard, according to the present embodiment, since the pixel related to the mark M1 or the like is not included in the search range 70, the reference point Ref in the image by the reference point calculation unit 105 due to the pixel value related to the mark M1 or the like. It is possible to eliminate the inconvenience that the precise calculation of the position of is hindered. As a result, according to this embodiment, highly accurate calibration can be realized.

However, as in the modified example shown in FIG. 11, the reference point Ref may be formed by the rectangular portions to which the marks M1, M2, and M3 are given. In the example shown in FIG. 11, the mark M3 is attached to the rectangular portion forming the reference point Ref (the rectangular portion of the white square at the lower right with respect to the reference point Ref). In this case, after the approximate position q of the reference point Ref is calculated by the reference approximate position calculation unit 104, and prior to the precise calculation of the position of the reference point Ref in the image by the reference point calculation unit 105, the image is processed by image processing. The processing for removing the marks M1, M2, and M3 inside may be performed.

Specifically, FIGS. 12A, 12B, 13A, and 13B are diagrams showing images of a calibration pattern when the mark M3 is applied to the rectangular portion forming the reference point Ref. 12A and 12B show the states before the processing for removing the marks M1, M2 and M3 in the image by the image processing, and FIGS. 13A and 13B show the marks M1, M2 in the image by the image processing. And the state after performing the process of removing M3 are shown. 12A and 13A show images when ΔZ = 0, that is, when the reference point Ref is located at the point Ref1 (see FIG. 3), and FIGS. 12B and 13B show images when ΔZ = 20, that is, The image when the reference point Ref is located at the point Ref2 (see FIG. 3) is shown. The search range 70 is schematically shown in FIGS. 12A, 12B, 13A, and 13B.

As shown in FIGS. 12A and 12B, when the mark M3 is attached to the rectangular portion forming the reference point Ref, the pixel related to the mark M3 is included in the search range 70. As a result, as described above, due to the pixel value related to the mark, the precise calculation of the position of the reference point Ref in the image by the reference point calculation unit 105 is hindered. On the other hand, when the processing of removing the marks M1, M2 and M3 in the image by the image processing is performed, the marks M1, M2 and M3 are displayed as shown in the ranges 1301 to 1303 in FIGS. 13A and 13B. The pixel value is replaced with a peripheral pixel value. That is, the pixel values related to the marks M1, M2, and M3 are replaced with the pixel values related to "white" in the white rectangular portion. This eliminates the inconvenience that the reference point calculation unit 105 hinders the precise calculation of the position of the reference point Ref in the image due to the pixel value related to the mark M1 and the like, and can realize highly accurate calibration.

In the examples shown in FIGS. 12A and 12B, the marks M1, M2, and M3 in the image are removed by the image processing, but only the marks M3 that can be within the search range 70 may be removed.

Next, with reference to FIGS. 14 and 15, operation examples of the control device 10 will be described in the calibration phase and the actual operation phase, respectively.

FIG. 14 is a schematic flowchart showing an operation example of the control device 10 in the calibration phase. The process shown in FIG. 14 is activated, for example, when the user places the calibration board on the stage 23 and inputs a calibration start command to the control device 10. In FIG. 14, it is assumed that a calibration board having the calibration pattern shown in FIG. 2 is used. The calibration board is placed on the stage 23 so that the marks M1, M2, and M3 are in the field of view of the image pickup apparatus 3.

In step S1400, the image acquisition unit 100 activates the lighting device 4 and the imaging device 3, and the lighting device 4 illuminates the stage 23.

In step S1402, the image acquisition unit 100 sets the variable “n” to n = 1.

In step S1404, the image acquisition unit 100 drives the drive mechanism 20 to adjust the positional relationship between the image pickup device 3 and the stage 23 to the initial positional relationship. In FIG. 14, as an example, in the initial positional relationship, the reference point Ref is located at the point Ref1 (see FIG. 3) in the Z-axis direction, and the image pickup apparatus 3 is located at the position P1 (see FIG. 3) in the X-axis direction. To position. It should be noted that this initial adjustment (positioning) to the positional relationship does not necessarily have to be precise.

In step S1406, the image acquisition unit 100 executes imaging by the image pickup apparatus 3 and acquires the nth image.

In step S1408, the mark recognition unit 101 recognizes three marks (pixel sets related to marks M1, M2, and M3) in the nth image based on the nth image obtained in step S1406. The pixel sets related to the marks M1, M2, and M3 are recognized by, for example, pattern matching.

In step S1410, the mark recognition unit 101 identifies two sides sandwiching a right angle among the triangles formed by connecting the three marks, based on the recognition results of the three marks obtained in step S1408. That is, the mark recognition unit 101 identifies the two sides related to the vector AB and the vector AC.

In step S1412, the mark recognition unit 101 recognizes the longer side of the two sides sandwiching the right angle specified in step S1410 as the side related to the vector AB. As shown in FIG. 2, the vector AB is a vector related to the longer side of the two sides sandwiching the right angle. As a result, which of the marks M1, M2, and M3 is determined as the three marks in the nth image, and the positions A to C of the marks M1, M2, and M3 in the image are determined. Will be done.

In step S1414, the inclination calculation unit 102 calculates the angle formed by the vector AB obtained in step S1412 and the vector x (the vector along the horizontal direction of the image) as the inclination angle θ. The method of calculating the inclination angle θ is as described above.

In step S1416, the ratio calculation unit 103 calculates the length (= D1 _AB ) of the vector AB obtained in step S1412, and the ratio (scale s = D1 _{AB ) to the actual distance D0 AB} between the two marks M1 and M2. / D0 _AB ) is calculated.

In step S1418, the reference approximate position calculation unit 104 outlines the reference point Ref based on the position of the mark M3 obtained in step S1412, the inclination angle θ obtained in step S1414, and the scale s obtained in step S1416. Calculate the position q. The method of calculating the approximate position q of the reference point Ref is as described above (see Equation 3).

In step S1420, the reference point calculation unit 105 sets the search range 70 centered on the approximate position q of the reference point Ref based on the approximate position q of the reference point Ref obtained in step S1418. The search range 70 is as described above with reference to FIG.

In step S1422, the reference point calculation unit 105 accurately calculates the position of the reference point Ref in the nth image obtained in step S1406 based on the search range 70 set in step S1420. The precise calculation method of the position of the reference point Ref is as described above. As a result, when n = 1, the parameter xR ₁ (see FIG. 3) is obtained, and when n = 3, the parameter xR ₂ (see FIG. 3) is obtained.

In step S1424, the image acquisition unit 100 increments the variable n by “1”.

In step S1426, the image acquisition unit 100 drives the drive mechanism 20 to move the image pickup device 3 to a distance b (FIG. 3) by the mechanism 20X so that the image pickup device 3 is located at the position P2 (see FIG. 3) in the X-axis direction. 3) Move in the negative direction of the X-axis. The movement of the distance b in the X-axis direction can be precisely realized by the mechanism 20X.

In step S1428, the image acquisition unit 100 executes imaging by the image pickup apparatus 3 and acquires the nth image.

In step S1430, the mark recognition unit 101 recognizes three marks (pixel sets related to marks M1, M2, and M3) in the nth image based on the nth image obtained in step S1406. Then, the mark recognition unit 101 derives the positions A to C of the marks M1, M2, and M3 in the image.

In step S1432, the reference approximate position calculation unit 104 outlines the reference point Ref based on the position of the mark M3 obtained in step S1430, the inclination angle θ obtained in step S1414, and the scale s obtained in step S1416. Calculate the position q. The method of calculating the approximate position q of the reference point Ref is as described above (see Equation 3). However, in the modified example, the reference approximate position calculation unit 104 is based on the approximate position q of the reference point Ref obtained in step S1418 (the approximate position q based on the image before moving the distance b), and the reference point Ref is as follows. The approximate position q = (x2n, y2n) of

ここで、（ｘ１ｎ，ｙ１ｎ）は、ステップＳ１４１８で得た基準点Ｒｅｆの概略位置qであり、（ｂｘ、ｂｙ）は、Ｘ軸方向の距離ｂの移動に伴うＸ軸方向及びＹ軸方向の移動量であり、ここでは、（ｂｘ、ｂｙ）＝（ｂ、０）である。この変形例の場合は、ステップＳ１４３０は不要である。

Here, (x1n, y1n) is the approximate position q of the reference point Ref obtained in step S1418, and (bx, by) is the X-axis direction and the Y-axis direction accompanying the movement of the distance b in the X-axis direction. It is the amount of movement, and here, (bx, by) = (b, 0). In the case of this modification, step S1430 is unnecessary.

In step S1434, the reference point calculation unit 105 sets the search range 70 centered on the approximate position q of the reference point Ref based on the approximate position q of the reference point Ref obtained in step S1432. The search range 70 is as described above with reference to FIG.

In step S1436, the reference point calculation unit 105 accurately calculates the position of the reference point Ref in the nth image obtained in step S1428 based on the search range 70 set in step S1434. The precise calculation method of the position of the reference point Ref is as described above. As a result, when n = 2, the parameter xL ₁ (see FIG. 3) is obtained, and when n = 4, the parameter xL ₂ (see FIG. 3) is obtained.

In step S1438, the image acquisition unit 100 determines whether or not the variable “n” is “4”, that is, whether or not n = 4. If n = 4, the process proceeds to step S1446, and in other cases (when n = 2), the process proceeds to step S1440.

In step S1440, the image acquisition unit 100 drives the drive mechanism 20 to move the image pickup device 3 to a distance b (FIG. 3) by the mechanism 20X so that the image pickup device 3 is located at the position P1 (see FIG. 3) in the X-axis direction. 3) Move in the positive direction of the X-axis. The movement of the distance b in the X-axis direction can be precisely realized by the mechanism 20X.

In step S1442, the image acquisition unit 100 drives the drive mechanism 20 to move the image pickup device 3 by the distance ΔZ (see FIG. 3) by the mechanism 20Z so that the reference point Ref is located at the point Ref2 in the Z-axis direction. Move in the negative axis direction. The movement of the distance ΔZ in the Z-axis direction can be precisely realized by the mechanism 20Z.

In step S1444, the image acquisition unit 100 increments the variable n by “1”. When step S1444 is completed, the process from step S1406 is repeated. However, in this case, step S1414 and step S1416 may be omitted. This is because the inclination angle θ and the scale s obtained when n = 1 can function effectively even when n = 3.

In step S1446, the calibration parameter value derivation unit 106 derives the f value based on the respective values of _{the parameters xL 1} , xL ₂ , xR ₁ , and xR _{2 obtained in steps S1422 and S1436.} The method for deriving the f value is as described above with reference to FIG. 3, and specifically, it is as follows.

校正パラメータ値導出部１０６は、導出したｆ値を校正パラメータ値記憶部１１０に記憶する。

The calibration parameter value derivation unit 106 stores the derived f value in the calibration parameter value storage unit 110.

According to the process shown in FIG. 14, the positional relationship between the calibration pattern and the image pickup apparatus 3 can be changed, and a highly accurate f-number can be derived based on the images obtained by the four positional relationships. As a result, highly accurate calibration can be realized.

FIG. 15 is a schematic flowchart showing an operation example of the control device 10 in the actual operation phase. The process shown in FIG. 15 is activated, for example, when the robot places a sample on the stage 23 and a distance calculation command is input to the control device 10. The sample is placed on the stage 23 so that the point of interest is in the field of view of the imaging device 3.

In step S1500, the image acquisition unit 100 activates the lighting device 4 and the imaging device 3, and the lighting device 4 illuminates the stage 23. If the lighting device 4 and the imaging device 3 have already been activated, step S1500 is omitted.

In step S1502, the image acquisition unit 100 executes imaging by the imaging device 3 and acquires an image (an image obtained by imaging the sample) before moving the imaging device 3 in the X-axis direction.

In step S1504, the point of interest position deriving unit 107 derives the position of the point of interest (position in the image) of the sample based on the image obtained in step S1502. For example, the points of interest in a sample may be recognized in a way such as pattern matching. As a result, the parameter xR (for example, the value of the x coordinate) corresponding to the position of the point of interest of the sample is obtained.

In step S1506, the image acquisition unit 100 drives the drive mechanism 20 to move the image pickup apparatus 3 in the negative direction of the X-axis by the distance b1 by the mechanism 20X. The distance b1 may be the same as or different from the above-mentioned distance b (see FIG. 3).

In step S1508, the image acquisition unit 100 executes imaging by the imaging device 3 and acquires an image (an image obtained by imaging a sample) after the imaging device 3 is moved in the X-axis direction.

In step S1510, the point of interest position deriving unit 107 derives the position of the point of interest (position in the image) of the sample based on the image obtained in step S1508. As a result, the parameter xL (for example, the value of the x coordinate) corresponding to the position of the point of interest of the sample is obtained.

In step S1512, the distance calculation unit 108 pays attention to the sample from the image pickup apparatus 3 based on the parameter xR obtained in step S1504, the parameter xL obtained in step S1510, and the f value in the calibration parameter value storage unit 110. Calculate the distance ds to the point. The distance ds from the image pickup apparatus 3 to the point of interest of the sample is the distance in the Z-axis direction, which is as described above, but can be calculated as follows.

尚、ｂ１は、ステップＳ１５０６で移動させた距離である。

Note that b1 is the distance moved in step S1506.

According to the process shown in FIG. 14, the distance to the point of interest of the sample can be calculated accurately using the f value calculated accurately as described above.

Although each embodiment has been described in detail above, the present invention is not limited to a specific embodiment, and various modifications and changes can be made within the scope of the claims. It is also possible to combine all or a plurality of the components of the above-described embodiment.

For example, in the above-described embodiment, the values of the parameters xL ₁ , xL ₂ , xR ₁ , and xR ₂ are used for deriving the f value, but the derivation of the f value is not limited to this. For example, when Z _{1 related to the parameters xL 1} and xR ₁ can be known, it is also possible to derive the f value from the above equation (2) using the _{parameters xL 1} and xR _1.

Further, in the above-described embodiment, in order to obtain a plurality of images in which the positional relationship between the calibration pattern and the image pickup device 3 is different in the calibration phase, the calibration pattern and the image pickup device are in the X-axis direction corresponding to the horizontal direction of the image. The positional relationship with 3 is changed, but the present invention is not limited to this. For example, in the calibration phase, in order to obtain a plurality of images having different positional relationships between the calibration pattern and the image pickup device 3, the position between the calibration pattern and the image pickup device 3 in the Y-axis direction corresponding to the vertical direction of the images. Relationships may change.

The following additional notes will be further disclosed with respect to the above examples.
[Appendix 1]
An image obtained by imaging a calibration pattern extending in a plane with two or more marks added to a regular pattern with an optical system, and the image obtained by imaging the calibration pattern and the optical system at the time of imaging. An image acquisition unit that acquires multiple images with different positional relationships,
For each image, a mark recognition unit that recognizes the two or more marks in the image, and a mark recognition unit.
Based on the recognition result by the mark recognition unit, the reference approximate position calculation unit that calculates the approximate position of the reference point of the regular pattern in the image, and the reference approximate position calculation unit.
For each image, the position of the reference point of the regular pattern in the image is calculated based on the approximate position calculated by the reference approximate position calculation unit and the feature amount in the predetermined region of the image. Reference point calculation unit and
A calibration device including a calibration parameter value derivation unit that derives a calibration parameter value of the optical system based on the position of the reference point for each image calculated by the reference point calculation unit.
[Appendix 2]
It further includes an inclination calculation unit that calculates the inclination of the calibration pattern with respect to the reference line in the plane based on the recognition result by the mark recognition unit.
The calibration device according to Appendix 1, wherein the reference approximate position calculation unit further calculates the approximate position based on the inclination.
[Appendix 3]
The calibration device according to Appendix 2, wherein the inclination calculation unit calculates the inclination based on the horizontal direction in the image and the direction of a vector connecting the two marks in the image.
[Appendix 4]
The two or more marks include a first mark, a second mark, and a third mark, and the first mark and the second mark are separated by a first distance in a direction parallel to the reference line. The first mark and the third mark are separated by a second distance shorter than the first distance in the direction perpendicular to the reference line in the plane.
The calibration device according to Appendix 3, wherein the inclination calculation unit calculates the inclination based on the horizontal direction in the image and the direction of a vector connecting the first mark and the second mark in the image.
[Appendix 5]
A ratio calculation unit for calculating the ratio between the actual distance between the two marks and the distance between the two marks in the image based on the recognition result by the mark recognition unit is further included.
The calibration device according to any one of Supplementary note 1 to 4, wherein the reference approximate position calculation unit further calculates the approximate position based on the ratio.
[Appendix 6]
The regular pattern is a checker pattern in which two types of rectangular portions having different colors are arranged alternately and continuously in each of two orthogonal directions in the plane.
The reference point is an intersection of the checker patterns.
The reference point calculation unit sets a search range smaller than the four rectangular portions of the checker pattern in such a manner that the center of the search range is located at the approximate position, and the reference point is set within the search range. The calibration device according to any one of Supplementary note 1 to 5, which calculates a position.
[Appendix 7]
The calibration device according to Appendix 6, wherein the two or more marks are given in a rectangular portion different from the rectangular portion forming the reference point.
[Appendix 8]
At least one of the two or more marks is given in the rectangular portion forming the reference point.
The calibration according to Appendix 6, wherein the reference point calculation unit derives the feature amount based on the image obtained by removing the pixel value related to the mark given in the rectangular portion forming the reference point. apparatus.
[Appendix 9]
The feature amount is an inner product of a vector connecting the first position and the second position in the search range and a gradient vector at the second position.
The reference point calculation unit changes the first position with the approximate position as the initial position of the first position, and sets the first position where the inner product becomes 0 even if a plurality of positions are set as the second position. The calibration device according to any one of Appendix 6 to 8, which is calculated as the position of the reference point.
[Appendix 10]
The plurality of images include first to fourth images obtained when the positional relationship between the calibration pattern and the optical system is the first to fourth positional relationships.
The first positional relationship and the second positional relationship are such that the distance between the calibration pattern and the optical system in the first direction perpendicular to the plane is the same at the third distance, but is parallel to the plane. The distance between the calibration pattern and the optical system in the second direction is different.
The third positional relationship and the fourth positional relationship are the same as the distance between the calibration pattern and the optical system in the first direction at a fourth distance different from the third distance, but in the second direction. The calibration apparatus according to any one of Supplementary note 1 to 9, wherein the distance between the calibration pattern and the optical system is different.
[Appendix 11]
The calibration device according to any one of Supplementary note 1 to 10, wherein the calibration parameter is an f value.
[Appendix 12]
The calibration device according to any one of Supplementary note 1 to 11, wherein the mark has a circular shape.
[Appendix 13]
An image obtained by imaging a calibration pattern extending in a plane with two or more marks on a regular pattern with an optical system, and the image obtained by imaging the calibration pattern and the optical system. Acquire multiple images with different positional relationships,
For each image, the two or more marks in the image are recognized,
Based on the recognition result of the mark, the approximate position of the reference point of the regular pattern in the image is calculated.
For each image, the position of the reference point of the regular pattern in the image is calculated based on the approximate position and the feature amount in the predetermined region of the image.
Including deriving the values of the calibration parameters of the optical system based on the position of the reference point for each image.
A calibration method performed by a computer.
[Appendix 14]
An image obtained by imaging a calibration pattern extending in a plane with two or more marks on a regular pattern with an optical system, and the image obtained by imaging the calibration pattern and the optical system. Acquire multiple images with different positional relationships,
For each image, the two or more marks in the image are recognized,
Based on the recognition result of the mark, the approximate position of the reference point of the regular pattern in the image is calculated.
For each image, the position of the reference point of the regular pattern in the image is calculated based on the approximate position and the feature amount in the predetermined region of the image.
Based on the position of the reference point for each image, the value of the calibration parameter of the optical system is derived.
A proofreading program that lets a computer perform processing.
[Appendix 15]
Optical system and
An image obtained by imaging a calibration pattern extending in a plane with two or more marks on a regular pattern with the optical system, and the calibration pattern and the optical system at the time of imaging. An image acquisition unit that acquires multiple images with different positional relationships,
For each image, a mark recognition unit that recognizes the two or more marks in the image, and a mark recognition unit.
Based on the recognition result by the mark recognition unit, the reference approximate position calculation unit that calculates the approximate position of the reference point of the regular pattern in the image, and the reference approximate position calculation unit.
For each image, the position of the reference point of the regular pattern in the image is calculated based on the approximate position calculated by the reference approximate position calculation unit and the feature amount in the predetermined region of the image. Reference point calculation unit and
A calibration system including a calibration parameter value derivation unit that derives a calibration parameter value of the optical system based on the position of the reference point for each image calculated by the reference point calculation unit.

1 System 2 Sample stage device 3 Imaging device 4 Lighting device 5 Output device 6A Stage controller 6B Lighting controller 6C Imaging controller 8 Peripheral device 10 Control device 14 Auxiliary storage device 15 Drive device 16 Recording medium 17 Communication interface 19 Bus 20 Drive mechanism 20X mechanism 20Y mechanism 20Z mechanism 22 fixed unit 23 stage 25 wired transmission / reception unit 26 wireless transmission / reception unit 70 search range 100 image acquisition unit 101 mark recognition unit 102 inclination calculation unit 103 ratio calculation unit 104 reference approximate position calculation unit 105 reference point calculation unit 106 calibration parameter Value derivation unit 107 Point of interest position derivation unit 108 Distance calculation unit 110 Calibration parameter value storage unit

Claims (9)

An image obtained by imaging a calibration pattern extending in a plane with two or more marks added to a regular pattern with an optical system, and the image obtained by imaging the calibration pattern and the optical system at the time of imaging. An image acquisition unit that acquires multiple images with different positional relationships,
For each image, a mark recognition unit that recognizes the two or more marks in the image, and a mark recognition unit.
Based on the recognition result by the mark recognition unit, the reference approximate position calculation unit that calculates the approximate position of the reference point of the regular pattern in the image, and the reference approximate position calculation unit.
For each image, the position of the reference point of the regular pattern in the image is calculated based on the approximate position calculated by the reference approximate position calculation unit and the feature amount in the predetermined region of the image. Reference point calculation unit and
A calibration parameter value deriving unit that derives a calibration parameter value of the optical system based on the position of the reference point for each image calculated by the reference point calculation unit .
An inclination calculation unit that calculates the inclination of the calibration pattern with respect to the reference line in the plane based on the recognition result by the mark recognition unit.
Only including,
The reference approximate position calculation unit further calculates the approximate position based on the inclination .
Calibration device. The tilt calculation unit, based on the direction of the vector connecting the horizontal direction, two of the marks in the image in the image, calculates the tilt, the calibration apparatus according to claim 1. A ratio calculation unit for calculating the ratio between the actual distance between the two marks and the distance between the two marks in the image based on the recognition result by the mark recognition unit is further included.
The calibration device according to claim 1 or 2 , wherein the reference approximate position calculation unit further calculates the approximate position based on the ratio. The regular pattern is a checker pattern in which two types of rectangular portions having different colors are arranged alternately and continuously in each of two orthogonal directions in the plane.
The reference point is an intersection of the checker patterns.
The reference point calculation unit sets a search range smaller than the four rectangular portions of the checker pattern in such a manner that the center of the search range is located at the approximate position, and the reference point is set within the search range. The calibration device according to any one of claims 1 to 3, which calculates a position. The calibration device according to claim 4 , wherein the two or more marks are given in a rectangular portion different from the rectangular portion forming the reference point. At least one of the two or more marks is given in the rectangular portion forming the reference point.
The fourth aspect of the present invention, wherein the reference point calculation unit derives the feature amount based on the image obtained by removing the pixel value related to the mark given in the rectangular portion forming the reference point. Calibration device. An image obtained by imaging a calibration pattern extending in a plane with two or more marks on a regular pattern with an optical system, and the image obtained by imaging the calibration pattern and the optical system. Acquire multiple images with different positional relationships,
For each image, the two or more marks in the image are recognized,
Based on the recognition result of the mark, the inclination of the calibration pattern with respect to the reference line in the plane is calculated.
Based on the inclination, the approximate position of the reference point of the regular pattern in the image is calculated.
For each image, the position of the reference point of the regular pattern in the image is calculated based on the approximate position and the feature amount in the predetermined region of the image.
Including deriving the values of the calibration parameters of the optical system based on the position of the reference point for each image.
A calibration method performed by a computer. An image obtained by imaging a calibration pattern extending in a plane with two or more marks on a regular pattern with an optical system, and the image obtained by imaging the calibration pattern and the optical system. Acquire multiple images with different positional relationships,
For each image, the two or more marks in the image are recognized,
Based on the recognition result of the mark, the inclination of the calibration pattern with respect to the reference line in the plane is calculated.
Based on the inclination, the approximate position of the reference point of the regular pattern in the image is calculated.
For each image, the position of the reference point of the regular pattern in the image is calculated based on the approximate position and the feature amount in the predetermined region of the image.
Based on the position of the reference point for each image, the value of the calibration parameter of the optical system is derived.
A proofreading program that lets a computer perform processing. Optical system and
An image obtained by imaging a calibration pattern extending in a plane with two or more marks on a regular pattern with the optical system, and the calibration pattern and the optical system at the time of imaging. An image acquisition unit that acquires multiple images with different positional relationships,
For each image, a mark recognition unit that recognizes the two or more marks in the image, and a mark recognition unit.
Based on the recognition result by the mark recognition unit, the reference approximate position calculation unit that calculates the approximate position of the reference point of the regular pattern in the image, and the reference approximate position calculation unit.
For each image, the position of the reference point of the regular pattern in the image is calculated based on the approximate position calculated by the reference approximate position calculation unit and the feature amount in the predetermined region of the image. Reference point calculation unit and
A calibration parameter value deriving unit that derives a calibration parameter value of the optical system based on the position of the reference point for each image calculated by the reference point calculation unit .
An inclination calculation unit that calculates the inclination of the calibration pattern with respect to the reference line in the plane based on the recognition result by the mark recognition unit.
Only including,
The reference approximate position calculation unit further calculates the approximate position based on the inclination .
Calibration system.

Images