JP2004199331A - Image converting method and image converting device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To image an object to be imaged such as an object to be inspected from the oblique upper part, and to convert the imaged image data into image data imaged from the right upper part of the object to be imaged. <P>SOLUTION: This image converting method for converting image data obtained by imaging an object to be imaged from an oblique direction into image data obtained by imaging the object from the direction of the right upper part, comprises a coordinate converting process and a re-sampling process. The coordinate converting process is provided to decide a relation between plane coordinates for deciding the position on a light receiving plane on which the oblique image is projected and plane coordinates for deciding the position on the light receiving plane on which the right upper image is projected. The re-sampling process is provided to integrate the light intensity information of one pixel configuring the right upper image data from the optical information of a plurality of pixels configuring the oblique image data by carrying out weighting. This weighting is carried out based on the area of overlapping when one pixel configuring the right upper image data is projected on the pixels configuring the oblique image data to be overlapped. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention is directed to a method and apparatus for capturing an image of an imaging target such as an inspection target from an oblique direction, and converting the captured image data into image data captured from a direction directly facing the imaging target, and a method and an apparatus for these. The present invention relates to a semiconductor manufacturing apparatus and a manufacturing inspection line used.
[0002]
[Prior art]
When measuring the position of an object from the process of creating a map from an aerial photograph or the position on the photograph, the position of the object on the real space coordinates (hereinafter sometimes referred to as "world coordinates") and the object It is necessary to know the correspondence between the photographed image (hereinafter also referred to as “image”) and the position on the screen coordinates. Also, there is a demand for capturing a target object and finding a pattern of a specific shape from the image based on the image of the target object projected on the imaging surface (hereinafter also referred to as “image search”). There is also. In this case, taking into account external parameters of the imaging device, such as the imaging direction, which is the direction of the target viewed from the imaging device, and internal parameters, such as the optical characteristics of the imaging system of the imaging device, the image being captured is It is necessary to perform processing such as image search in consideration of distortion.
[0003]
Conventionally, a reference point is provided in an object, the world coordinate value of the reference point is associated with the screen coordinate value, and the external and internal parameters of the image capturing apparatus are obtained with high accuracy. There is a method of performing calibration (for example, see Patent Document 1). According to this method, regardless of the imaging direction or the distance between the target object and the imaging device, if the calibration of the imaging device is performed and an unknown point is specified on the image, the world coordinates of the unknown point Can be obtained. Therefore, the world coordinates of the unknown point on the object can be obtained using the image of the object, which is a useful technique when measuring the position of the object from the position on the photograph.
[0004]
Further, in a semiconductor manufacturing apparatus such as a stepper, there is a wafer orientation flat detector (hereinafter, referred to as Patent Documents 2 and 3) as a device for detecting an orientation flat (hereinafter, abbreviated as “Orientation flat”) of a semiconductor wafer. ). In this wafer orientation flat detector, an imaging device for imaging a wafer (object to be imaged) having an orientation flat is installed at a position directly facing the wafer so that the imaged object is not distorted and imaged.
[0005]
[Patent Document 1]
JP 2001-264037 A
[Patent Document 2]
JP-A-9-14918
[Patent Document 3]
JP 9-14938 A
[0006]
[Problems to be solved by the invention]
However, in the conventional method based on the imaging device calibration, an unknown point is specified on a captured image plane to obtain world coordinates of a target unknown point. That is, the conventional method merely associates a position in the image (screen coordinates) with a position in the space (world coordinates) based on the image. When the screen search is performed, there is a problem that an object is distorted. In other words, as a model pattern that is a search purpose, usually, a shape facing directly from the pattern, for example, a shape viewed from directly above is given, and from an image captured in an oblique direction based on that, This is because a model pattern is searched. This means that it is difficult to perform a screen search unless an image captured in an oblique direction is converted into an image captured in a directly facing direction by some method.
[0007]
In the prior art wafer orientation flat detectors (for example, see Patent Literatures 2 and 3), an imaging device that captures an image of a wafer (object to be imaged) uses an upper surface of the wafer so that the wafer is not distorted and captured. It is installed in a vertical position with respect to. However, since the semiconductor manufacturing equipment is installed in a clean room, it is necessary to make the size as small as possible, and when it is difficult to mount the imaging device so that the wafer can be imaged from a direction perpendicular to the upper surface of the wafer. There is.
[0008]
The screen search technique is an important technique other than the detection of the orientation flat of the semiconductor manufacturing apparatus, for example, for monitoring or inspecting an article in a manufacturing line including a plurality of manufacturing processes. Therefore, even in the monitoring and inspection of articles on these production lines, there is a need for a technique for converting an image captured from an oblique direction into an image captured from a directly facing direction.
[0009]
[Means for Solving the Problems]
Therefore, the image conversion method and the image conversion apparatus of the present invention have the following steps or means.
[0010]
That is, a method and an apparatus according to the present invention provide an image pickup apparatus configured with an imaging optical system and a light receiving plane, which images an image of an object to be imaged from an oblique direction and receives the image on an oblique light receiving plane (hereinafter also referred to as a “first light receiving plane”). ) Is obtained on a directly-facing light-receiving plane (hereinafter also referred to as a "second light-receiving plane") when the imaging device temporarily takes an image of the imaging target from a directly facing direction. An image conversion method and apparatus for converting into second image data.
[0011]
First, coordinate conversion is performed to determine the relationship between the first plane coordinates for determining the pixel position on the oblique light receiving plane and the second plane coordinates for determining the pixel position on the directly-facing light receiving plane. Next, the light intensity of one pixel constituting the directly-facing light receiving plane is obtained as described below. If one pixel constituting the directly-facing light receiving plane is projected onto the oblique light receiving plane, light is received over a plurality of pixels constituting the oblique light receiving plane. Therefore, it is calculated that this one pixel is projected on which pixels constituting the oblique light receiving plane and over what area, and the amount of light received by the pixels constituting the oblique light receiving plane is calculated for each of a plurality of pixels. Weighting is performed in proportion to the area of the projected portion to obtain a weighted average, and resampling is performed by a method in which the value is used as the light intensity of one pixel forming the directly-facing light receiving plane.
[0012]
In carrying out the present invention, it is preferable to perform coordinate conversion for determining the relationship between the first plane coordinates and the second plane coordinates by the following procedure.
(A) Four or more points selected so that no three or more points are aligned on a straight line, or a plurality of points selected including three points on the straight line and including at least two points outside the straight line (Hereinafter, these points are referred to as “a plurality of characteristic points.”) The imaging object having a mark on the mark, and the coordinate value of the mark in a coordinate system provided on a plane on which the mark exists is known. Using the imaging target as a test pattern, the test pattern is imaged by the imaging device.
(B) Reading the coordinate values of the first plane coordinates with respect to marks on a plurality of characteristic points in the test pattern, which are imaged on the first light receiving plane when the image is imaged from an oblique direction.
(C) An arbitrary position in the test pattern is defined as an origin, and based on an image of the test pattern imaged obliquely on the first light receiving plane, the test pattern is defined by the second light receiving position. In the case where the image is taken from the direction opposite to the center projected on the center of the plane, the position of the center of the second light receiving plane on which the origin is projected and the image of the test pattern taken obliquely from the geometrical direction. Is virtually set.
(D) For a mark on a plurality of characteristic points imaged in oblique directions, a second light reception when each of the marks in the test pattern is imaged from a virtually set opposite direction. By projecting the mark on a plane, the position of the mark at the second plane coordinates is calculated.
(E) the first coordinate of each of the marks on the characteristic points in the test pattern projected on the first light receiving plane when the image is taken from an oblique direction, and a virtually opposite direction that is directly set; On the second light receiving plane when imaged from, based on the calculated second coordinates of the marks on the plurality of characteristic points in the calculated test pattern, coordinate conversion parameters b _i (However, i = 1, 2, 3, 4, 5, 6, 7, 8).
[0013]
In order to perform the calibration step of virtually setting the center position and the scale in the second plane coordinate system when capturing an image from the directly facing direction, it is preferable to perform the following.
(C ′) The scale is determined so that the size of the image at the position of the center of the first light receiving plane and the position of the center of the second light receiving plane virtually set are equal, and The position of the center portion is virtually set when imaging the object to be imaged from the directly facing direction so that the position of the center portion matches.
[0014]
In practicing the present invention, it is preferable that resampling is performed by obtaining the light intensity Img2 (I, J) of the pixel [I, J] of the second image data according to the following equation.
[0015]
Img2 (I, J) = Σ [W (k, m) × Img1 (k, m)] / ΣW (k, m)
Here, I, J, k, and m are integers, and Img1 (k, m) is the light intensity of the pixel [k, m] at the k-th row and the m-th column of the first image data. W (k, m) is the pixel [I, J] in the I-th row and the J-th column of the second image data projected on the first light receiving plane, and is located on the pixel [k, m] on the first light receiving surface. Represents the area projected to
[0016]
The range for calculating the sum of the right side of the above equation is such that the pixel [I, J] of the second image data projected on the first light receiving plane is one pixel above the pixel [k, m] on the first light receiving surface. If it is also projected on the part, it is assumed that it extends over all the projected pixels [k, m].
[0017]
According to the image conversion method or the image conversion apparatus of the present invention, the first image data (hereinafter, also referred to as “oblique image data”) obtained by imaging the imaging target from an oblique direction is placed directly above the imaging target. The image data is converted into second image data (hereinafter also referred to as “directly above image data”) obtained by imaging from the direction. Since the image of the image data directly above is similar to the object, an existing program library that realizes the following various image processing functions can be effectively used.
[0018]
That is, identification of the object, detection of the difference between the object and the standard, measurement of the position of the object, measurement of the shape of the object, and the like. Since these existing program libraries and the like are made on the premise that they are applied to an image obtained by capturing an object from directly above, they are applied to image data obtained by capturing an object from obliquely above. In such a case, it is an essential condition that the captured oblique image data be converted into image data directly above.
[0019]
According to the screen search technique, the image conversion method of the present invention can effectively perform a screen search in detecting an orientation flat in a semiconductor manufacturing apparatus, and monitoring and inspecting a manufacturing line including a plurality of manufacturing processes.
[0020]
Further, in a semiconductor manufacturing apparatus having a wafer cassette for stocking a plurality of wafers and a transfer means for taking out wafers one by one from the wafer cassette and transferring the wafers to a stage, the image conversion apparatus of the present invention is provided. It is suitable. According to the semiconductor manufacturing apparatus provided with the image conversion device of the present invention, the oblique image of the wafer acquired by the imaging device installed obliquely with respect to the upper surface of the wafer is converted into the directly-facing image. The orientation flat can be detected by using a well-known image search technique for. Therefore, it is not always necessary to install the imaging device at a position directly facing the upper surface of the wafer.
[0021]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, an embodiment of the present invention will be described with reference to FIGS. Note that the drawings merely schematically show the shapes, sizes, and arrangements of the components so that the present invention can be understood, and the present invention is not limited to the illustrated examples.
[0022]
With reference to FIG. 1, a description will be given of a case where an image of an object to be imaged is taken from a direction directly opposite, for example, a position directly above, and a case where an image is taken obliquely from above. In the following description, the case where an image is taken from the directly facing direction only assumes a virtual model, and does not necessarily mean that an imaging device or the like exists. Further, for convenience of explanation, a lattice pattern 14 on a plane is used as an imaging target, but the technical content described below is valid regardless of whether the target is a lattice pattern. .
[0023]
The imaging device includes an imaging optical system and a light receiving plane. A position where the imaging device is directly opposed to the imaging target 14 and an image is taken is taken as a directly facing position B, and a position where an image is taken in an oblique direction is taken as an oblique position A. In the description of this configuration example, when the imaging device is disposed at the oblique position A, this imaging device is displayed as the first imaging device 10, its imaging optical system is shown as 10a, and its light receiving plane is shown as the oblique light receiving plane 10b. I do. Further, when an apparatus equivalent to the image pickup apparatus arranged at the oblique position A is virtually arranged at the directly opposite position B, this image pickup apparatus is referred to as a second image pickup apparatus 12, its imaging optical system is referred to as 12a, and Each is shown as a light receiving plane 12b.
[0024]
To image an object to be imaged from the directly facing direction means to install the image capturing device 12 in the normal direction of the plane on which the lattice pattern 14 exists and perform image capturing. Therefore, the optical axis of the imaging optical system 12a constituting the imaging device 12 is parallel to this normal direction, and the light receiving plane is perpendicular to this normal direction. On the other hand, the case where the imaging target 14 is imaged from an oblique direction means that the imaging device 10 is installed in a direction deviated from the normal direction of the plane on which the lattice pattern exists, and performs imaging.
[0025]
In the configuration example shown in FIG. 1, the directly-facing position B is a position in a direction directly above (facing direction) with respect to the imaging object 14, and the oblique position A is an obliquely upward direction with respect to the imaging object 14. Position. For the sake of simplicity, the x 'axis is set in the direction parallel to the left and right, and the y' axis is set in the direction perpendicular to the lattice pattern 14 regularly arranged in a matrix as shown in FIG. , The case where the imaging device 10 is installed in a direction rotated clockwise about the y ′ axis will be described.
[0026]
<Coordinate transformation>
It is assumed that the first imaging device 10 and the second imaging device 12 image a lattice pattern in this manner. As shown in FIGS. 2A, 2B, and 2C, images 18a and 18b of a lattice pattern are captured on the light receiving planes of the first and second imaging devices 10 and 12, respectively. 2 (A) and 2 (B), the grid pattern indicated by a thin line indicates the image 18a projected on the light receiving plane 10b of the first imaging device 10, and the grid pattern indicated by a thick line indicates the second pattern. An image 18b projected on the light receiving plane 12b of the imaging device 12 is shown, and each of them is shown in FIG.
[0027]
On the light receiving plane, a minute light receiving device (hereinafter, each of the light receiving devices is sometimes referred to as a “pixel”) that receives light and generates a voltage or a current proportional to the intensity of the light is uniformly provided. They are arranged in a matrix. A set of light receiving intensity sets corresponding to pixels arranged in order on the light receiving plane may be hereinafter referred to as “image data”. In order to identify individual pixels in order on the light receiving plane, a rectangular coordinate axis is provided on the light receiving plane, and along this rectangular coordinate axis, the i-th position in the horizontal axis counting from the starting point and the j-th position on the vertical axis. A pixel is referred to as a pixel [i, j]. That is, the pixel [i, j] is a pixel on the i-th row and the j-th column. Details will be described later.
[0028]
When converting the first image data captured by the first imaging device 10 into second image data captured by the second imaging device 12 having the same structure as the first imaging device 10, the first image It is important to convert the image quality of the data image so that the image quality of the second image data is not deteriorated as much as possible.
[0029]
When performing image conversion from an oblique image to a directly-facing image, in order to minimize the image quality as image data, the central part of the light receiving plane 10b of the first imaging device 10 and the central part of the light receiving plane 12b of the second imaging device 12 The scale at the time of coordinate conversion from the xy coordinate system to the XY coordinate system may be determined so that the dimensions of the image become equal. Here, a description will be given using a lattice pattern as an imaging target. 2A to 2C, the lattice pattern 18b indicated by a thick line is a directly-facing image, and the lattice pattern 18a indicated by a thin line is an oblique image. That is, in the figure, the scale is set so that the interval between the grid points arranged on the thick line shown by the symbol A in the image 18b is equal to the interval between the grid points arranged on the thin line shown by the symbol B in the image 18a. Confirm.
[0030]
Incidentally, a thick straight line indicated by a symbol A is a straight line connecting grid points arranged in the y'-axis direction in the central portion of the light receiving plane 12b of the second imaging device 12, and this straight line and the grid in the horizontal direction (x'-axis direction). The intersection with the thick straight line connecting the points is the position of the lattice point imaged on the light receiving plane 12b of the second imaging device 12. Further, the thin straight line indicated by the symbol b is a straight line in the center of the light receiving plane 10b of the first imaging device 10, which is a line of grids similarly arranged in the y'-axis direction. Therefore, in the central part of the light receiving plane 10b, the grid points indicated by the thick lines imaged by the second imaging device 12 are arranged such that the intervals between the grid points linearly arranged along the straight lines indicated by the symbols A and B are equal. When the scale of the pattern image 18b is determined, the image quality of the image-converted directly-facing image is minimized with respect to the entire screen.
[0031]
In order to define the positional relationship between the oblique light receiving plane 10b of the first imaging device 10 and the directly-facing light receiving plane 12b of the second imaging device 12, (x, y) and (X, Y) Each coordinate system is provided. As is well known, there is a relationship given by the following equation between the two coordinate systems. The conversion from the (x, y) coordinate system of the oblique light receiving plane 10b to the (X, Y) coordinate system of the directly-facing light receiving plane 12b based on the relationship given by the following equation is hereinafter referred to as “coordinate conversion”.
[0032]
X = (b ₁ x + b _Two y + b _Three ) / (b ₇ x + b ₈ y + 1) (1)
Y = (b _Four x + b _Five y + b ₆ ) / (b ₇ x + b ₈ y + 1) (2)
Where b _i (However, i = 1, 2, 3, 4, 5, 6, 7, 8) is a parameter of the coordinate transformation. Since there are a total of eight, if the relationship of four or more corresponding points on the light receiving planes 10b and 12b of the first imaging device 10 and the second imaging device 12 is known, the parameter b _i (Where i = 1, 2, 3, 4, 5, 6, 7, 8).
[0033]
That is, as shown in FIG. 2C, points corresponding to points p, q, r, and s on the oblique light receiving plane 10b of the first imaging device 10 are located on the directly-facing light receiving plane 12b of the second imaging device 12. Let the coordinates of these eight points be (x _p , Y _p ), (X _q , Y _q ), (X _r , Y _r ), (X _s , Y _s ), (X _P , Y _P ), (X _Q , Y _Q ), (X _R , Y _R ), (X _S , Y _S ). And (x _p , Y _p ) And (X _P , Y _P ), (X _q , Y _q ) And (X _Q , Y _Q ), (X _r , Y _r ) And (X _R , Y _R ), (X _s , Y _s ) And (X _S , Y _S ), The parameter b can be obtained by a method that is analytically well known. _i (Where i = 1, 2, 3, 4, 5, 6, 7, 8).
[0034]
Here, a method for determining the correspondence between the coordinate positions of the oblique light receiving plane 10b and the directly-facing light receiving plane 12b will be described. In order to obtain this correspondence, a grid-like test pattern is provided as the imaging target 14. As shown in FIG. 1, four points p ′, q ′, r ′ and s ′ are defined in this test pattern. The test pattern is imaged by the first imaging device 10 to obtain a first image (an image projected on the first light receiving plane).
[0035]
In the second imaging device 12, the coordinates of the points corresponding to the points p ′, q ′, r ′, and s ′ on the test pattern on the second light receiving plane are defined assuming that the second imaging device 12 is a virtual model. Determined by calculation. First, the origin is determined for the points p ′, q ′, r ′ and s ′ on the test pattern. The second imaging device 12 is installed in a direction normal to the plane of the test pattern. Accordingly, on the light receiving plane 12b of the second imaging device 12, the positional relationship between the origin of the test pattern and the points p ', q', r ', and s' is similar. The X-axis direction and the Y-axis direction of the second imaging device 12 are matched with the x ′ direction and the y ′ direction of the grid on the test pattern, respectively, and when the similarity ratio is determined, the points P, Q, R and The position of S is determined. That is, the coordinate values of points P, Q, R, and S in FIG. 2C are calculated.
[0036]
The coordinates of the points p, q, r and s in the first image thus obtained are read by the first imaging device 10 and stored in a memory (storage device), and the second image (the second light receiving plane) is read. Are stored in the memory (storage device). As described above, since the first and second imaging devices 10 and 12 are one imaging device having an equivalent light receiving plane, that is, an imaging surface, the (x, y) coordinates of the oblique light receiving plane and the directly facing light receiving plane are used. The (X, Y) coordinate system of the plane is an equivalent plane orthogonal coordinate system (that is, mutually coordinate-transformable).
[0037]
Therefore, each of the points p ′, q ′, r ′ and s ′ in the test pattern is imaged at two coordinate positions in the same coordinate system. That is, p 'is imaged at points p and P, q' is q and Q, r 'is r and R, and s' is imaged at s and S. Since (p, q, r, s) and (P, Q, R, S) are in the same coordinate system, the corresponding relations between p and P, q and Q, r and R, and s and S, respectively. Is defined by the eight parameters of the above equations (2) and (3).
[0038]
When converting the image of the first image data to the image of the second image data, it is important that the image quality is not deteriorated as much as possible. For this purpose, the position of the second imaging device 12 is determined and the scale is adjusted so that the center portion of the first image data in the x direction matches the center portion of the second image data in the X direction. Hereinafter, a method for adjusting the scale will be described. At the grid points shown in FIG. 1, three points o ′, t ′ and u ′ are newly selected at the center. The three points, o, t, u, O, T, and U in the first and second images shown in FIG. 2C corresponding to the newly selected three points respectively correspond to the three points. When the first imaging device 10 images a test pattern, the position of the test pattern is adjusted so that o ′ is at the center of the first image. After that, the first imaging device 10 reads the positions of the seven points o ′, p ′, q ′, r ′, s ′, t ′ and u ′ on the test pattern. The origin of the second imaging device 12 is set so as to match the coordinates of the point o in the read first image (the position of the point O in the second image is determined). Then, the position of the point T in the second image is determined so that the interval between the points o and t in the first image is equal to the interval between the points O and T in the second image. At this time, the X coordinate of the T point is set to be the same as the O point, and only the Y coordinate is changed. When the position of the point T in the second image is determined in this manner, the scale of the second image data is determined. That is, the distance of the second imaging device 12 from the test pattern is determined. Since the second imaging device 12 is installed in the direction directly opposite to the point o 'of the test pattern, the points p', q ', r', s ', t', and u 'for the point o' on the test pattern. In the second image, the positional relationship is the position of each point with respect to the point O at the same similarity ratio. That is, the coordinates of the points P, Q, R, S, T, and U in the second image are calculated by performing similarity conversion on the point O. Since the seven points on the test pattern obtained in this way have a corresponding relationship on the light receiving plane of the first and second imaging devices, the number of expressions related to Expressions (2) and (3) is large. Therefore, the eight parameters of the equations (2) and (3) are obtained using the least squares method.
[0039]
Parameter b _i (Where i = 1, 2, 3, 4, 5, 6, 7, 8) is stored in the storage device, and any point on the light receiving plane 10b of the first imaging device 10 is Which point on the light receiving plane 12b of the second imaging device 12 corresponds to which point is uniquely determined using Expressions (2) and (3). In other words, the central processing unit (CPU) reads out the coordinate values and parameters stored in the storage device and the above equations (2) and (3) and calculates them to thereby determine the position on the light receiving plane 10b. It can be converted to the upper position and corresponded.
[0040]
Further, in the above-described example, seven points are selected on the light receiving planes 10b and 12b of the first imaging device 10 and the second imaging device 12, respectively, and the coordinate conversion parameters are obtained from the corresponding relationships. You don't have to choose 7 points like.
[0041]
The number of the selected plurality of points is at least four if at least three points are selected so as not to be aligned on a straight line. Further, when three points on a straight line are selected and at least two points outside the straight line are selected, the number of the selected plurality of points is at least five. That is, a plurality of points that satisfy the above-described conditions may be selected. Of course, the more the selected points, the higher the calculation accuracy of the required coordinate conversion parameters.
[0042]
Parameter b of the coordinate transformation of the coordinate transformation described above _i (However, i = 1, 2, 3, 4, 5, 6, 7, 8) The procedure (process) for obtaining is summarized as follows with reference to the flowchart shown in FIG.
[0043]
(a) Test pattern imaging step (ST1) of imaging a test pattern with the first imaging device 10
(b) Coordinate reading step (ST2) of reading the coordinates on the light receiving plane of a plurality of characteristic points in the test pattern on the light receiving plane 10b of the first imaging device 10
(c) Marks on a plurality of characteristic points imaged in oblique directions for setting the origin at an arbitrary position in the test pattern and setting the scale of the image in which the test pattern is imaged by the second imaging device 12 Calibration step for determining the center position and scale of the second imaging device, which geometrically resembles the position with respect to the origin (ST3)
(d) Using the center position and the scale set in the above step (d), on the light receiving plane 12b of the second imaging device 12, the light receiving plane of each of the points selected in the step (a) in the test pattern. Coordinate calculation step (ST4) for calculating the above coordinates
(e) coordinates of a plurality of characteristic points in the test pattern imaged on the light receiving plane 10b of the first imaging device 10 and the plurality of points in the test pattern calculated on the light receiving plane 12b of the second imaging device 12 Calculate the parameter to obtain the coordinate transformation parameter bi (where i = 1, 2, 3, 4, 5, 6, 7, 8) by solving simultaneous equations or the least squares method based on the coordinates of Process (ST5)
[0044]
Further, the steps (a) and (c) preferably include the following steps.
(a ′) In order to make the center point of the light receiving plane 10b of the first imaging device 10 coincide with the center point of the light receiving plane 12b of the second imaging device 12, a test pattern is provided at the center point of the light receiving plane 10b of the first imaging device 10. A test pattern position adjustment step of adjusting a test pattern installation position to adjust an installation position of the test pattern so that a lattice point located at the center on the upper side is imaged (ST1-1).
(c ′) The center point of the light receiving plane 12b of the second imaging device 12 is matched with the coordinate value read as the center point of the light receiving plane 10b of the first imaging device 10, and the center of the light receiving plane 10b of the first imaging device 10 is Calibration step of setting the scale of the second imaging device so that the size of the image is equal between the portion and the central portion of the light receiving plane 12b of the second imaging device 12 (ST3-1)
[0045]
Note that the parameter b of the coordinate conversion of the coordinate conversion described above is used. _i (However, i = 1, 2, 3, 4, 5, 6, 7, 8) An apparatus for realizing the procedure (process) for obtaining is described later.
[0046]
<Resampling>
With reference to FIG. 4, the coordinates provided on the light receiving plane of the imaging device and the index for designating the pixels constituting the light receiving plane will be described. Here, the pixels are arranged in a matrix, for example, and the shape of each pixel is assumed to be a square for the sake of simplicity, and the boundary region between adjacent pixels has only a negligible width. Since each pixel has light receiving sensitivity inside the square and the boundary area between adjacent pixels has only a negligible width, the description will be made assuming that the pixel has light receiving sensitivity everywhere on the light receiving plane.
[0047]
Assuming that the dimension of one side of the pixel is L, the center of the pixel is indicated by a black circle in FIG. 4 and the vertexes of the four corners of the pixel are indicated by crosses. As shown in FIG. 4, an xy coordinate system is used, and coordinate values are indicated as (x, y). On the other hand, an index for designating each pixel is represented as [i, j]. These i and j correspond to the i-th row and the j-th column. For example, the coordinates of the vertices of the four corners of the pixel specified by the index [0, 0] are (+ 0.5L, -0.5L), (+ 0.5L, + 0.5L), (-0.5L, + 0.5L). ) And (−0.5 L, −0.5 L), and the center coordinates of the pixel are (0, 0). Generally, the coordinates of the vertices of the four corners of the pixel specified by the index [i, j] are ((i + 0.5) L, (j-0.5) L), ((i + 0.5) L, (j + 0.5) L), ((i-0.5) L, (j + 0.5) L) and ((i-0.5) L, (j-0.5) L), and the center coordinates of the pixel are (iL, jL).
[0048]
Also, x or y representing coordinate values and index values i, j or k, m for designating pixels are displayed using lowercase letters as values relating to the light receiving plane of the first imaging device 10, that is, the oblique light receiving plane 10b. It is assumed that the value relating to the light receiving plane of the second imaging device 12, that is, the directly-facing light receiving plane 12b is displayed using capital letters. That is, the coordinate value of the light receiving plane 10b of the first imaging device 10 is (x, y), and the index for designating the pixels constituting the light receiving plane 10b of the first imaging device 10 is [i, j] or [k, m].
[0049]
On the other hand, the coordinate value of the light receiving plane 12b of the second imaging device 12 is (X, Y), and the index for specifying the pixels constituting the light receiving plane 12b of the second imaging device 12 is [I, J]. Note that the image data obtained from the second imaging device 12 is referred to as directly-facing image data. Here, since the directly-facing position is a position directly above the imaging target, it is also referred to as directly above image data. Further, image data obtained from the first imaging device 10 is referred to as oblique image data.
[0050]
With reference to FIG. 5, a method of acquiring image data of each pixel constituting the light receiving plane will be described. Figure 5 shows the index [i _p , j _q (Here, p = 1, 2, 3, 4, 5 and q = 1, 2) are enlarged and displayed. A case where a black band-like image having a width is formed on the pixel will be described. The shaded portion in FIG. 5 is the portion of the black image. It is assumed that the light does not reach the black portions indicated by oblique lines, and the light reaches uniform brightness to the other white portions. The light intensity received by each pixel is digitized into 256 gradations and received. That is, it is assumed that the pixel is set so as to output brightness information as digital data in which 0 indicates no light is received and 255 indicates the brightest light is received. In this case, the indicator [i _p , j _q (Where p = 1, 2, 3, 4, 5 and q = 1, 2), the digital value captured as one of the image data by the eight pixels is
[i ₁ , j _Two ] = 0, [i _Two , j _Two ] = 10, [i _Three , j _Two ] = 40, [i _Four , j _Two ] = 96, [i _Five , j _Two ] = 128,
[i ₁ , j ₁ ] = 20, [i _Two , j ₁ ] = 0, [i _Three , j ₁ ] = 0, [i _Four , j ₁ ] = 0, [i _Five , j ₁ ] = 0
And These values are in proportion to the area occupied by the bright portion in each pixel.
[0051]
Here, for the sake of simplicity, an example has been described in which an image is composed of two areas, a pure white part and a pure black part. However, the same applies to an image including a more complicated area of intermediate brightness. The digital value captured by a pixel as one of the image data is a value proportional to the product of the brightness of the image projected on the pixel and the area occupied by the brightness portion on the pixel.
[0052]
Next, referring to FIG. 6, a pixel for acquiring directly above image data (hereinafter, also simply referred to as a “directly above image pixel”) and a pixel for acquiring oblique image data (hereinafter simply referred to as “oblique image pixel”). This is sometimes referred to as "."). FIG. 6 is a diagram in which a pixel located in a range of i to i + 2 for a column position of an oblique image pixel and a pixel existing in a range of j to j + 2 for a row position of an oblique image pixel are separated by a thin line. Indicated. That is, the upper left pixel is a pixel given by the index [i, j], and the lower right pixel is a pixel given by the index [i + 2, j + 2]. In addition, the image directly above the pixel given by the index [I, J] is projected on the pixel from which the oblique image is obtained by a thick square ABCD. In the following, a pixel given by an index [I, J] or [i, j] is abbreviated as a pixel [I, J] or a pixel [i, j], respectively.
[0053]
The vertices A, B, C, and D of the pixel [I, J] projected on the oblique image pixel are A ((I−0.5) L, (J) on the light receiving plane 10b of the first imaging device 10, respectively. -(0.5) L), B ((I + 0.5) L, (J-0.5) L), C ((I + 0.5) L, (J + 0.5) L), D ((I-0.5) L, ( J + 0.5) L). The coordinates ((I-0.5) L, (J-0.5) L), ((I + 0.5) L, (J-0.5) L), ((I + 0.5) L) of the four vertices of the pixel [I, J] , (J + 0.5) L), and ((I-0.5) L, (J + 0.5) L) are located at positions A, B, C, and D on the light receiving plane 12a of the second imaging device 12, respectively. Projected.
[0054]
Next, the following equation (1) that gives resampling will be described.
Img2 (I, J) = Σ [W (k, m) × Img1 (k, m)] / ΣW (k, m) (1)
As shown in FIG. 6, consider a case where the pixel [I, J] is projected onto a pixel constituting the light receiving plane 10b of the first imaging device 10. In this case, the range for obtaining the sum indicated by Σ is a range from i to (i + 2) for k, and a range from j to j + 2 for m.
[0055]
Here, the meaning of W (k, m) will be described. W (k, m) represents the area of the pixel [I, J] projected on the pixel [k, m] forming the light receiving plane 10b of the first imaging device 10. That is, a part of the pixel [I, J] is projected on the pixel [k, m], but W (k, m) is the image of the pixel [I, J] on the pixel [k, m]. Represents the area of the projected and overlapping portion. The area of the overlapping portion is also referred to as “partial projection area”. FIG. 7 is a graph for explaining the meaning of W (k, m), where k is in the range from i to (i + 2) and m is in the range from j to j + 2. FIG. 9 is a diagram in which the portion where the image of the pixel [I, J] is projected and overlapped is indicated by oblique lines and displayed in an easily viewable manner. Each square indicated by a thin line indicates a pixel [k, m]. A hatched portion demarcated by a thick straight line is a polygon having an area given by W (k, m) (part of the projected area where the image of pixel [I, J] is projected on pixel [k, m]) ).
[0056]
Equation (1) shows that the pixel [I, J] is a range of pixels projected on the pixels constituting the light receiving plane 10b of the first imaging device 10 (k is a range from i to (i + 2), For m, the value obtained by weighting and averaging W (k, m) with respect to the range of j to j + 2 is used as the value of the pixel [I, J] on the light receiving plane 12b of the second imaging device 12 ( (Digital value related to brightness).
[0057]
Img1 (k, m) is a digital value (value of pixel [k, m]) relating to the brightness received by pixel [k, m], and W (k, m) is pixel [k, m] Since it is the image partial projection area of the pixel [I, J] above, W (k, m) × Img1 (k, m) is the value of the pixel [k, m] of only W (k, m). The value takes into account the weight. Σ [W (k, m) × Img1 (k, m)] is the sum of k over the range from i to (i + 2) and m over j from j to j + 2. Further, ΣW (k, m) represents the total projected area of the image of the pixel [I, J] projected on the light receiving plane 10b of the first imaging device 10. Therefore, the right side of Expression (1) represents the range of pixels (k is from i to (i + 2) for the range of pixels in which the pixel [I, J] is projected on the pixel constituting the light receiving surface of the first imaging device. For the range, m, the range from j to j + 2), a weighted average value with W (k, m) as the weight is given.
[0058]
Next, the procedure for determining W (k, m) will be described. FIG. 8 is a diagram for explaining a procedure for obtaining W (k, m), and is an enlarged view of a peripheral portion of a pixel [i, j]. Positions where the four vertices of the pixel [I, J] are projected on the light receiving plane 10b of the first imaging device 10 are A, B, C, and D. The four vertices of the pixel [i, j] on the light receiving plane 10b are A ', B', C ', and D'. As described above, these eight points are in the same (x, y) coordinate system.
[0059]
W (k, m) is determined by the following procedure (process).
(A) It is checked whether the points A, B, C, and D are included in the quadrilateral A'B'C'D '.
Pick up A because A is included.
(B) Check whether the points A ', B', C ', and D' are included in the quadrilateral ABCD.
Since C 'is included, pick up C'.
(C-1) The intersection of line segment A'B 'with line segment AB, line segment BC, line segment CD, line segment DA is examined.
There are no intersections.
(C-2) The intersection of the line segment B'C 'with the line segment AB, the line segment BC, the line segment CD, and the line segment DA is examined.
Since a is an intersection (exists as the intersection of line segment B'C 'and line segment AB), pick up a.
(C-3) The intersection of the line segment C'D 'with the line segment AB, the line segment BC, the line segment CD, and the line segment DA is examined.
Since b is an intersection (exists as the intersection of line segment C'D 'and line segment DA), b is picked up.
(C-4) The intersection of the line segment D'A 'with the line segments AB, BC, CD, and DA is examined.
There are no intersections.
(D) The top left point is found for the picked-up point (four points are registered in the list in the order of A, C ', a, b in this embodiment). Specifically, as shown in FIG. 9, each line is drawn one by one so that a straight line inclined at 45 degrees to the X axis passes through each point, and the intersection of the Y axis and the straight line inclined at 45 degrees is drawn. The smallest one of the Y coordinates is selected. The reason for selecting the smallest Y coordinate is that the coordinate set on the light receiving plane is a left-handed coordinate system. Thus, the first point is obtained. Here, it is point A.
(E-1) On the basis of a straight line passing through point A and having an inclination of 45 degrees, this line is rotated clockwise around point A to find the first contact point, and this is the second point And In this case, point a is obtained as the second point.
(E-2) Next, the straight line A a is rotated clockwise about the point a to find a first contact point, and this is set as a third point. In this case, point C ′ is obtained as the third point.
(E-3) Next, the straight line a C ′ is rotated clockwise about the point C ′ to find the first contact point, and this is set as the fourth point. In this case, point b is obtained as the fourth point.
(E-4) The above steps are sequentially performed on the picked-up points. By this process, the picked up points are arranged clockwise in order from the upper left.
(F) If the picked-up points are arranged clockwise in order from the top left, the coordinates of the four vertices of the quadrilateral have been determined, and the coordinates of the quadrilateral A a C ' The area of b, that is, W (k, m) can be obtained by a known method.
[0060]
Next, a method performed in steps (a) and (b) for checking whether or not the point of interest is included in the quadrilateral will be described with reference to FIG. FIG. 10A is a diagram showing a case where the point O is inside the quadrilateral EFGH, and FIG. 10B is a diagram showing a case where the point O is outside the quadrilateral EFGH. It is assumed that there is a point starting from the point E of the quadrilateral EFGH and moving on the side in the order of the points E, F, G, and H. At this time, the angle θ between the point E and the point F ₁ Is measured with the direction from point E to point F being a positive direction. Similarly, the angle between the point F and the point G is θ2, the angle between the point G and the point H is θ3, and the angle between the point H and the point E is θ4. The direction toward is measured in the positive direction.
[0061]
When the point O shown in FIG. 10A is inside the quadrilateral EFGH, θ ₁ + θ _Two + θ _Three + θ _Four Becomes 2π. On the other hand, when the point O shown in FIG. 10B is outside the quadrilateral EFGH, θ ₁ + θ _Two + θ _Three + θ _Four Becomes 0. Although not shown, when the point O is on the side of the quadrilateral EFGH, θ ₁ + θ _Two + θ _Three + θ _Four Becomes π. As described above, θ ₁ + θ _Two + θ _Three + θ _Four Is obtained, and if the value is 2π, 0, or π, it can be checked whether or not the point of interest O is included in the quadrilateral EFGH.
[0062]
In the steps (a) and (b) for obtaining W (k, m) described above, in the steps (a) and (b), even when the point to be picked up is on the side of the quadrilateral, Treated as
[0063]
Next, with reference to FIG. 11 and FIG. 12, intersections of line segments, which are performed in the above-described steps (c-1), (c-2), (c-3) and (c-4) The rules for requesting are explained. FIG. 11 illustrates a case where the line segments do not overlap, and FIG. 12 illustrates a case where the line segments overlap. If an intersection exists on the extension of the line segment as shown in FIG. 11 (A), it is treated as having no intersection. It is determined that the intersection exists, as shown in FIG. 11B, when the intersection exists within the range where the line segment exists. As shown in FIG. 11C, it is determined that an intersection exists even when the end point of the line segment is an intersection. When the thin line segment 1 and the thick line segment 2 overlap as shown in FIG. 12, two points are picked up as intersections.
Further, the following processing is performed as a derivative step.
[0064]
(A) In step (a) described above, if all points of A, B, C, and D are not included in the quadrilateral A'B'C'D ', W (k, m) = The processing ends with 0.
(B) In the above step (b), when all points of A ′, B ′, C ′, and D ′ are not included in the quadrilateral ABCD, W (k, m) = 0, and finish.
(C) When the above steps (c-1) to (c-4) are completed, the number of picked-up points is one to two or three to eight.
[0065]
Therefore,
(C1) If the number of points to be picked up is 0 or 2, W (k, m) = 0 and the processing ends.
(C2) If the number of picked-up points is three to eight, processing is performed up to step (f) to obtain W (k, m).
[0066]
In FIG. 6, the relationship between the pixel for the overhead image and the pixel for the oblique image is shown, which shows the relationship between the quadrilateral (shape of the pixel for the overhead image) and the square (shape of the pixel for the oblique image). Things. In such a relationship, when the relationship between the quadrilateral and the square becomes the relationship shown in FIG. 13, the number of points picked up is eight, which is the maximum number. FIG. 13 shows the relationship between pixels for directly above images (quadrilaterals shown by thick lines) and pixels for oblique images (squares shown by thin lines). From this, the maximum number of points to be picked up is eight.
[0067]
The procedure for obtaining W (k, m) described above is merely an example, and there may be other suitable procedures other than the method described here. However, no matter what procedure is used to determine W (k, m), it is clear that the present invention can be applied to the image conversion method or image conversion apparatus according to the present invention.
[0068]
The procedure for obtaining W (k, m) described above is summarized as follows with reference to the flowchart shown in FIG.
[0069]
The positions where the four vertices of the pixel [I, J] forming the second light receiving plane are projected on the light receiving plane of the first imaging device are A, B, C, and D, and the pixel [ i, j] as A ', B', C ', D'
(D) It is checked whether the points A, B, C, and D are included in the quadrilateral A'B'C'D '(step S1). If neither is included (N), W (k, m) is set to W (k, m) = 0 (step S2), and this procedure ends. On the other hand, when any point is included (Y), the included point is picked up (step S3).
(E) It is checked whether or not each point of A ', B', C ', D' is included in the quadrilateral ABCD (step S4). If neither is included (N), W (k, m) is set to W (k, m) = 0 (step S5), and this procedure is terminated. On the other hand, if any point is included (Y), the included point is picked up (step S6).
(F) Next, the intersection of line segment A'B 'with line segment AB, line segment BC, line segment CD, line segment DA,
Intersection of line segment B'C 'with line segment AB, line segment BC, line segment CD, line segment DA,
Intersection of line segment C'D 'with line segment AB, line segment BC, line segment CD, line segment DA, and
The intersection of the line segment D'A 'with the line segment AB, the line segment BC, the line segment CD, and the line segment DA are examined and picked up (step S7).
(G) W (k, m) is obtained from the coordinates of all the points picked up in the steps S3, S6 and S7 (step S8).
[0070]
<Description of image conversion device>
Next, an apparatus for realizing the above-described image conversion method will be described with reference to FIGS. FIG. 15 is a diagram for explaining the configuration of the image conversion device 32 of the present invention. The image conversion device 32 includes the imaging device 10 described with reference to FIG.
The image conversion device 32 includes a first image data acquisition unit 36, an image conversion processing unit 38, a storage device 40, and a second image data output unit 42. The image conversion processing unit 38 includes a CPU as described later. The first image data acquisition unit 36 receives an image captured by the first imaging device 10, and sends the image to the image conversion processing unit 38 as first image data. The storage device 40 stores coordinate transformation parameters and all parameters related to the equation (1) giving resampling. Further, in the image conversion step, all parameters related to the equation (1) giving resampling are called, transferred to the image conversion processing unit 38, and used for conversion into the second image data. Here, all parameters related to equation (1) that give resampling mean the range of k and m for pixel [I, J] and W (k, m) for each pixel [k, m]. However, the pixel [I, J] is the entire range of the second image data. The second image data converted by the image conversion processing unit 38 is output to the outside via the second image data output unit as needed.
The processing related to the writing and calling of the information into and from the storage device 40 is a well-known matter in a control device using a computer, and thus detailed description is omitted.
[0071]
First, the test pattern 14 is placed on the transport arm 26, and an operation for calibrating the position of the imaging device is performed on the test pattern 14 as described above. First, the test pattern 14 is imaged by the first imaging device 10 at the position A. This step corresponds to step ST1 shown in the flowchart of FIG. In order not to deteriorate the image quality of the directly converted image data as compared with the image quality of the oblique image data as much as possible, a grid point located at the center on the test pattern is located at the center point of the light receiving plane 10b of the first imaging device 10. The installation position of the test pattern is adjusted so that an image is taken. This step corresponds to step ST1-1 shown in the flowchart of FIG. Next, the scale of the second imaging device is set such that the dimensions of the images at the central portion of the light receiving plane 10b of the first imaging device 10 and the central portion of the light receiving surface 12b of the second imaging device 12 are equal. Since the second imaging device is a virtual model, this setting can be easily performed. This step is Step ST3-1 shown in the flowchart of FIG.
[0072]
The (x, y) coordinate system is provided on the light receiving plane of the first imaging device 10, and the first image data of the test pattern is taken into the first image data acquisition unit 36 provided in the image conversion device 32. The first image data captured by the first image data acquisition unit 36 provided in the image conversion device 32 is sent to the image conversion processing unit 38 and used when calculating coordinate conversion parameters. On the other hand, the second imaging device 12 is a virtual device that does not actually exist. Therefore, the coordinate system (X, Y) provided on the light receiving plane 12b of the second imaging device 12 exists only on the internal memory 54 of the image conversion processing unit 38 (see FIG. 16 described later). After the center position and the scale of the second imaging device 12 are set, the coordinate values of the selected grid point on the test pattern are input to the image conversion device 32. When the coordinate values of the selected grid point are input to the image conversion processing unit 38, for each grid point on the selected test pattern, the coordinate value on the light receiving plane of the second imaging device 12 is Are created on the internal memory 54 of the image conversion processing unit 38 (see FIG. 16 described later). In this way, data for obtaining the coordinate conversion parameters is prepared. When these data are prepared, the data image conversion processing unit 38 performs an operation to calculate the coordinate conversion parameters bi (where i = 1, 2, 3, 4, 5, 6, 7, 8) and re-calculates. Relational expressions used in sampling
Img2 (I, J) = Σ [W (k, m) × Img1 (k, m)] / ΣW (k, m)
Is determined. The above-described coordinate conversion parameters and the resampling relational expression are sent to the storage device 40 and stored.
[0073]
As described above, the installation of the imaging device 10 for actually imaging the imaging target object and the setting of the parameters necessary for the image conversion in the storage device of the image conversion device are completed. If the imaging object is imaged by the imaging device 10, the imaging result is sent to the first image acquisition unit 36 and further sent to the image conversion processing unit 38. In the image conversion processing unit 38, the above-described coordinate conversion parameters and the resampling relational expression are extracted from the storage device 40, and the conversion to the directly-facing image data is performed using these parameters. It is created on the internal memory of the conversion processing unit 38, and further image processing can be continuously performed. Alternatively, in some cases, the directly-facing image data obtained as a result of the conversion is output to the outside from the second image data output unit 42, and further image processing or the like is performed outside.
[0074]
With reference to FIG. 16, the configuration of the image conversion processing unit 38 provided in the image conversion device 32 will be described. The image conversion processing unit 38 includes a control unit 46, a coordinate conversion unit 48, a resampling unit 50, and an internal memory 54. The internal memory 54 temporarily stores W (k, m) and directly-facing image data obtained by image conversion in order to perform image conversion processing at high speed. Further, the resampling unit 50 includes a W (k, m) calculating unit 51 and an Img2 (I, J) calculating unit 52.
[0075]
The control unit 46 controls the coordinate conversion unit 48 and the resampling unit 50, and controls data exchange with the storage device 40 and the like, operation timing, and the like. Further, the control unit 46 can control the operations of the first image data acquisition unit 36 and the second image data output unit 42 described with reference to FIG. The part of the control unit 46 that controls the first image data acquisition unit 36 and the second image data output unit 42 may be configured as a part of another CPU outside the image conversion processing unit 38. How to configure the control unit 46 is only a matter of design.
[0076]
The coordinate conversion unit 48 reads the standard coordinates of the first light receiving plane, which are imaged on the light receiving plane when a plurality of characteristic points in the test pattern are imaged in oblique directions. This step is step ST2 of the flowchart shown in FIG. In addition, the center position and the scale of the second light receiving plane 12b are set. This step is step ST3 in the flowchart shown in FIG. Further, the coordinate values of the grid points on the plurality of selected characteristic points in the test pattern imaged in the oblique direction by the first imaging device on the second light receiving plane 12b are calculated. This step is step ST4 in the flowchart shown in FIG. After this step, the coordinate conversion parameter b is calculated based on the coordinate values of the grid points on the corresponding characteristic points on the test pattern. _i (However, i = 1, 2, 3, 4, 5, 6, 7, 8). This step is step ST5 in the flowchart shown in FIG.
[0077]
The resampling unit 50 performs a process of converting the first image data, which is oblique image data obtained by imaging the imaging target, into the second image data, which is directly-facing image data. The W (k, m) calculation unit 51 performs a process corresponding to the steps S1 to S8 shown in FIG. 14 and performs the processing on the I-th row and the J-th column of the second image data projected on the first light receiving plane. Of the pixel [I, J] is calculated as W (k, m) representing the area projected onto the pixel [k, m] on the first light receiving plane, and the result is stored in the storage device 40. The Img2 (I, J) calculation unit 52 calculates the value of the light intensity Img1 (k, m) of the pixel [k, m] at the k-th row and the m-th column of the first image data sent from the first image acquisition unit Is the following equation including W (k, m) stored in the storage device 40.
Img2 (I, J) = Σ [W (k, m) × Img1 (k, m)] / ΣW (k, m)
, The light intensity Img2 (I, J) of the pixel [I, J] of the second image data is calculated.
[0078]
According to the apparatus described above, if the element Img1 (k, m) of the oblique image data obtained by imaging the imaging target with the first imaging device is given over the entire oblique light receiving plane, this is taken as the imaging target. Is calculated over the entire directly-facing light-receiving plane, and the desired image conversion is performed.
[0079]
<Example of conversion processing by image conversion method>
With reference to FIG. 17, an example of an image obtained by performing image conversion from diagonal image data obtained by imaging a lattice as an imaging target in an oblique direction to image data directly above will be described. FIG. 17A shows an image taken obliquely from above. In this image, it can be seen that the grid spacing (the spacing between the intersections of the black lines) becomes narrower toward the right. Also, it can be seen that the thickness of the black line forming the lattice image becomes thinner toward the right side.
[0080]
The image shown in FIG. 17B is obtained by converting the lattice image shown in FIG. 17A into an image that would be obtained if the image was taken from directly above. In this image conversion, in order to determine the magnification of the image, the grid interval p on the representative black line at the center of the image forming the grid image indicated by the arrow 16 in FIG. The condition that it is equal to the lattice spacing p 'on the line was imposed. In the converted image shown in FIG. 17B, grid points (intersection points of black lines) are arranged in a square shape, and the black lines have a uniform thickness. In other words, it can be seen that it is equivalent to a grid image taken from directly above.
[0081]
Note that black triangles indicated by arrows 17 and 18 in FIG. 17B are portions not shown in the image of FIG. 17A. That is, this black triangle portion is a portion in which the corresponding image data does not exist in the oblique image data forming the image shown in FIG.
[0082]
<Example of a semiconductor manufacturing apparatus equipped with an apparatus for realizing an image conversion method>
Referring to FIG. 18, an example of a semiconductor manufacturing apparatus provided with a mechanism for performing the image conversion method according to the present invention will be described. This semiconductor manufacturing apparatus has a function of taking out wafers 24 one by one by a transfer arm 26 from a wafer cassette 22 in which a plurality of wafers (single-crystal thin plates) are stacked horizontally and stocked, and transporting the wafers 24 to a stage 28. Things. The wafer 24 transferred to the stage 28 must be set at a predetermined position and direction according to the orientation flat attached to the wafer 24.
[0083]
The orientation flat is a portion obtained by linearly cutting a part of the periphery of the wafer 24 so that the crystal axis direction of the single crystal constituting the wafer 24 can be visually or mechanically read. For the same purpose of making the crystal axis direction legible, besides using an orientation flat, there is also a case where a notch is cut into a part of the periphery of the wafer.
[0084]
When the wafer 24 is set in the wafer cassette 22, the orientation flat is not aligned in a certain direction. Therefore, before being taken out of the wafer cassette 22 by the transfer arm 26 and set on the stage 28, the orientation flat of the wafer 24 is detected by some method, and the wafer 24 is transferred so that the orientation flat is oriented in a predetermined direction. Must. Therefore, the wafer 24 taken out of the wafer cassette 22 by the orientation flat detection camera 20 is set as an imaging target. Here, the wafer 24 on the transfer arm 26 in the state of being transferred by the imaging device 20 is imaged by the orientation flat detection camera 20, and the orientation flat is detected based on this image to determine the required rotation amount of the wafer. This information is provided to the transfer arm 26, and the transfer arm 26 rotates the wafer 24 by a required amount based on the information and installs the wafer 24 on the stage 28.
[0085]
2. Description of the Related Art An imaging device provided in a semiconductor manufacturing apparatus is mounted to capture an image of an object such as a wafer and acquire images used for various purposes such as detection of an orientation flat and a notch using a technique such as image search. Therefore, it is preferable to install the imaging device so that the wafer can be imaged in a direction perpendicular to the upper surface of the wafer because there is no distortion in the image to be imaged. This is because programs and the like for realizing functions such as image search are generally made on the assumption that there is no external image distortion or the like such as an image captured in an oblique direction in a captured image. It is.
[0086]
However, since the semiconductor manufacturing apparatus is installed in a clean room, it is required to make the size as small as possible. In order to reduce the size of the entire apparatus such as a semiconductor manufacturing apparatus, it may not be possible to mount the imaging apparatus so that the wafer can be imaged in a direction perpendicular to the wafer surface. Therefore, even if an image is captured by an imaging device installed in any place, the image is converted to an image that should be obtained by setting the imaging device at the originally desired position and capturing the image. If possible, this problem will be solved.
[0087]
An image conversion method according to the present invention is an image conversion method suitable for achieving the above-described object. With reference to FIGS. 14, 15 and 17, the configuration and handling of a semiconductor manufacturing apparatus provided with a mechanism for performing the image conversion method according to the present invention will be described. In the following description, the step names in parentheses correspond to the respective step names shown in the flowcharts of FIGS.
[0088]
The test pattern on the transfer arm 26 is imaged from an oblique direction by the imaging device 20 installed at an oblique direction position (step ST1). A plurality of characteristic points set on the upper surface of the test pattern are read from the oblique direction, and the coordinate values of the first light receiving plane imaged on the light receiving plane are read (step ST2). This is performed by the first image acquisition unit 36 provided in the image conversion device 32 shown in FIG. Then, these coordinate values (x, y) are sent to the image conversion processing unit 38. The virtual center position and scale of the second imaging device 12 are determined (step ST3). In addition, coordinate values (X, Y) of a plurality of characteristic points set on the upper surface of the test pattern imaged in the oblique direction by the first imaging device on the second light receiving plane 12b are calculated (step ST4). After this step, coordinate conversion is performed based on first plane coordinate values (x, y) and second plane coordinate values (X, Y) corresponding to a plurality of characteristic points set on the upper surface of the test pattern. Parameter b _i (Where i = 1, 2, 3, 4, 5, 6, 7, 8) is calculated by the method of solving simultaneous equations or the method of least squares (step ST5). The calculation of this parameter is performed in the coordinate conversion unit 48 in the image conversion processing unit 38 shown in FIGS.
[0089]
Subsequently, the light intensity Img2 (I, J) of the pixel [I, J] of the second image data forming the light receiving plane of the second imaging device 12 and the light receiving plane of the first imaging device 10 are configured. The following equation (1) that gives a relationship with the light intensity Img1 (k, m) of the pixel [k, m] in the k-th row and the m-th column of the first image data is obtained (Steps S1 to S8).
[0090]
Img2 (I, J) = Σ [W (k, m) × Img1 (k, m)] / ΣW (k, m) (1)
This processing is also performed in the W (k, m) calculation unit 51 and the Img2 (I, J) calculation unit 52 in the image conversion processing unit 38 shown in FIG.
[0091]
Parameter b for coordinate transformation described above _i (Where i = 1, 2, 3, 4, 5, 6, 7, 8) and the above equation (1) which gives the relationship between Img2 (I, J) and Img1 (k, m) All relevant parameters are stored in the storage device 40. Here, all the parameters related to the above equation (1) that give the relationship between Img2 (I, J) and Img1 (k, m) include the ranges of k and m for the pixel [I, J], It means W (k, m) for the pixel [k, m], and the pixel [I, J] is the entire range of the second image data.
[0092]
As described above, if the first imaging device 10 is set to the semiconductor manufacturing device, every time the first imaging device 10 is taken out of the wafer cassette 22, the wafer 24 on the transfer arm 26 is imaged by the first imaging device 10. One image data is converted to second image data and output to the outside. That is, if the oblique image data is acquired by the first image acquisition unit 36 and sent to the image conversion processing unit 38, all the parameters related to the conditional expression (1) for resampling are called from the storage device 40, and the image The first image data is transferred to the internal memory 45 of the conversion processing unit 38, and is converted into directly-facing image data that should be obtained by the second imaging device. Then, the result is output from the second image data output unit 42 to the outside. That is, in this semiconductor manufacturing apparatus, in normal use, it is not necessary to install the second imaging device, which is the position directly facing the wafer, and the first imaging device 20 is replaced with the second imaging device. In order to perform subsequent image processing at high speed, the second image data is held in the internal memory 54 of the image conversion processing unit 38, and the next image processing is performed.
[0093]
If the second image data is output from the second image data output unit 42 as described above, the orientation flat can be detected using a known image search program or the like, and from the result, the known orientation flat is oriented in a predetermined direction. A wafer can be set on the stage. Obviously, according to the method of the present invention, the wafer can be set on the stage in a predetermined direction as described above, not limited to the orientation flat and notch.
[0094]
Also, in a large-scale manufacturing apparatus that needs to monitor an imaging target such as a wafer at a plurality of locations, images captured from a plurality of locations are all captured by the imaging device not only at the distance to the wafer but also in the imaging direction. Are preferably imaged under the same condition. As a large-scale manufacturing apparatus that needs to monitor an imaging target such as a wafer at a plurality of places described above, a manufacturing line or a manufacturing plant may be used. Also in this case, similarly to the example of the semiconductor manufacturing apparatus described above, there are many cases where the imaging device cannot always be set at a position that satisfies the same condition with respect to the imaging target.
[0095]
According to the image conversion method and the image conversion apparatus of the present invention, the following advantages can be obtained even in a large-scale manufacturing apparatus that needs to monitor an object at a plurality of locations. If image data obtained by a plurality of imaging devices installed in a large-scale manufacturing device is subjected to an image conversion process by the image conversion method of the present invention, an image obtained by capturing all the objects at the same distance from the vertical direction. Each can be converted to data.
[0096]
In this manner, if the image data is converted into image data that are all taken at the same distance from the object and from the vertical direction, based on the converted image, position detection and shape defects of the imaging object at each imaging position are performed. The image processing program used for the determination of the above can be used efficiently. That is, in order to process an image including image distortion or the like based on a difference in imaging conditions or the like with the above-described image processing program, the distortion of the image processing program is adjusted in correspondence with each imaging device. Etc. need to be changed. However, if the image conversion method of the present invention is implemented, the step of changing such parameters can be omitted.
[0097]
【The invention's effect】
As described above, according to the image conversion method and the image conversion device of the present invention, oblique image data obtained by imaging an imaging target from an oblique direction is obtained by imaging the imaging target from a directly facing direction. Is converted to the directly facing image data. Since the directly-facing image data has a similar shape to the imaging target, various types of operations such as identification of the imaging target, detection of a difference between the standard of the imaging target, measurement of the position of the imaging target, measurement of the shape of the imaging target, and the like are performed. An existing program library or the like that realizes the image processing function can be used effectively. In addition, since the screen search can be performed effectively, the image conversion method of the present invention is an effective technique in detecting an orientation flat of a semiconductor manufacturing apparatus and monitoring and inspecting a manufacturing line including a plurality of manufacturing processes.
[Brief description of the drawings]
FIG. 1 is a diagram provided for describing a state in which a lattice pattern is imaged from directly above and obliquely above.
FIGS. 2A, 2B, and 2C are diagrams showing images of a grid pattern taken from directly above and obliquely upward.
FIG. 3 is a flowchart for obtaining parameters for coordinate conversion.
FIG. 4 is a diagram provided for describing coordinates of a light receiving plane and an index for designating a pixel;
FIG. 5 is a diagram provided for explanation of a method of acquiring image data of pixels forming a light receiving plane.
FIG. 6 is a diagram provided for describing a relationship between a pixel for a directly above image and a pixel for an oblique image.
FIG. 7 is a diagram provided for description of W (k, m).
FIG. 8 is a diagram for explaining how to obtain W (k, m).
FIG. 9 is a diagram for explaining points picked up for obtaining W (k, m).
FIGS. 10A and 10B are diagrams for explaining the relationship between a point O and a quadrilateral EFGH, respectively.
FIGS. 11A, 11B, and 11C are diagrams for explaining the determination of the presence or absence of an intersection between line segments;
FIG. 12 is a diagram for explaining a case where a line segment overlaps a line segment;
FIG. 13 is a diagram for explaining the relationship between pixels for directly above images and pixels for oblique images.
FIG. 14 is a flowchart for obtaining W (k, m).
FIG. 15 is a diagram provided for describing a configuration of an image conversion apparatus.
FIG. 16 is a diagram provided for describing a configuration of an image conversion processing unit.
FIGS. 17A and 17B are diagrams illustrating conversion examples of a lattice image, respectively.
FIG. 18 is a schematic view of a semiconductor manufacturing apparatus provided with a mechanism for detecting an orientation flat.
[Explanation of symbols]
10: First imaging device
12: Second imaging device
14: Lattice pattern
20: Camera for alignment
22: Wafer cassette
24: Wafer
26: Transfer arm
28: Stage
32: Image conversion device
50: Resampling unit

Claims (12)

First image data obtained on an oblique light receiving plane (hereinafter, referred to as “first light receiving plane”) by obliquely imaging an object to be imaged by an imaging apparatus including an imaging optical system and a light receiving plane. An image conversion method for converting the image-capturing object into second image data obtained on a directly-facing light-receiving plane (hereinafter, referred to as a “second light-receiving plane”) when the imaging object is temporarily imaged from a directly facing direction,
A coordinate conversion step of determining a relationship between first plane coordinates for determining the pixel position on the oblique light receiving plane and second plane coordinates for determining the pixel position on the directly-facing light receiving plane;
A resampling step of obtaining the light intensity Img2 (I, J) of the pixel [I, J] of the second image data according to the following equation (1).
Img2 (I, J) = Σ [W (k, m) × Img1 (k, m)] / ΣW (k, m) (1)
Where I, J, k and m are integers,
Img1 (k, m) is the light intensity of the pixel [k, m] at the k-th row and the m-th column of the first image data.
W (k, m) is the pixel [I, J] in the I-th row and the J-th column of the second image data projected on the first light receiving plane, and is located on the pixel [k, m] on the first light receiving surface. Represents the area projected to
In addition, the range for calculating the sum of the right side of Expression (1) is such that the pixel [I, J] of the second image data projected on the first light receiving plane is located above the pixel [k, m] on the first light receiving surface. Is projected over all of the projected pixels [k, m]. The image conversion method according to claim 1, wherein the coordinate conversion step includes:
(A) Four or more points selected so that no three or more points are aligned on a straight line, or a plurality of points selected including three points on the straight line and including at least two points outside the straight line (Hereinafter, these points are referred to as “a plurality of characteristic points.”) The imaging object having a mark on the mark, and the coordinate value of the mark in a coordinate system provided on a plane on which the mark exists is known. An imaging step of imaging the test pattern with the imaging device using the imaging target as a test pattern;
(B) a coordinate reading step of reading a coordinate value of the first plane coordinates for a mark on a plurality of characteristic points in the test pattern, which is captured on the first light receiving plane when the image is captured in an oblique direction;
(C) An arbitrary position in the test pattern is defined as an origin, and based on an image of the test pattern imaged obliquely on the first light receiving plane, the test pattern is defined by the second light receiving position. In the case where the image is taken from the direction opposite to the center projected on the center of the plane, the position of the center of the second light receiving plane on which the origin is projected and the image of the test pattern taken obliquely from the geometrical direction. A calibration process that virtually sets the scale to be converted to similar to
(D) For a mark on a plurality of characteristic points imaged in oblique directions, a second light reception when each of the marks in the test pattern is imaged from a virtually set opposite direction. Projecting on a plane to calculate the position of the mark at the second plane coordinates, a position calculating step,
(E) the first coordinate of each of the marks on the characteristic points in the test pattern projected on the first light receiving plane when the image is taken from an oblique direction, and a virtually opposite direction that is directly set; On the second light receiving plane when the image is taken from, based on the calculated second coordinates of the marks on the plurality of characteristic points in the calculated test pattern, a coordinate conversion parameter b _i (where, i = 1, 2, 3, 4, 5, 6, 7, 8).
And an image conversion method. In the image conversion method according to claim 2, a calibration step of virtually setting a center position and a scale in a second plane coordinate when imaging from the directly facing direction,
(C ′) The scale is determined so that the size of the image at the position of the center of the first light receiving plane and the position of the center of the second light receiving plane virtually set are equal, and An image conversion method, comprising: a calibration step of virtually setting a position of a center portion when an image of the imaging target is taken from a direction facing the center so that the position of the center portion matches. 4. The image conversion method according to claim 2, wherein the coordinate reading step, the step of calculating the mark position in the second plane coordinates, and the parameter calculating step are: (b ′) a plurality of characteristic steps in the test pattern; A coordinate reading step of selecting points and reading first plane coordinate values (X, Y) for each selected point;
(D ′) a position calculating step of calculating a mark position (x, y) in the second plane coordinates for each of the selected points selected in (b ′);
(E ′) Eight coordinate conversion parameters defined by the following equations (2) and (3) from the first plane coordinate values (X, Y) and the second plane coordinate values (x, y) b _i (where i = 1,2,3,4,5,6,7,8) is obtained by a method of solving simultaneous equations or a method of least squares, including a parameter calculating step. Image conversion method.
_{X = (b 1 x + b} 2 y + b 3) / (b 7 x + b 8 y + 1) (2)
_{Y = (b 4 x + b} 5 y + b 6) / (b 7 x + b 8 y + 1) (3) 2. The image conversion method according to claim 1, wherein a pixel [I, J] of the second image data projected on the first light receiving plane is projected on a pixel [k, m] on the first light receiving surface. An image conversion method, wherein the step of obtaining W (k, m) representing the area of the image includes the following steps.
The positions where the four vertices of the pixel [I, J] forming the second light receiving plane are projected on the light receiving plane of the first imaging device are A, B, C, and D, and the pixel [ i, j] as A ', B', C ', D'
(D) a step of checking whether or not each of the points A, B, C and D is included in the quadrilateral A'B'C'D ',
(E) picking up whether or not each point of A ', B', C ', D' is included in the quadrilateral ABCD;
(F-1) the step of picking up the intersection of line segment A'B 'with line segment AB, line segment BC, line segment CD, line segment DA,
(F-2) a step of picking up the intersection of line segment B'C 'with line segment AB, line segment BC, line segment CD, line segment DA,
(F-3) a step of examining the intersection of the line segment C'D 'with the line segment AB, the line segment BC, the line segment CD, and the line segment DA, and picking up;
(F-4) a step of examining the intersection of the line segment D'A 'with the line segment AB, the line segment BC, the line segment CD, and the line segment DA, and picking up;
(G) a step of obtaining W (k, m) from the coordinates of all points picked up in the steps (d) to (f-4). First image data obtained on an oblique light receiving plane (hereinafter, referred to as “first light receiving plane”) by obliquely imaging an object to be imaged by an imaging apparatus including an imaging optical system and a light receiving plane. An image conversion device for converting the image-capturing object into second image data obtained on a directly-facing light-receiving plane (hereinafter, referred to as a “second light-receiving plane”) when the image-capturing object is temporarily imaged from a directly facing direction,
Coordinate conversion means for determining a relationship between first plane coordinates for determining the pixel position on the oblique light receiving plane and second plane coordinates for determining the pixel position on the directly-facing light receiving plane;
An image conversion apparatus comprising: resampling means for obtaining light intensity Img2 (I, J) of a pixel [I, J] of second image data according to the following equation (1).
Img2 (I, J) = Σ [W (k, m) × Img1 (k, m)] / ΣW (k, m) (1)
Where I, J, k and m are integers,
Img1 (k, m) is the light intensity of the pixel [k, m] at the k-th row and the m-th column of the first image data.
W (k, m) is the pixel [I, J] in the I-th row and the J-th column of the second image data projected on the first light receiving plane, and is located on the pixel [k, m] on the first light receiving surface. Represents the area projected to
In addition, the range for calculating the sum of the right side of Expression (1) is such that the pixel [I, J] of the second image data projected on the first light receiving plane is located above the pixel [k, m] on the first light receiving surface. Is projected over all of the projected pixels [k, m]. The image conversion device according to claim 6, wherein the coordinate conversion unit includes:
Four or more points selected so that no three or more points are aligned on a straight line, or a plurality of points selected so as to include at least two points outside the straight line including three points on the straight line (hereinafter referred to as Is referred to as “a plurality of characteristic points”.) The image pickup object having a mark on the mark, and the coordinate value of the mark is known by a coordinate system provided on a plane on which the mark exists. Imaging means for imaging the test pattern by the imaging device, using an object as a test pattern,
Imaged on the first light receiving plane when imaged in an oblique direction, for a mark on a plurality of characteristic points in the test pattern, read the coordinate value of the first plane coordinates, coordinate reading means,
An arbitrary position in the test pattern is defined as the origin, and based on the image of the test pattern imaged obliquely on the first light receiving plane, the test pattern is located at the center of the second light receiving plane. In the case where the image is taken from the opposite direction projected to the portion, the position of the center portion of the second light receiving plane where the origin is projected and the image of the test pattern taken obliquely are geometrically similarized. Calibration means for virtually setting the scale to be performed,
For a mark on a plurality of characteristic points imaged in oblique directions, each mark in the test pattern is placed on the second light receiving plane when imaged from a virtually opposite direction. Projecting, calculating the position of the mark in the second plane coordinates, position calculating means,
The first coordinates of the marks on the plurality of characteristic points in the test pattern projected on the first light receiving plane when imaged from an oblique direction, and the image is imaged from a direction directly opposite to a virtually set position. On the second light receiving plane in the case of the above, based on the calculated second coordinates of the marks on the plurality of characteristic points in the calculated test pattern, coordinate conversion parameters b _i (where i = 1 , 2, 3, 4, 5, 6, 7, 8), and parameter calculation means. In the image conversion device according to claim 7, a calibration unit that virtually sets a center position and a scale in a second plane coordinate when capturing an image from the opposite direction,
At the position of the center portion of the first light receiving plane and the position of the center portion of the second light receiving plane virtually set, the scale is determined so that the dimensions of the images are equal, and the positions of the center portions of the images of each other An image conversion apparatus comprising: a calibration unit that virtually sets a position of a center portion when an image of the imaging target is imaged from a directly facing direction so as to match. The image conversion apparatus according to claim 7, wherein the coordinate reading unit, the unit that calculates a mark position in the second plane coordinates, and the parameter calculation unit are:
Coordinate reading means for selecting a plurality of characteristic points in the test pattern and reading first plane coordinate values (X, Y) for each selected point;
Position calculating means for calculating a mark position (x, y) in a second plane coordinate for each of the selected points selected by the coordinate reading means;
First plane coordinate values (X, Y) and the second plane coordinate values (x, y) from the following equation (2) and (3) 8 coordinate transformation parameters b _i (but that are defined in , I = 1, 2, 3, 4, 5, 6, 7, 8) by a method of solving simultaneous equations or a method of least squares. apparatus.
_{X = (b 1 x + b} 2 y + b 3) / (b 7 x + b 8 y + 1) (2)
_{Y = (b 4 x + b} 5 y + b 6) / (b 7 x + b 8 y + 1) (3) 7. The image conversion device according to claim 6, wherein a pixel [I, J] of the second image data projected on the first light receiving plane is projected on a pixel [k, m] on the first light receiving surface. An image conversion apparatus characterized in that the means for obtaining W (k, m) representing the area of the image includes the following means.
The positions where the four vertices of the pixel [I, J] forming the second light receiving plane are projected on the light receiving plane of the first imaging device are A, B, C, and D, and the pixel [ i, j] as A ', B', C ', D'
Means 1 for checking whether or not each of the points A, B, C, D is included in the quadrilateral A'B'C'D ';
Means 2 for checking whether or not each point of A ', B', C ', D' is included in the quadrilateral ABCD;
Means 3 for examining and picking up the intersection of line segment A'B 'with line segment AB, line segment BC, line segment CD, line segment DA;
Means 4 for examining and picking up the intersection of line segment B'C 'with line segment AB, line segment BC, line segment CD, line segment DA;
Means 5 for examining and picking up the intersection of line segment C'D 'with line segment AB, line segment BC, line segment CD, line segment DA;
Means 6 for examining and picking up the intersection of line segment D'A 'with line segment AB, line segment BC, line segment CD, line segment DA;
Means 7 for obtaining W (k, m) from the coordinates of all points picked up by the means 1 to 6; 11. A semiconductor manufacturing apparatus comprising: a wafer cassette for stocking a plurality of wafers; and transport means for removing wafers one by one from the wafer cassette and transporting the wafers to a stage. A semiconductor manufacturing apparatus comprising an image conversion device, wherein the object to be imaged is the wafer taken out of the wafer cassette. A transport line including a transport unit that connects a plurality of processes, and one to a plurality of inspection units based on images, comprising an image conversion device according to any one of claims 6 to 10, wherein A transport line characterized by being an object transported on a transport line.

Images