KR102129069B1 - Method and apparatus of automatic optical inspection using scanning holography - Google Patents

Abstract

The present invention relates to a scanning hologram-based automatic optical inspection apparatus and a method thereof. The automatic optical inspection apparatus includes: a hologram photographing unit photographing a hologram of an object present on an object plate by using a scanning hologram camera; a depth position and rotation angle extraction unit extracting a depth position and a rotation angle with respect to the object surface of the object plate based on the hologram; a rotated coordinate system forming unit forming a rotated coordinate system corresponding to the object surface using the depth position and the rotation angle; and a hologram restoration unit obtaining an image of the object by restoring the hologram in a plane formed in the depth direction of the rotated coordinate system. According to one embodiment of the present invention, the automatic optical inspection apparatus can acquire image information for high-precision automatic optical inspection without precise mechanical adjustment even in a tilting position of an object plate.

Description

A scanning hologram-based automatic optical inspection apparatus and method{METHOD AND APPARATUS OF AUTOMATIC OPTICAL INSPECTION USING SCANNING HOLOGRAPHY}

The present invention relates to an automatic optical inspection equipment and method, and more specifically, by taking a hologram of a photographed object and processing it in a numerical manner, it is possible to obtain image information capable of high-precision automatic optical inspection regardless of rotation and defocus. It relates to a scanning hologram camera device and method.

In the related art, an automatic optical inspection apparatus and method according to an optical microscope is a method of extracting the focused image information of an object located on a target plate in the form of a digital signal using an optical microscope and numerically recognizing it using a computer to process defects of the object Ina was examined for distortion. However, in order to acquire an image of an object at a high resolution using the above-described automatic optical inspection equipment and method, the depth of focus of the objective lens is reduced to a few micrometer level, so that the objective plate is precisely aligned, and that the objective lens is precisely mechanically adjusted. Is required.

In addition, in order to obtain a 3D image of the object, the object plate is sequentially positioned in the depth direction to obtain a focused image at a sequential depth position, and a 3D image of the object is obtained by combining them, or the objective lens is directed to the depth direction. Three-dimensional images of the object were obtained by sequentially relocating the images focused on the sequential depth direction by sequentially positioning them.

Therefore, the automatic optical inspection equipment based on the existing imaging optical system according to the mechanical alignment and focus adjustment requires an additional device for precise mechanical control, and it is difficult to perform ultra-high-speed inspection according to mechanical movement. For example, as an example of inspecting an object placed on the conveyor belt, the object on the conveyor belt is positioned obliquely by rotation at a defocused depth position beyond the depth region of the objective lens according to the rotation of the conveyor belt.

At this time, in order to rearrange the object positioned obliquely in the defocusing area to the depth region of the objective lens, precise detection of the defocused depth position and inclined rotation angle, as well as sophisticated mechanical control using the detected depth position and inclined rotation angle The subject needs to be rearranged. In practice, it is structurally difficult to add a mechanical device for moving or rotating an object located on the conveyor belt in a depth direction on the conveyor belt, so that the object is placed on a precisely aligned target plate using a robot arm or the like on the conveyor belt. After positioning, focusing is obtained through the mechanical movement of the objective lens to obtain the focused image. Therefore, by taking a hologram of the object without elaborate mechanical adjustment and processing it in a numerical way, it acquires a clear image in focus regardless of rotation and out-of-focus even at the rotating and defocused position of the objective board, thereby automatically automating optically without any mechanical reordering. Invented hologram camera and numerical processing method capable of obtaining inspectable image information.

Korean Registered Patent No. 10-1304695 (2013.08.30)

One embodiment of the present invention is a scanning hologram-based automatic optical inspection that can acquire a depth position and a rotation angle with respect to an objective plate by photographing a hologram of an object to be shot in a single shot using a scanning hologram camera and processing it in a numerical manner It is intended to provide an apparatus and method.

An embodiment of the present invention is a scanning hologram-based automatic optical inspection capable of obtaining image information capable of high-precision automatic optical inspection without precise mechanical adjustment by acquiring a clear image in focus regardless of rotation and out-of-focus even in an inclined target plate position It is intended to provide an apparatus and method.

Among the embodiments, the scanning hologram-based automatic optical inspection apparatus is a hologram photographing unit for photographing a hologram of an object existing on an objective plate using a scanning hologram camera, and a depth position with respect to an objective surface of the objective plate based on the hologram and Depth position and rotation angle extraction unit for extracting a rotation angle, a rotated coordinate system forming unit for forming a rotated coordinate system corresponding to the objective surface using the depth position and rotation angle, and the hologram in a depth direction of the rotated coordinate system It includes a hologram reconstruction unit for acquiring the image of the object by restoring in the plane formed by.

The scanning hologram camera includes a light source for generating electromagnetic waves, dividing means for dividing the electromagnetic waves, scanning means for scanning the object using an interference beam formed by the divided electromagnetic waves, and reflected, fluorescent, or transmitted beams from the object It includes a light detecting means for detecting, the hologram photographing unit may generate a complex number hologram as a result of the photographing.

The depth position and rotation angle extraction unit may extract the depth position and the rotation angle with respect to the object surface after extracting the depth position from each of the three regions defined and independently spaced from each other.

The depth position and rotation angle extraction unit includes a first step of restoring the hologram at a sequential depth position, a second step of calculating a focus metric in each of the three regions for reconstructed images, and the focus metric is the maximum. A third step of determining a depth position as a value as a depth position of each of the three areas may be performed.

The depth position and rotation angle extraction unit may input hologram data to a CNN model to obtain depth positions in each of the three areas as outputs, and then extract depth positions and rotation angles with respect to the objective surface.

The depth position and rotation angle extraction unit uses the complex hologram obtained through the photographing as the hologram data, a real hologram corresponding to the real part of the complex hologram, an imaginary hologram corresponding to the imaginary part of the complex hologram, and the complex hologram. Any one of the synthesized off-axis holograms can be input to the CNN model.

The depth position and rotation angle extraction unit uses the CNN model generated through learning to input a specific region forming the hologram as an input and output the rotation angle of the rotated object surface, and then the depth position and rotation angle of the object surface. Can be extracted.

After the depth position and rotation angle extraction unit pre-converts the hologram into a spatial frequency domain, an area corresponding to the specific domain in the spatial frequency domain may be used as the input.

The depth position and rotation angle extraction unit may use at least one of a real part, an imaginary part, an amplitude part, and a phase part of the complex hologram related to the specific region as the input.

The depth position and rotation angle extraction unit may extract a depth position and rotation angle with respect to the objective surface using a gradient search based on a partial or entire area of the hologram.

The depth position and rotation angle extraction unit searches for a rotation angle and a depth position where the focus metric of the guided hologram has the maximum value after guiding a part or the entire area of the hologram with respect to the rotation area and the positionable depth area where the object plate is rotatable. can do.

The rotated coordinate system forming unit may generate a transformation matrix that is generated using the depth position and the rotation angle extracted by the depth position and rotation angle extraction unit and converts a reference coordinate system into a rotated coordinate system.

The hologram restoration unit may restore the image of the object by converting the hologram into the rotated coordinate system and guiding in the depth direction of the rotated coordinate system.

The hologram restoration unit may obtain the image of the object by converting the hologram to each spectral rotational transformation, and guiding it in the depth direction of the rotated coordinate system.

The hologram restoration unit may obtain the 3D image of the object by converting the hologram into the rotated coordinate system and restoring each of them in a plane formed in a depth direction in the rotated coordinate system.

The hologram restoring unit may acquire the image of the object by restoring the hologram at a depth position of a reference coordinate system and then guiding it to a plane formed in a depth direction of the rotated coordinate system.

The hologram restoring unit may obtain the 3D image of the object by restoring the hologram at a depth position of the reference coordinate system and then guiding each of them in a sequential plane formed in the depth direction of the rotated coordinate system.

The hologram restoring unit may restore the hologram in a sequential plane formed in the depth direction of the reference coordinate system, and then interpolate the rotated coordinate system.

The hologram reconstruction unit generates a 3D matrix based on the depth position of each sequential plane with respect to the images reconstructed from the sequential plane, and interpolates the 3D matrix with the coordinate axes of the rotated coordinate system, thereby 3 A dimensional image can be obtained.

Among the embodiments, the scanning hologram-based automatic optical inspection method uses the scanning hologram camera to photograph a hologram of an object existing on the objective plate, based on the hologram, a depth position and a rotation angle with respect to the objective surface of the objective plate Extracting, forming a rotated coordinate system corresponding to the objective surface using the depth position and rotation angle, and restoring the hologram in a plane formed in a depth direction of the rotated coordinate system to obtain an image of the object. And obtaining.

The disclosed technology can have the following effects. However, since a specific embodiment does not mean that all of the following effects should be included or only the following effects are included, the scope of rights of the disclosed technology should not be understood as being limited thereby.

A scanning hologram-based automatic optical inspection apparatus and method according to an embodiment of the present invention uses a scanning hologram camera to take a hologram of an object to be shot in a single shot and processes it in a numerical method to depth position and rotation angle of the objective plate Can be obtained.

The scanning hologram-based automatic optical inspection apparatus and method according to an embodiment of the present invention acquires a clear image in focus regardless of rotation and out-of-focus even in an inclined objective plate position, thereby enabling high-precision automatic optical inspection without precise mechanical adjustment. Can be obtained.

1 is a view for explaining the configuration of a scanning hologram-based automatic optical inspection system according to the present invention.
FIG. 2 is a block diagram illustrating the functional configuration of the automatic optical inspection device in FIG. 1.
3 is a flowchart illustrating a scanning hologram-based automatic optical inspection process performed in the automatic optical inspection device of FIG. 1.
4 is a view for explaining an embodiment of the scanning hologram camera-based automatic optical inspection equipment.
5 is a view for explaining hologram imaging of a test object using an optical scanning hologram.
6 is a diagram illustrating an image reconstructed at a sequential depth location and a depth location measurement region.
7 is a view for explaining a hologram and a depth position measurement area.
8 is a diagram illustrating a reference coordinate system and a rotated coordinate system.
9 is a view for explaining a hologram and a partial region for measuring a depth position.

Since the description of the present invention is merely an embodiment for structural or functional description, the scope of the present invention should not be interpreted as being limited by the embodiments described in the text. That is, since the embodiments can be variously changed and have various forms, it should be understood that the scope of the present invention includes equivalents capable of realizing technical ideas. In addition, the purpose or effect presented in the present invention does not mean that a specific embodiment should include all or only such an effect, and the scope of the present invention should not be understood as being limited thereby.

Meanwhile, the meaning of terms described in the present application should be understood as follows.

Terms such as "first" and "second" are for distinguishing one component from other components, and the scope of rights should not be limited by these terms. For example, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.

When a component is said to be "connected" to another component, it may be understood that other components may exist in the middle, although it may be directly connected to the other component. On the other hand, when a component is said to be "directly connected" to another component, it should be understood that no other component exists in the middle. On the other hand, other expressions describing the relationship between the components, that is, "between" and "immediately between" or "adjacent to" and "directly neighboring to" should be interpreted similarly.

Singular expressions are to be understood as including plural expressions unless the context clearly indicates otherwise, and terms such as “comprises” or “have” are used features, numbers, steps, actions, components, parts or the like. It is to be understood that a combination is intended to be present, and should not be understood as pre-excluding the existence or addition possibility of one or more other features or numbers, steps, operations, components, parts or combinations thereof.

In each step, the identification code (for example, a, b, c, etc.) is used for convenience of explanation. The identification code does not describe the order of each step, and each step clearly identifies a specific order in context. Unless stated, it may occur in a different order than specified. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

The present invention can be embodied as computer readable code on a computer readable recording medium, and the computer readable recording medium includes all types of recording devices in which data readable by a computer system is stored. . Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, optical data storage devices, HDDs, and SSDs. In addition, the computer-readable recording medium can be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

All terms used herein have the same meaning as commonly understood by a person skilled in the art to which the present invention pertains, unless otherwise defined. The terms defined in the commonly used dictionary should be interpreted to be consistent with the meanings in the context of the related art, and cannot be interpreted as having ideal or excessively formal meanings unless explicitly defined in the present application.

1 is a view for explaining the configuration of a scanning hologram-based automatic optical inspection system according to the present invention.

Referring to FIG. 1, the scanning hologram-based automatic optical inspection system 100 may include a scanning hologram camera 110, an automatic optical inspection device 130, and a database 150.

The scanning hologram camera 110 may correspond to a device capable of photographing a hologram of a test object located on an objective board. The scanning hologram camera 110 may be connected to the automatic optical inspection device 130 through a network, and the plurality of scanning hologram cameras 110 may be connected to the automatic optical inspection device 130 simultaneously.

In one embodiment, the scanning hologram camera 110 may be implemented as a component of the automatic optical inspection device 130, in which case the scanning hologram camera 110 performs an operation of photographing a hologram of the object It may correspond to an independent module.

In one embodiment, the scanning hologram camera 110 includes a light source that generates electromagnetic waves, dividing means for dividing electromagnetic waves, scanning means for scanning an object using an interference beam formed by the divided electromagnetic waves, and reflection, fluorescence, or reflection from the object. It may include a light detection means for detecting the transmitted beam.

The light source may include various means such as a laser generator capable of generating electromagnetic waves, a light emitting diode (LED), a beam having low coherence, such as a halogen light having a short coherence length.

The dividing means may split an electromagnetic wave generated from a light source, for example, a laser beam into a first beam and a second beam. In one embodiment, the dividing means may include an optical fiber coupler, a beam splitter, a geometric phase lens, and a method of guiding a free space to deliver a beam to the outside Not only can be implemented as, it can be divided into a first beam and a second beam on a coaxial axis by using a means capable of dividing the beam in-line, such as the geometric phase lens.

The scanning means may scan an object to be photographed using an interference beam (or interference pattern) formed by the divided electromagnetic waves. The scanning means may correspond to a mirror scanner, but is not necessarily limited thereto, and may be replaced with various known scanning means. For example, the scanning means may scan a photographed object by moving a Fresnel zone plate across the photographed object. In this case, the scanning means can adjust the scanning position according to the control signal. In addition, the scanning means may scan the object by placing the object on the object plate and moving the object plate horizontally.

The light detecting means can detect the beam and convert it into a current signal. In this case, the light detecting means can generate a current according to the detected beam intensity. The light detection means may be implemented using a photodiode, but is not necessarily limited thereto, and may include various light detection means such as a photo-multiplier tube. In addition, the light detection means may be implemented by including a condenser for condensing the reflected, fluorescent or transmitted beam from the object to be photographed.

The automatic optical inspection apparatus 130 may be implemented as a server corresponding to a computer or program capable of acquiring a clear image regardless of defocus and rotation by photographing a hologram of a test object and processing it in a numerical method. The automatic optical inspection device 130 may be wirelessly connected to an external system (not shown in FIG. 1) that performs an independent operation through Bluetooth, WiFi, a communication network, etc., and may exchange data with an external system through a network. .

In one embodiment, the automatic optical inspection device 130 may store information required in the automatic optical inspection process in conjunction with the database 150. Meanwhile, unlike the FIG. 1, the automatic optical inspection device 130 may be implemented by including the database 150 therein. In addition, the automatic optical inspection device 130 may be implemented as a physical configuration including a processor, a memory, a user input/output unit and a network input/output unit.

The database 150 may correspond to a storage device that stores various information necessary in the process of performing a scanning hologram-based automatic optical inspection. The database 150 may store information on a hologram of an object acquired from the scanning hologram camera 110, and is not necessarily limited to this, and the automatic optical inspection apparatus 130 processes the acquired hologram by a numerical method In the process of obtaining a clear image of a test object regardless of defocus and rotation, information collected or processed in various forms may be stored.

FIG. 2 is a block diagram illustrating the functional configuration of the automatic optical inspection device in FIG. 1.

Referring to FIG. 2, the automatic optical inspection apparatus 130 includes a hologram photographing unit 210, a depth position and rotation angle extraction unit 230, a rotated coordinate system forming unit 250, a hologram restoration unit 270, and a control unit ( 290).

The hologram photographing unit 210 may use the scanning hologram camera 110 to photograph the hologram of the object existing on the objective board. In one embodiment, the hologram photographing unit 210 may generate a complex number hologram as a result of photographing.

In one embodiment, the hologram photographing unit 210 may photograph a hologram of an object using an optical scanning hologram-based photographing method. More specifically, the hologram photographing unit 210 may take a hologram of an object using the optical scanning hologram camera in the configuration of FIG. 5A. In FIG. 5A, the beam reflected or fluoresced from the object is illustrated to be detected, but it is needless to say that the beam transmitted from the object can be detected by placing the photo detector at the bottom of the object. At this time, it is preferable that the objective surface is transparent glass or the part of the object is pierced. At this time, the photographed hologram may be expressed by the following equations 1 to 5.

는 대상물의 반사율(reflectance)의 3차원 분포로서 대상물의 3차원 영상이고,

는 콘볼루션(convolution) 연산이다. 그리고,

는 스캔수단에 의해 지정되는 스캔빔의 스캔 위치이고, z는 대상물의 깊이위치로서 구면파의 초점에서부터 대상물까지의 거리에 해당된다.here,

Is a three-dimensional distribution of the reflectance of the object, and is a three-dimensional image of the object,

Is a convolution operation. And,

Is the scan position of the scan beam designated by the scanning means, and z is the depth position of the object, which corresponds to the distance from the focus of the spherical wave to the object.

Here, d is the distance between the focus of the first spherical wave and the focus of the second spherical wave. The hologram may adjust d to correct distortion caused by reduction and enlargement. As a method of adjusting d, it can be adjusted by changing the lens position and focal length according to the imaging law of the lens.

Here, M _img is the reduction or enlargement ratio of the image by the first lens when imaging the pattern of the polarization-sensitive lens (geometric phase lens) surface as the surface of the object region, z _img is the distance from the focal position of the second spherical wave to the object , 2M ² _img f _gp is the distance between each focal point of the adjusted first and second spherical waves.

The depth position and rotation angle extraction unit 230 may extract the depth position and rotation angle of the object surface of the object plate based on the hologram. Here, the objective surface may correspond to a plane parallel to the objective plate. That is, the depth position and rotation angle extraction unit 230 may process the hologram by a numerical method to calculate the depth position and rotation angle of the object surface. At this time, the numerical method may include a three-domain analysis-based extraction method, a CNN-based extraction method, and a gradient descent-based extraction method.

In one embodiment, the depth position and rotation angle extraction unit 230 is spaced apart from each other and extracts the depth position from each of the three regions defined independently, and then extracts the depth position and rotation angle with respect to the object surface can do. For example, when the object on the conveyor belt in FIG. 4 is not positioned horizontally on the object surface, the depth position and rotation angle extraction unit 230 extracts the depth position and rotation angle using a focus metric. The rotated coordinate system forming unit 250 may form a rotated coordinate system based on this.

Here, the three regions may correspond to regions indicated by small squares in FIG. 6, and may be defined as first, second, and third cell regions 610, 620, and 630, respectively. The three regions may be independently defined by being spaced apart from each other on a plane including the objective surface, and may correspond to a specific region including three different locations. In addition, the three regions can be independently defined on a plane corresponding to the object surface according to the depth position.

In one embodiment, the depth position and rotation angle extraction unit 230 respectively restores the hologram at a sequential depth position, and the second step calculates a focus metric in each of the three regions for the restored images. And a third step of determining the depth position at which the focus metric becomes the maximum value as the depth position of each of the three regions. The depth position and rotation angle extraction unit 230 may restore the holograms of Equations 1 to 5 at a sequential depth position by a method of digital back propagation, respectively. 6 shows a hologram restored at a sequential depth position.

As a specific example of a method of restoring a hologram at each sequential depth location, after forming the holograms of Equations 1 to 5, the conjugate complex number of the Fresnel zone plate of Equation 51 corresponding to each Convolution to the hologram at a sequential depth position to restore. This is given by the following equation (6). In addition, at each depth position, each angular spectrum corresponding to the Fresnel wheel of Equation 51 can be convolutionally restored, and can also be restored by the Rayleigh-Sommerfeld method. Rather, it can be restored through a variety of well-known digital back propagations.

Here, l={1, 2, 3, 4, 5} denotes a hologram obtained through Equations 1 to 5 and a Fresnel Yundae board corresponding to Equation 51, respectively. * Means complex conjugate.

In addition, the depth location and rotation angle extraction unit 230 can find the depth location at three points (610, 620, and 630 in FIG. 6) of the image reconstructed from the sequential depth location. As a specific method of finding the depth position, after the rotated hologram is restored at the sequential depth position (FIG. 6), the depth position focused on the first, second, and third cell regions of the images restored at the sequential depth position is obtained. . Various algorithms for finding the depth of focus are well known in the field of computer vision.

In addition, the depth position and rotation angle extraction unit 230 may determine the depth position at which the focus metric of the first, second, and third areas is the maximum, as the depth positions of the first, second, and third areas. The depth position and rotation angle extraction unit 230 may use various focus metrics, for example, a Tamura coefficient as a focus metric. In the n={1, 2, 3} region of the image reconstructed at the depth position z, the Tamura coefficient is as shown in Equation 7.

는 깊이위치 z에서 복원된 이미지의 제n={1, 2, 3} 영역에서 표준편차이고

는 깊이위치 z에서 복원된 이미지의 제n={1, 2, 3} 영역에서 평균이다. 상기 수학식 7에 따른 타무라 계수가 최대값을 갖는 제1, 제2 및 제3 영역에서의 깊이위치를 각각 z₁, z₂ 및 z₃이라 하면, 제1, 제2 및 제3 영역의 중앙점 좌표는 각각 (x₁, y₁, z₁), (x₂, y₂, z₂) 및 (x₃, y₃, z₃)이다. 여기에서, 깊이위치를 추출하기 위해 제1, 제2 및 제3 셀영역을 포함하는 홀로그램을 복원하는 것으로 설명하였으나, 제1 셀영역을 포함하는 일부분, 제2 셀영역을 포함하는 일부분, 제3 셀영역을 포함하는 일부분을 복원해서 상기 방법으로 깊이위치를 찾는 것도 물론 가능하다. From here,

Is the standard deviation in the n={1, 2, 3} region of the image reconstructed at the depth position z

Is the average in the n={1, 2, 3} region of the image reconstructed at the depth position z. If the depth positions in the first, second, and third regions in which the Tamura coefficient according to Equation 7 has the maximum value are z ₁ , z ₂ and z ₃ , respectively, the centers of the first, second, and third regions The point coordinates are (x ₁ , y ₁ , z ₁ ), (x ₂ , y ₂ , z ₂ ) and (x ₃ , y ₃ , z ₃ ), respectively. Here, the hologram including the first, second, and third cell regions is reconstructed to extract the depth location, but a portion including the first cell region, a portion including the second cell region, and a third Of course, it is also possible to find the depth position by restoring a portion including the cell region.

In addition, when the relative depth positions in the first, second, and third regions are different according to the 3D distribution of the object, the depth position and rotation angle extraction unit 230 determines the relative depth positions of the depth positions extracted by the above method. Depth position with respect to the objective surface can be extracted by correcting the information using. Referring to Figure (b) of Figure 5, when the object has a three-dimensional distribution, according to the method described above, the depth position in each corresponding area of the object may be extracted, not the depth position of the object surface. The depth position and rotation angle extraction unit 230 can grasp the 3D distribution of the object in advance, and correct the extracted depth position by using the information about the relative depth position (δd ₁ and δd ₂ ) to objective the object Depth position relative to the surface can be extracted.

In one embodiment, the depth location and rotation angle extraction unit 230 may extract the depth location directly from the hologram without restoring the image from the sequential depth location. At this time, the method used may include a method of numerically analyzing the fringe of the hologram.

More specifically, the depth position and rotation angle extracting unit 230 does not restore the hologram, a portion of the hologram including the first cell region, a portion of the hologram including the second cell region, and a portion including the third cell region Depth positions of the first, second, and third regions may be extracted from the hologram of the fringe analysis method. Here, since the amount of change in the hologram fringe is linearly proportional to the depth position in focus, the amount of change in the fringe is obtained to extract the focused depth position.

That is, after extracting the first, second, and third cell regions from the photographed hologram of FIG. 7, the real and imaginary portions of the hologram of the cell region are extracted, converted to pre, and then the real portions of each hologram converted to pre are extracted. Then, it is added by the complex number addition method to synthesize a real unique hologram. This is given by Equation 8 below.

Here, l={1,2,3,4,5} indicates holograms obtained through Equations 1 to 5, respectively, and n, {1,2,3}, first, second, and second 3 indicates the cell area. F{} is a two-dimensional free transform, Re[], Im[] are real operators that extract real and imaginary parts from complex numbers, imaginary operators, and (k _x ,k _y ) are spaces. It is a frequency axis. The fringe-adjusted filtering is performed by real-housing the hologram to obtain a Fresnel tread plate in which the amount of change in the fringe increases with depth. This is given by the following equation (9).

은 깊이위치 연계 파라미터로 상기 수학식 1 내지 5를 통해 획득된 홀로그램에 따라 다음의 수학식 10과 같다. Here, δ is a small value for preventing a pole problem in the 멱 fringe matching filter, and l={1,2,3,4,5} denotes the holograms obtained through Equations 1 to 5, respectively. Instruct

Is a depth location linkage parameter and is as shown in Equation 10 below according to the holograms obtained through Equations 1 to 5.

과 같다. 따라서, 기울기에서

를 구한 후 상기 수학식 10에 따라 초점 맺은 깊이위치 z를 구한다. 또는, 상기 수학식 9의 프레넬 윤대판의 축을 보간의 방법을 통해 새로운 축, (

) 변환하여 새로운 축에서의 상기 수학식 9를 프리에 변환하면 주파수 축에서

의 위치에서 피크 신호를 생성한다. 따라서, 신호의 피크 위치에서

를 추출하고 상기 수학식 10에 따라 초점 맺은 깊이위치 z를 구한다.When the wigner distribution of the Fresnel Yundae plate of Equation 9 is obtained, a straight line can be obtained in the spatial and spatial frequency domains. The slope of the straight line

Same as Therefore, at the slope

After obtaining, the depth position z in focus is obtained according to Equation 10 above. Alternatively, the new axis through the method of interpolating the axis of the Fresnel Yoon Dae-pan of Equation 9, (

) To pre-convert Equation 9 on the new axis to the frequency axis.

Generates a peak signal at the position of. Therefore, at the peak position of the signal

And extract the depth position z focused according to Equation 10 above.

In one embodiment, the depth position and rotation angle extraction unit 230 inputs hologram data to a convolutional neural network (CNN) model to obtain depth positions in each of the three areas as outputs, and then depth positions with respect to the object surface. And a rotation angle. Here, the CNN model utilizes hologram data for a learning object acquired from various 3D location points that are known as a target for a learning object and position information corresponding to the learning object as learning data, to a neural network previously learned by the learning unit. It may be. At this time, the position information is information including the depth position of the object. In this embodiment, CNN is described as an exemplary artificial neural network, but of course, various types of artificial neural networks may be used.

In one embodiment, the depth position and rotation angle extraction unit 230 is a holographic data obtained through the photographing, a complex hologram, a real hologram corresponding to the real part of the complex hologram, an imaginary hologram and a complex number corresponding to the imaginary part of the complex hologram. Any of the off-axis holograms synthesized using a hologram can be input to the CNN model.

In one embodiment, the depth position and rotation angle extracting unit 230 uses the CNN model generated through learning to input a specific area forming a hologram as an input and output the rotation angle of the rotated object surface to the object surface. Depth position and rotation angle can be extracted. Referring to FIG. 9, the depth position and rotation angle extraction unit 230 inputs a specific area 910 of the hologram to the CNN, and the rotation angles θ _x , θ _y of the rotated plane 830 of FIG. 8. CNN can be trained using θ _z ) as the output.

In one embodiment, the depth position and rotation angle extraction unit 230 may pre-convert the hologram into a spatial frequency domain and use an area corresponding to a specific domain as an input in the spatial frequency domain. That is, the depth position and rotation angle extraction unit 230 pre-converts the hologram into a spatial frequency domain, extracts a partial region from the spatial frequency domain, and uses it as an input of the CNN, and the rotation angle of the rotated plane 830 of FIG. 8. You can train CNN by outputting.

In one embodiment, the depth position and rotation angle extraction unit 230 may use at least one of a real part, an imaginary part, an amplitude part, and a phase part of a complex hologram related to a specific area forming a hologram as input. That is, the depth position and rotation angle extraction unit 230 is a combination of the real part, the imaginary part, the amplitude part, or the phase part of each part of the hologram in the spatial frequency domain, based on the fact that the hologram in the spatial frequency domain is a complex number. As an input of the CNN and outputting the rotation angle of the rotated plane 830 of FIG. 8 as an output, the CNN can be trained using a known CNN learning method.

In one embodiment, the depth position and rotation angle extraction unit 230 may extract the depth position and rotation angle with respect to the object surface using a gradient descent based on a partial or entire area of the hologram. That is, the depth position and rotation angle extraction unit 230 may search the depth position and the rotation angle to maximize the focus metric in a partial or entire area of the hologram by the gradient descent method.

In one embodiment, the depth position and rotation angle extraction unit 230 guides a part or the entire area of the hologram with respect to the rotational area and the positionable depth area where the object plate is rotatable, and then the focus metric of the waveguided hologram has the maximum value. You can search the rotation angle and depth position.

이다. 여기에서,

는 다음의 수학식 11에서 주어지는 변환 행렬이고

는 회전된 좌표계의 위치벡터이다. Referring to FIG. 8, if the x-axis, y-axis, and z-axis of the reference coordinate system are converted to a coordinate system rotated by θ _x , θ _y , and θ _z , respectively, the position in the rotated coordinate system is

to be. From here,

Is the transformation matrix given by the following equation (11)

Is the position vector of the rotated coordinate system.

From here,

The rotation matrix is given by and θ _x ,θ _y ,θ _z are rotation angles rotated around the x-axis, y-axis, and z-axis, respectively, as shown in FIG. 8.

라 하면, 여기에서 l={1,2,3,4,5}로 상기 수학식 1 내지 5의 홀로그램을 나타낸다. 기준 좌표계에서 특정 영역의 홀로그램을 회전된 좌표계로 각 스팩트럼 회전 변환법(angular spectrum rotational transformation)으로 변환하여 회전된 좌표계의 깊이방향으로 도파시킨 홀로그램은 다음의 수학식 12로 주어진다.In the reference coordinate system, a specific area 910 of the hologram of FIG. 9 is

Here, l={1,2,3,4,5} denotes the holograms of Equations 1 to 5 above. In the reference coordinate system, a hologram transformed into a spectral rotational transformation by transforming the hologram of a specific region into a rotated coordinate system in an angular spectrum rotational transformation is given by Equation 12 below.

는 회전된 공간 주파수축에서 회전된 공간축으로 2차원 프리에 역변환,

로

의 2차원 프리에 변환이고, (u,v)는 (x,y)축 방향의 공간 주파수축이고,

은 상기 수학식 11에서 변환 행렬(

)의 원소이고, (u_tilt,v_tilt) 각각 (x_tilt,y_tilt)축 방향의 주파수축이고

으로 z_tilt방향의 공간 주파수축이고,

는 좌표계 변환에 따른 미소적분면적소의 비례량을 나타내는 야코비안(Jacobian) 즉,

이다. From here,

Is an inverse transform from a rotated spatial frequency axis to a rotated spatial axis in a two-dimensional free,

in

Is a two-dimensional free-form transform of (u,v) is the spatial frequency axis in the (x,y) axis direction,

Is the transformation matrix (11)

), (u _tilt ,v _tilt ) and each (x _tilt ,y _tilt ) frequency axis in the axial direction.

Is the spatial frequency axis in the z _tilt direction,

Is a Jacobian that represents the proportional amount of the micro integral area according to the coordinate system transformation.

to be.

를 새로운 변수인 (u_tilt,v_tilt)축에 대해 보관을 이용해 등 간격으로 재 정렬(resampling)한 후 고속 푸리에 변환(Fast Fourier Transform)을 하는 방법으로 회전된 좌표계에서 홀로그램을 구하거나, 비 등 간격 데이터(non-equispaced data)에 대한 비 등간격 이산 프리에 변환(non-uniform discrete Fourier Transform) 또는 비 등간격 고속 프리에 변환(non-uniform Fast Fourier Transform) 등 비등간격 프리에 변환을 실시하여 구할 수 있다. When obtaining the image reconstructed from the z _tilt in the depth direction from the rotated axis according to equation (12),

For the new variables (u _tilt ,v _tilt ), the hologram is obtained from the rotated coordinate system by resampling at equal intervals using storage and then performing fast Fourier transform, or It can be obtained by performing non-uniform discrete Fourier Transform or non-uniform Fast Fourier Transform transformation on non-equispaced data. .

By way of example, the hologram is transformed and waveguided by angular spectrum rotational transformation, but it is needless to say that the hologram can be transformed and waveguided by a coordinate system rotated by various transformation methods.

과 가능한 깊이위치 즉

의 영역에 대해서 상기 수학식 12로 특정 영역(910)의 홀로그램을 도파시켜, 도파된 특정 영역(910)의 홀로그램의 초점 메트릭이 최대값을 갖게 하는 회전각과 깊이위치를 찾는다. 여기에서,

는 대물판이 회전 가능한 회전각 영역의 최소값과 최대값이고

는 회전된 좌표계에서 대물판이 위치가능한 깊이위치의 최소값과 최대값이다. 초점 메트릭의 최대값을 갖도록 하는 회전된 좌표계의 회전각과 깊이위치를 찾는 검색 방법으로는 기 공지된 다양한 검색 방법이 있으나 본 실시예에서는 경사 하강법(gradient descent)으로 찾는다. 여기에서, 경사 하강의 대상함수로 초점 메트릭 함수를 사용할 수 있다. 대표적인 초점 메트릭인 타무라 계수를 예로 들어 설명하면, 경사 하강법을 이용한 회전각과 깊이위치를 찾아내기 위한 대상함수는 다음의 수학식 121과 같다. The waveguide pattern in the coordinate system in which the specific region 910 of the hologram of FIG. 9 is rotated by θ _x and θ _{y along the} x-axis and y-axis may be obtained by Equation 12 above. Therefore, the rotation area where the objective plate is rotatable, that is

And possible depth positions i.e.

By guiding the hologram of the specific region 910 with respect to the region of Equation 12, the rotation angle and depth position of the hologram of the specific region 910 that has been guided to have the maximum value are found. From here,

Is the minimum and maximum value of the rotation angle area where the objective plate is rotatable.

Is the minimum and maximum values of the depth position where the objective plate can be positioned in the rotated coordinate system. As a search method for finding a rotation angle and a depth position of a rotated coordinate system having a maximum value of a focus metric, there are various known search methods, but in this embodiment, it is searched by a gradient descent method. Here, the focus metric function can be used as the target function of the gradient descent. When the Tamura coefficient, which is a typical focus metric, is described as an example, the target function for finding the rotation angle and depth position using the gradient descent method is as shown in Equation 121 below.

으로 회전된 좌표계의 회전각과 회전된 좌표계 깊이 방향으로 도파된 길이이고,

는 θ_x,θ_y,θ_z의 회전각을 갖는 회전된 좌표계에서 상기 수학식 12에 따라 회전된 좌표계의 깊이방향으로 z_tilt만큼 도파된 영상,

의 평균이고,

는 표준편차이다. 대상 함수가 최소값이 되는 정의역 중 대응 지점에서 초점 맺은 영상이 복원되므로, 회전된 좌표계의 회전각과 깊이위치는 대상함수의 정의역 중 대상함수의 최소값에 대응하는 지점, 즉

인

를 경사 하강법(gradient descent)으로 구한다. 첫째, 초기값

를 추정하고 다음으로 수학식 122의 반복식(iteration)에 따라, 수렴하는

를 찾는다.From here,

The rotation angle of the rotated coordinate system and the length guided in the depth direction of the rotated coordinate system,

Is an image waveguided by z _{tilt in} the depth direction of the rotated coordinate system according to equation (12) in the rotated coordinate system having a rotation angle of θ _x , θ _y , θ _z ,

Is the average of

Is the standard deviation. Since the image focused on the corresponding point in the domain where the target function is the minimum value is restored, the rotation angle and depth position of the rotated coordinate system are the points corresponding to the minimum value of the target function in the domain of the target function, that is,

sign

Is determined by gradient descent. First, the initial value

And then converging according to the iteration of Equation 122

Find

는 그래디언트(gradient) 연산자이고 α_k는 반복에 따른 단계크기(step size)로 코시(cauchy)법, 바르질리와 보웨인(Barzilai and Borwein)법, 다이와 위안(Dai and Yuan)법을 포함한 다양한 방법에 의해 결정될 수 있다. 예시적으로, 바르질리와 보웨인법에 따른 단계크기는

이다.here,

Is the gradient operator and α _k is the step size following repetition, and various methods including the Cauchy method, Barzilai and Borwein method, and Dai and Yuan method. It can be determined by. Exemplarily, the step size according to the Barzilli and Bowein method is

to be.

The rotated coordinate system forming unit 250 may form a rotated coordinate system corresponding to the objective surface using the depth position and the rotation angle.

In one embodiment, the rotated coordinate system forming unit 250 may form a rotated coordinate system through the following first method. Rotated planes passing depth locations (x ₁ ,y ₁ ,z ₁ ), (x ₂ ,y ₂ ,z ₂ ), (x ₃ ,y ₃ ,z ₃ ) of the first, second, and third cell regions The plane equation of is given by Equation 13 below.

는 회전되지 않은 기준 좌표계에서 위치벡터이고,

는 내적 연산자이고

는 제1, 제2 및 제3 셀영역을 지나는 평면의 법선벡터로 다음의 수학식 14로 주어진다. From here,

Is the position vector in the unrotated reference coordinate system,

Is the dot product operator

Is a normal vector of the plane passing through the first, second, and third cell regions, and is given by Equation 14 below.

행렬의 역행렬이다. Here, A ^-1 consists of rows of the first, second, and third cell regions.

It is the inverse of the matrix.

When a rectangular coordinate system is formed with an axis perpendicular to the rotated plane in the depth direction of the rotated plane, the unit direction vector in the depth direction is a normalized normal vector of the rotated plane and is given by Equation 15 below.

The unit direction vector in a direction parallel to the rotated plane is a unit vector having two points satisfying the plane equation of Equation 13 as a start point and an end point, and is given by Equation 16 below.

는 회전된 평면상의 임의의 두 위치로 상기 수학식 13을 만족시키는 두 개의 위치벡터이다. 회전된 평면상의 나머지 단위 위치벡터는 깊이 방향의 단위 위치벡터와 평면에 평행한 방향의 단위 방향벡터 곱(vector product)으로 다음의 수학식 17로 주어진다. From here,

Is two position vectors satisfying the above equation (13) with any two positions on the rotated plane. The remaining unit position vectors on the rotated plane are the unit position vectors in the depth direction and the unit direction vector products in the direction parallel to the plane, and are given by Equation 17 below.

는 회전된 평면 좌표계의 단위 방향벡터,

는 회전되지 않은 기준 좌표계의 단위 방향벡터이다. 상기 수학식 15 내지 17에 따라 기준 좌표계의 위치는 회전된 좌표계에서 다음의 수학식 18로 주어진다. If a rotated coordinate system having a unit direction vector rotated at the origin of the non-rotated reference coordinate system is defined, it is as shown in FIG. 8. In Figure 8,

Is the unit direction vector of the rotated plane coordinate system,

Is a unit direction vector of an unrotated reference coordinate system. The position of the reference coordinate system according to Equations 15 to 17 is given by Equation 18 below in the rotated coordinate system.

로 주어지는 변환 행렬(transform matrix)이다. From here,

Is a transformation matrix given by.

The center of the first, second, and third cell regions is (x ₁ ,y ₁ ,z ₁ ), (x ₂ ,y ₂ ,z ₂ ), (x ₃ ,y ₃ ,z ₃ ), and the reference Since the coordinate system and the rotated coordinate system share the origin, the depth position of the object plane rotated from the horizontal origin ((x,y)=(0,0)) of the reference coordinate system is calculated using the equation of a straight line passing two points in three-dimensional space. Is given by Equation 181 of

Where (x _n ,y _n ,z _n );n={1,2,3}, (x _m ,y _m ,z _m );m={1,2,3} is the first, second, and It is the center position of the third cell region.

In one embodiment, the rotated coordinate system forming unit 250 may form a rotated coordinate system through the following second method. The rotated coordinate system forming unit 250 may obtain the transformation matrix of Equation 11 by applying an input corresponding to the learned CNN model and obtaining the rotation angle of the rotated plane as an output. Here, in order to achieve the objective of the efficiency of CNN learning, CNN was trained by using the rotation angle as the output of the CNN. However, it goes without saying that the transform matrix can be obtained by learning the CNN by using each element of the transform matrix as the output. In addition, when extracting the rotation angle of the rotated plane using CNN, it is needless to say that the depth position can be found at an arbitrary position on the objective surface.

In another embodiment, the rotated coordinate system forming unit 250 outputs the CNN as the output of the rotation angle (θ _x ,θ _y ,θ _z ) of the rotated plane 830 of FIG. 8 and the depth position at the origin. By training, the rotation angle, which is the output of the CNN, and the depth position at the origin are obtained as the output of the learned CNN model, so that the transformation matrix and depth position according to Equation 11 of the rotated plane 830 can be obtained.

The hologram restoration unit 270 may acquire an image of the object by restoring the hologram in a plane formed in a depth direction of the rotated coordinate system.

; l={1,2,3,4,5}을 각 스팩트럼 회전 변환법(angular spectrum rotational transformation)으로 변환하여, 회전된 좌표계((

))의 깊이 방향으로 도파시켜 영상을 복원한다. 복원된 영상은 다음의 수학식 19으로 주어진다. In one embodiment, the hologram restoration unit 270 is a hologram obtained from the reference coordinate system ((x,y,z)),

; l={1,2,3,4,5} is converted to each spectral rotational transformation, and the rotated coordinate system ((

)) to restore the image by guiding it in the depth direction. The reconstructed image is given by Equation 19 below.

는 회전된 공간 주파수축에서 회전된 수축으로 2차원 프리에 역변환,

로

의 2차원 프리에 변환이고, (u,v)는 각각 (x,y)축 방향의 공간 주파수축이고,

은 상기 수학식 13에서 변환 행렬(T)의 원소이고,

는 각각

축 방향의 주파수축이고

으로

방향의 공간 주파수축이고,

로 좌표계 변환에 따른 미소적분면적소의 비례량을 나타내는 야코비안(Jacobian) 즉,

이다.From here,

Is a two-dimensional free inverse transform from the rotated spatial frequency axis to the rotated contraction.

in

Is a two-dimensional free-form transform of (u,v) is a spatial frequency axis in the (x,y) axis direction,

Is an element of the transformation matrix (T) in Equation 13,

Each

It is the frequency axis in the axial direction.

to

The spatial frequency axis in the direction,

Jacobian, which represents the proportional amount of the micro integral area according to the transformation of the low coordinate system, i.e.

to be.

에서 복원된 영상을 구할 때,

를 새로운 변수인

축에 대해 보관을 이용해 등 간격으로 재정렬(resampling)한 후 고속 푸리에 변환(Fast Fourier Transform)을 하는 방법으로 회전된 좌표계에서 홀로그램을 구하거나, 비 등 간격 데이터(non-equispaced data)에 대한 비 등간격 이산 프리에 변환(non-uniform discrete Fourier Transform) 또는 비 등간격 고속 프리에 변환(non-uniform Fast Fourier Transform) 등 비등간격 프리에 변환을 실시하여 구할 수 있다. From the rotated axis according to Equation 19 in the depth direction

When you retrieve the restored video from

Is the new variable

A method for obtaining a hologram in a rotated coordinate system by performing a fast Fourier transform after resampling at equal intervals using storage for an axis, or for non-equispaced data It can be obtained by performing a transform on a non-uniform discrete Fourier Transform or a non-uniform Fast Fourier Transform.

In the above embodiment, when the hologram of the reference coordinate system is converted into a rotated coordinate system and restored in a plane formed in a depth direction in the rotated coordinate system, a 3D image of the object may be obtained.

Exemplarily, the hologram is transformed and waveguided by each spectral rotational transformation method, but it can be understood that the hologram can be transformed and waveguided by a coordinate system rotated by various transformation methods.

In one embodiment, the hologram restoration unit 270 may acquire an image of the object by restoring the hologram at a depth position of the reference coordinate system and then guiding it to a plane formed in the depth direction of the rotated coordinate system.

을 상기 수학식 6에 따라, 기준 좌표계의 깊이위치에서 복원한 후, 회전된 좌표계의 깊이 방향에 수직인 평면으로 도파된 영상을 구하는 방식으로 회전에 무관하게 초점 맺은 영상을 구할 수 있다. 여기에서, 도파는 레일리-서머에펠드(Rayleigh-Sommerfeld) 방법을 포함한 다양한 디지털 도파 방법을 이용해 구할 수 있다. 다음의 수학식 191은 예시적으로 레일리-서머에펠드 방법을 이용해, 기준 좌표계의 특정 깊이위치에서 복원된 영상을 특정 깊이위치에 대응하는 회전된 좌표계의 평면으로 도파되어 회전된 평면에서 복원된 영상을 보여준다. That is, the hologram restoration unit 270 obtains the hologram obtained from the reference coordinate system ((x,y,z)),

According to Equation 6, after restoring at the depth position of the reference coordinate system, an image focused on regardless of rotation may be obtained by obtaining a waveguided image in a plane perpendicular to the depth direction of the rotated coordinate system. Here, waveguides can be obtained using a variety of digital waveguide methods, including the Rayleigh-Sommerfeld method. The following Equation 191, for example, uses the Rayleigh-Summerfeld method to guide an image reconstructed at a specific depth location in a reference coordinate system to a plane of a rotated coordinate system corresponding to a specific depth location, and then reconstruct the image reconstructed from the rotated plane. Shows

Here, λ is the wavelength of the laser used when recording the hologram, and O _rec (x,y,z) is an image reconstructed according to Equation 6.

In the above embodiment, when the hologram of the reference coordinate system is converted into a rotated coordinate system and restored in a plane formed in a depth direction in the rotated coordinate system, a 3D image of the object may be obtained.

In one embodiment, the hologram restoring unit 270 may restore the hologram in a sequential plane formed in the depth direction of the reference coordinate system, and then interpolate the rotated coordinate system.

In one embodiment, the hologram restoration unit 270 generates a 3D matrix based on the depth position of each sequential plane for the images reconstructed from the sequential plane, and interpolates the 3D matrix as the coordinate axes of the rotated coordinate system. By doing so, a 3D image of the object can be obtained.

을 기준 좌표계에서 상기 수학식 6에 따라, 순차적 깊이위치에 대해서 복원할 수 있다. 홀로그램 복원부(270)는 순차적인 깊이위치에 대해서 복원된 영상을 순차적인 깊이위치를 축으로 하여 3차원 행렬을 형성한 후 회전된 좌표계의 좌표축으로 순차적인 깊이위치에서 복원된 영상을 보간(interpolation)할 수 있고, 이는 다음의 수학식 20로 주어진다. That is, the hologram restoration unit 270 obtains the hologram obtained from the reference coordinate system ((x,y,z)),

In the reference coordinate system, sequential depth positions may be restored according to Equation (6). The hologram restoring unit 270 forms a 3D matrix using the reconstructed image for the sequential depth position as an axis, and then interpolates the reconstructed image at the sequential depth position as the coordinate axis of the rotated coordinate system. ), which is given by Equation 20 below.

은 기준 좌표계에서 회전된 좌표계로 변환하는 변환 행렬,

은 기준 좌표계의 위치벡터,

는 회전된 좌표계의 위치벡터이다.here,

Is a transformation matrix that converts from a reference coordinate system to a rotated coordinate system,

Is the position vector of the reference coordinate system,

Is the position vector of the rotated coordinate system.

The control unit 290 controls the overall operation of the automatic optical inspection device 130, the hologram photographing unit 210, the depth position and rotation angle extraction unit 230, the rotated coordinate system forming unit 250 and the hologram restoration unit ( 270) between the control flow or data flow can be managed.

3 is a flowchart illustrating a scanning hologram-based automatic optical inspection process performed in the automatic optical inspection device of FIG. 1.

Referring to FIG. 3, the automatic optical inspection apparatus 130 may take a hologram of an object existing on an object plate using the scanning hologram camera 110 through the hologram photographing unit 210 (step S310). The automatic optical inspection apparatus 130 may extract the depth position and rotation angle of the object surface of the objective plate based on the hologram obtained by the hologram photographing unit 210 through the depth position and rotation angle extraction unit 230. Yes (step S330).

In addition, the automatic optical inspection apparatus 130 uses the depth position and the rotation angle extracted by the rotation angle extraction unit 230 through the rotated coordinate system forming unit 250 to rotate the coordinate system corresponding to the objective surface. Can be formed (step S350). The automatic optical inspection apparatus 130 may acquire the image of the object by restoring the hologram in a plane formed in the depth direction of the rotated coordinate system through the hologram restoration unit 270 (step S370).

The automatic optical inspection apparatus 130 according to an embodiment of the present invention may calculate the depth position and the rotation angle of the objective surface by processing the hologram in a numerical manner through the depth position and rotation angle extraction unit 230. At this time, the numerical method may include a three-domain analysis-based extraction method (method A), a CNN-based extraction method (method B), and a gradient descent-based extraction method (method C). Depth positions and rotation angles can also be extracted in combination.

For example, the depth position and rotation angle extraction unit 230 analyzes 3 areas in method A to extract the depth position and rotation angle, but when extracting only the depth position by the method B or C, in method A, 2 areas The rotation angle can be extracted by analyzing the bay and including the extracted depth position information. At this time, when combined with method B, the CNN is trained to output only the depth position as an output, and when combined with method C, only the depth position can be found by the gradient descent method.

In addition, the depth position and rotation angle extraction unit 230 obtains the rotation angle or depth position as partial outputs from methods A to C, respectively, and combines the rotation angle and depth position information obtained by the methods A to C to rotate and The rotated coordinate system may be formed by extracting depth location information.

Although described above with reference to preferred embodiments of the present invention, those skilled in the art variously modify and change the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. You can understand that you can.

100: scanning hologram-based automatic optical inspection system
210: hologram photographing unit 230: depth position and rotation angle extraction unit
250: rotated coordinate system forming unit 270: hologram restoration unit
290: control
610: first area 620: second area
630: third area
810: reference plane 830: rotated plane
910: specific area

Claims (20)

A hologram photographing unit for photographing a hologram of an object existing on an object board using a scanning hologram camera;
A depth position and rotation angle extraction unit for extracting a depth position and a rotation angle with respect to an object surface of the object plate based on the hologram;
A rotated coordinate system forming unit for forming a rotated coordinate system corresponding to the objective surface using the depth position and rotation angle; And
A scanning hologram-based automatic optical inspection apparatus including a hologram restoration unit for acquiring an image of the object by restoring the hologram in a plane formed in a depth direction of the rotated coordinate system.
According to claim 1,
The scanning hologram camera includes a light source for generating electromagnetic waves, dividing means for dividing the electromagnetic waves, scanning means for scanning the object using an interference beam formed by the divided electromagnetic waves, and reflected, fluorescent, or transmitted beams from the object It includes a light detection means for detecting,
The hologram photographing unit is a scanning hologram-based automatic optical inspection device, characterized in that for generating a complex hologram as a result of the photographing.
According to claim 1, The depth position and rotation angle extraction unit
Scanning hologram-based automatic optical inspection apparatus characterized in that the depth position and the rotation angle with respect to the object surface are extracted after extracting a depth position in each of the three regions defined and independently spaced from each other.
According to claim 3, The depth position and rotation angle extraction unit
A first step of restoring the holograms at sequential depth positions, respectively;
A second step of calculating a focus metric in each of the three areas for reconstructed images; And
Scanning hologram-based automatic optical inspection apparatus characterized in that it performs a third step of determining the depth position at which the focus metric becomes the maximum value as the depth position of each of the three areas.
According to claim 3, The depth position and rotation angle extraction unit
Scanning hologram-based automatic, characterized in that by inputting hologram data into a convolutional neural network (CNN) model, a depth position in each of the three areas is obtained as an output, and then a depth position and a rotation angle with respect to the objective surface are extracted. Optical inspection device.
The depth position and rotation angle extraction unit of claim 5
As the hologram data, the complex hologram obtained through the photographing, the real hologram corresponding to the real part of the complex hologram, the imaginary hologram corresponding to the imaginary part of the complex hologram, and the off-axis synthesized using the complex hologram ) A hologram-based automatic optical inspection apparatus characterized in that any one of the holograms is input to the CNN model.
The depth position and rotation angle extraction unit of claim 1
Scanning characterized by extracting the depth position and rotation angle of the object surface using a CNN model generated through learning to input a specific area forming the hologram and output the rotation angle of the rotated object surface as an output. Hologram-based automatic optical inspection device.
The depth position and rotation angle extraction unit of claim 7
Scanning hologram-based automatic optical inspection apparatus, characterized in that after transforming the hologram into a spatial frequency domain in advance, an area corresponding to the specific region in the spatial frequency domain is used as the input.
The method of claim 7 or 8, wherein the depth position and rotation angle extraction unit
A scanning hologram-based automatic optical inspection apparatus, characterized in that at least one of a real part, an imaginary part, an amplitude part and a phase part of the complex hologram related to the specific area is used as the input.
According to claim 1, The depth position and rotation angle extraction unit
A scanning hologram-based automatic optical inspection apparatus characterized by extracting a depth position and a rotation angle with respect to the objective surface using a gradient search based on a partial or entire area of the hologram.
11. The method of claim 10, The depth position and rotation angle extraction unit
Scanning hologram-based, characterized in that, after guiding a part or the entire area of the hologram with respect to the rotatable area and the positionable depth area where the object plate is rotatable, the focus angle of the guided hologram is searched for the rotation angle and depth position having the maximum value. Automatic optical inspection device.
According to claim 1, The rotated coordinate system forming unit
Scanning hologram-based automatic optical inspection apparatus characterized by generating a transformation matrix that is generated using the depth position and the rotation angle extracted by the depth position and rotation angle extraction unit and converts a reference coordinate system into a rotated coordinate system.
According to claim 1, The hologram restoration unit
Scanning hologram-based automatic optical inspection apparatus, characterized in that by converting the hologram to the rotated coordinate system, by guiding in the depth direction of the rotated coordinate system to restore the image of the object.
The method of claim 13, wherein the hologram restoration unit
Scanning hologram-based automatic optical inspection apparatus characterized by acquiring the image of the object by converting the hologram into each spectral rotational transformation, and guiding in the depth direction of the rotated coordinate system.
The method of claim 13, wherein the hologram restoration unit
Scanning hologram-based automatic optical inspection apparatus, characterized in that by converting the hologram to the rotated coordinate system, each of the planes formed in the depth direction in the rotated coordinate system to obtain a three-dimensional image of the object.
According to claim 1, The hologram restoration unit
A scanning hologram-based automatic optical inspection apparatus characterized by acquiring an image of the object by restoring the hologram at a depth position of a reference coordinate system and then guiding it to a plane formed in a depth direction of the rotated coordinate system.
The method of claim 16, wherein the hologram restoration unit
Scanning hologram-based automatic optical inspection apparatus, characterized in that, after restoring the hologram at a depth position of a reference coordinate system, each of them is guided in a sequential plane formed in a depth direction of the rotated coordinate system to obtain a 3D image of the object.
According to claim 1, The hologram restoration unit
Scanning hologram-based automatic optical inspection apparatus, characterized in that the hologram is reconstructed in a sequential plane formed in the depth direction of the reference coordinate system, and then interpolated with the rotated coordinate system.
The method of claim 18, wherein the hologram restoration unit
Acquiring an image of the object by generating a 3D matrix with the depth position of each sequential plane as an axis for the images reconstructed from the sequential plane and interpolating the 3D matrix with the coordinate axes of the rotated coordinate system. Scanning hologram-based automatic optical inspection device characterized by.
In the method performed in the automatic optical inspection device,
Photographing a hologram of the object existing on the objective board using a scanning hologram camera;
Extracting a depth position and a rotation angle with respect to an object surface of the object plate based on the hologram;
Forming a rotated coordinate system corresponding to the objective surface using the depth position and the rotation angle; And
And acquiring the image of the object by restoring the hologram in a plane formed in a depth direction of the rotated coordinate system.

Images