BR102015011233A2 - AUTOMATIC POSITIONING METHOD AND EQUIPMENT FOR IN SITU PROBE SCANNING AND OPTICAL SPECTROSCOPY - Google Patents

Abstract

The present patent application describes a method and equipment capable of automatically positioning a probe used for scanning probe microscopy or spm procedures, including its coupling with optical systems, such as field optical microscopy. scanning near-field optical microscopy or snom) or probe enhanced raman spectroscopy (ters), which require it to be precisely positioned and manipulated. The proposed method and equipment are based on computational vision and closed-loop positioning control capabilities using visual feedback. spm, snom and ters experiments are useful in precision manufacturing, microelectronics and biomedical industries, among other areas of science and technology where such monitoring results in improved product quality and reliability.

Description

(54) Title: AUTOMATIC POSITIONING METHOD AND EQUIPMENT FOR PROBE SCANNING MICROSCOPY AND OPTICAL SPECTROSCOPY IN SITU (51) Int. Cl .: G01Q 60/20; G01Q 60/22; G01Q 20/02; G01J 3/44; G01Q 60/00; (...) (52) CPC: G01Q 60/20, G01Q 60/22, G01Q 20/02, G01J 3/44, G01Q 60/00, G01N 2021/656 (73) Holder (s): UNIVERSIDADE FEDERAL DE MINAS GERAIS (72) Inventor (s): LAURA PINTO COELHO AMORIM; HUDSON LUIZ SILVA DE MIRANDA; JOHNATHAN MAYKE MELO NETO; ADO JÓRIO DE VASCONCELOS; LUIZ FERNANDO ETRUSCO; LUIZ GUSTAVO DE OLIVEIRA LOPES CANÇADO; CASSIANO RABELO E SILVA (57) Abstract: This patent application describes a method and equipment capable of automatically positioning a probe used for probe scanning microscopy procedures (in English, Scanning Probe Microscopy or SPM), including its coupling with optical systems, such as near-field optical microscopy (from English Scanning Near-field Optical Microscopy or SNOM) or Raman spectroscopy by probe effect (from English Tip Enhanced Raman Spectroscopy or TERS), which require it to be precisely positioned and manipulated . The proposed method and equipment are based on computer vision and closed-loop positioning control using visual feedback. SPM, SNOM and TERS experiments are useful in precision manufacturing, microelectronics and biomedical industries, among other areas of science and technology where such monitoring results in an improvement in product quality and reliability.

Image Manipulation I

g

1/10 “AUTOMATIC POSITIONING METHOD AND EQUIPMENT FOR PROBE SCANNING MICROSCOPY AND OPTICAL SPECTROSCOPY IN SITU” [001] This patent application describes a method and equipment capable of automatically positioning a probe used for scanning microscopy procedures. probe (from English, Scanning Probe Microscopy or SPM), including its coupling with optical systems, as in near-field optical microscopy (from English Scanning Near-field Optical Microscopy or SNOM) or Raman spectroscopy by probe effect (from English Tip Enhanced Raman Spectroscopy or TERS), which require it to be precisely positioned and manipulated. The proposed method and equipment are based on computer vision and closed-loop positioning control using visual feedback. SPM, SNOM and TERS experiments are useful in precision manufacturing, microelectronic and biomedical industries, among other areas of science and technology where such monitoring results in an improvement in product quality and reliability.

[002] Experiments by SPM, SNOM and TERS are able to exceed the diffraction limit of light and produce images with atomic resolution, as well as extract information such as morphology, electrical conductivity, hardness and magnetic and optical properties of the materials under study.

[003] The document US7355422B2 entitled “OPTICALLY ENHANCED PROBE ALIGNMENT” of 04/08/2008 refers to a method of positioning verification by means of computer vision used in the inspection of integrated circuits, but does not apply in positioning probes in scanning optical microscopy.

[004] The work “Positioning and Calibration of an Underwater Robotic Manipulator with the Use of Computer Vision”, available at http://www.thinktank.net.br/publication/posicionadorsubmarino.pdf, proposes the use of computer vision and a control automatic to perform the calibration and movement of an underwater robotic manipulator. However, the proposal does not involve applications in scientific experiments, of nanoscale or precision, just as it does not propose hardware solutions, interfaces

2/10 man-machine, nor the assembly in which the technology will be implemented, just the theory and idea behind the implementation. In addition, no real prototype was developed, the ideas were tested only in simulations.

[005] In TERS experiments carried out in the laboratory, for example, there is an intrinsic difficulty in manually performing the probe positioning procedure with the LASER point centroid used. To carry out this experiment, the probe is attached to a positioning instrument that is then manually moved by the specialist. This manual positioning, which is performed automatically by the instrument presented here, is done with the aid of the image captured by the camera-telescope and camera-microscope systems, identical to those used in the present technology. In the manual adjustment procedure, based on the image, the specialist moves, with his hands, nano manipulators that move the measurement system until the tip is correctly positioned. This procedure, which requires the presence of a specialist technician, requires time, patience and training, as well as being subject to human errors, variations in the accuracy and duration of the positioning. In addition, the quality of the experiment itself is strongly associated with the quality of the positioning done, including the repeatability of the process, explaining its importance. These difficulties constitute limitations to the use and dissemination of SPM, SNOM and TERS techniques, being, therefore, barriers against not only the advancement of the techniques themselves, but also of any innovation that could come with their use.

[006] In view of these difficulties, a method and automatic positioning equipment with an easy-to-operate interface was developed, which allows lay users to use the system according to their needs and configurations and also to independently control all actuators integrated, eliminating the need for a dedicated specialist and the disadvantages associated with manual positioning.

3/10 [007] The equipment presented in the present invention can be used to support equipment capable of carrying out SPM, SNOM and TERS experiments. With it, greater repeatability, reliability, speed and robustness are added to the experiments, as it provides automatic positioning for Probe Scanning Microscopy and Optical Spectroscopy in situ, and opens up the possibility for other applications, such as sample placement .

[008] The equipment is divided into three subsystems: a vision, a positioning and a third, intelligence and control subsystem. The vision subsystem consists of one or more cameras; the positioning subsystem consists of actuators capable of moving the elements of interest in three dimensions. These actuators must communicate with the control elements (such as microcontrollers) that, in turn, communicate with the software; Finally, the intelligence and control subsystem (the software) consists of computer vision algorithms, closed-loop positioning control, optionally autofocus algorithms, and modules for communication with the sensors and actuators used in the solution.

[009] BRIEF DESCRIPTION OF THE FIGURES [0010] FIGURE 1 - Figure 1 shows a flowchart in which the stages of manipulating the images obtained by means of a camera are presented in order to obtain their contours, the images to be manipulated are the image of the LASER point and the image containing the probe and its reflection.

[0011] FIGURE 2 - Figure 2 shows a flowchart that indicates the steps for manipulating the contours, obtained from the actions indicated in Figure 1, the LASER point, the probe and its reflection. The manipulation of these contours aims to provide the coordinates of the LASER point, the probe and its reflection.

[0012] FIGURE 3 - Figure 3 shows a flowchart with the steps for locating the LASER point, the probe and its reflection, in order to update the positions of these elements.

4/10 [0013] FIGURE 4 - Figure 4 has a flowchart that shows the probe positioning steps that are performed iteratively until the end of the positioning.

[0014] FIGURE 5 - Figure 5 shows, through a flowchart, the actions that precede the actuation of the actuators: stepper motors (9) and (10) and picomotor (4).

[0015] FIGURE 6 - Figure 6 shows a probe scanning microscope provided with automatic positioning technology by visual feedback and its main constituent parts: tuning fork (1), probe (2), scanning unit (3) containing (1 ) and (2), picomotor (4), computer (5), telescope controller (6), picomotor controller (7), XY stage positioning system controller (8), X stage positioning system (9) , Y stage positioning system (10), microscope (11), optical system for image capture (12), telescope camera (13), microscope camera (14) and XY stage (15).

[0016] [0017] FIGURE 7 - Figure 7 shows a chronological sequence of images, numbered from 1 to 12, which illustrates the automatic positioning performed by the prototype of the proposed equipment.

[0018] DETAILED TECHNOLOGY DESCRIPTION [0019] The present patent application describes a method and equipment capable of automatically positioning a probe used for probe scanning microscopy procedures (in English, Scanning Probe Microscopy or SPM), including its coupling with optical systems, such as near-field optical microscopy (from English Scanning Near-field Optical Microscopy or SNOM) or probe effect Raman spectroscopy (from Tip Enhanced Raman Spectroscopy or TERS), which require the probe to be precisely positioned and manipulated. It can also be used in other applications where precise positioning is required, for example, sample positioning. The method and equipment presented are based on computer vision and closed-loop positioning control using visual feedback.

5/10 [0020] The method used to automatically position a probe used for scanning microscopy procedures comprises the following steps:

a) acquire images through a camera controlled by an image acquisition system capable of controlling the cadence and resetting it during the execution of the method, timing the activities of the camera and requesting the acquisition of images, in addition to counting images and use it to coordinate the method instructions;

b) recognize the laser point in an acquired image, isolating the channel with the most intense brightness, applying a filter, preferably the Gaussian one, intensifying the points of greater brightness and eliminating the points of less brightness, binarizing the image is extracted, the most relevant contour is extracted within the region of the most intense brightness, the relevant contour is circumscribed by a circle and its center is used as the local centroid;

c) detect in an image the nanoantenna and its reflection and determine the positions of both, applying a segmentation method that separates the regions that correspond to the first and second plan of the image by comparing the pixel intensity between the first and the second plane of the image in relation to the threshold value obtained by the adaptive method, preferably the Otsu method; after applying the threshold, the most relevant contours are obtained, which are subjected to a verification that examines pairs of contours and look for those that are spaced and aligned vertically and have similar and pertinent sizes, finally the selected contours are circumscribed; the intermediate circle is obtained by calculating the arithmetic mean of the coordinates of the other two circles obtained from the probe and its reflection and from the contour points closest to the circle, the innermost points that will be used as coordinates of the nanoantenna and its reflection;

d) verify and update data regarding the positions of the LASER point and the probe set and its reflection by submitting the images to a filter

6/10 low passes to provide more stability to the procedure and prevent the sending of commands to the actuators based on wrong measurements, analyze the coherence of the variations of the images, checking if the changes are physically possible and expected before updating them;

e) control the movements of the probe based on the calculation of the relative distances, horizontal (x) and vertical (Y), between the elements of interest (laser point, probe and its reflection) that are calculated through the images obtained after the step “d”; define, based on the images obtained after step “d”, the directions and directions of the movements of the probe approaching the LASER point, define the type of movement based on the calculated distances, and the movement of the probe is carried out by means of motors and it can be subdivided into three scales: movements of greater amplitude, moderate and steps, in accordance with the increasing order of precision intrinsic to the positioning; send the resulting commands to the actuators containing all the movement information and perform the steps iteratively until the necessary conditions for positioning are reached.

[0021] Steps “a” to “c” are indicated in the flowcharts represented in figures 1 and 2; step “d” is indicated in the flowchart shown in figure 3; and step “e” is indicated in the flowcharts represented in figures 4 and 5. [0022] Step “b” of the automatic positioning method for probe scanning microscopy and optical spectroscopy in situ, described previously, can be implemented using if other filters such as filter of average and median, preferably Gaussian. In step “c”, you can use other segmentation methods in addition to the one mentioned, for example, template matching (Template matching) or Harris corner detection (Harris corner detection) or background subtraction.

[0023] The probe positioning process begins with the detection of the positions of the laser point and the probe by measuring the relative distances of both in the X and Y directions of the coordinate system used,

7/10 then an algorithm determines the procedures adopted to carry out the approach of the probe to the center of the laser point that represents the reference of the control coordinate system.

[0024] The steps of image acquisition, probe and laser spot detection, focus measurement, emission of control signals to move the probe, as well as the change of focus occur in an iterative way until a pre-stop criterion -established is achieved.

[0025] The images obtained by the camera are fundamental in the visual feedback system present in the proposed technology. Camera settings and cadence are determined at the start of the alignment procedure and can be reset during the run. To recognize the laser spot, first isolate the channel with the most intense brightness and then apply a filter, for example, Gaussian, which softens the image. In the next stage the points of greater brightness are intensified and those of less brightness are eliminated. The image is then binarized and, within the region of interest (the center of the image), the most relevant outline is extracted, after distinguishing the main outline, it is circumscribed by a circle and its center is then used as the local centroid.

[0026] To discover the positions of the nanoantenna end and its reflection, a simple segmentation method is used, separating regions of the image that correspond to characteristics of interest. This separation is based on the comparison of pixel intensity between the foreground and background of the image in relation to the threshold value. Instead of feeding the software with a fixed limit value, an adaptive method, for example, of Otsu, is used. It is assumed that the image contains two classes of pixels (identified as the foreground and background) and the optimal separation limit between these two classes of pixels is calculated. After applying the limit, the most relevant contours are selected. Some prerequisites must be checked before deciding which outlines correspond to those being researched. The algorithm is executed, the instructions check pairs of contours and look for pairs of contours that are vertically spaced and aligned and have similar and pertinent sizes. The contours

8/10 selected are circumscribed and from the contour points closest to the circle (with three levels of proximity), the innermost points are chosen. These points will be used as coordinates of the nanoantenna and its reflection.

[0027] For each new image acquired through the camera, the positions of the elements (probe and laser point) are updated. However, the detection algorithm can sometimes give erroneous results for subsequent modules. Sudden changes in lighting intensity and other disturbances can lead to errors in detection and position. To avoid such a problem, a low-pass filter was implemented to provide greater stability to the procedure and prevent the sending of commands to actuators based on erroneous measurements. Before accepting the update in the element positions, the algorithm checks whether the variation is consistent with what is physically possible and expected. If so, the positions are updated, otherwise the commands are temporarily rejected and the previous position is stored as the most recent valid position. If the variation persists, which indicates that it is correct, the new position will be considered correct and the procedure will continue normally.

[0028] Based on the updates considered in the previous step, the relative distances, horizontal (x) and vertical (y), between the elements of interest (laser point, probe and its reflection) are calculated. Based on these distances, the direction and type of movement are determined. Three movement scales were outlined for the probe's movement system: movements of greater amplitude, moderate and steps, in increasing order of precision. After establishing the directions and distances of the movement, the commands are sent to the actuators. Finally, it is checked whether the necessary conditions for positioning have been reached. [0029] The quality of the focus of an image represents the perception of its brightness, contrast and definition. These characteristics can be measured according to a certain indicator and when the value of this indicator increases, for specific conditions, it indicates that the focus is improving. The Sobel function is used as a possible focus measure. This operator

9/10 performs a gradient measurement on an image, emphasizing regions of high spatial frequency, which correspond to the edges. By measuring this operator, the algorithm is able to diagnose whether the image is in focus, if the focus has been lost, the positioning algorithm is interrupted until the autofocus algorithm starts and re-establishes the proper focus, its objective is to move the telescope lens, changing the focus, until maximum sharpness is achieved. To do this, the module sends direction and speed information to the controller via the optical image acquisition system. As soon as the image reaches the proper focus, the positioning procedure is resumed.

[0030] The equipment is divided into three subsystems: a vision, a positioning and a third, intelligence and control subsystem. The vision subsystem consists of one or more cameras; the positioning subsystem consists of actuators capable of moving the elements of interest in three dimensions. These actuators must communicate with the control elements (such as microcontrollers, for example) which, in turn, communicate with the software whose lines of code contain instructions to automatically position a probe in the form of computer vision, control algorithms positioning in closed loop, automatic focus algorithms, optionally, and by modules for communication with the sensors and actuators used in the presented solution.

[0031] The equipment proposed in this patent deals with the probe scanning microscope provided with automatic positioning technology by visual feedback and its main constituent parts comprise the following elements: tuning fork (1), probe (2), scanning unit (3) containing (1) and (2), picomotor (4), computer (5), telescope controller (6), picomotor controller (7), XY stage positioning system controller (8), X stage positioning system (9), Y stage positioning system (10), microscope (11), optical system for image capture (12), telescope camera (13), microscope camera (14) and XY stage (15).

10/10 [0032] Elements (8) to (15) are versatile systems that can involve stepper motors or piezoelectric systems, optical telescope or objective lenses, depending on the need for precision, compactness of the system, or other technical and economic needs .

[0033] The proposed equipment is provided with an automatic focus algorithm that has a step that uses the focus measurement of the images acquired by the cameras, which indicates the sharpness of the images, in order to achieve the maximum sharpness of the images in which the images occur. probe and laser spot detections. For this to happen, there is a communication with the telescope controller that acts on the focus through a motor.

[0034] The invention can be better understood through the examples below, not limiting.

[0035] Example 1 - Experimental results of the prototype [0036] The prototype built of the probe scanning microscope provided with the present automatic positioning technology by visual feedback has the following constituent elements: HP Compaq DC 5800 Small Form Factor computer; the MSE1650B controller; the TTL / Analog Picomotor ™ driver Picomotor driver; the Arduino Motor Shield L293D controller; two SM1.8 stepper motors - A1734CMN; Nikon's microscope, Eclipse TE2000-U; motorized telescope KC VideoMax ™ Long Distance Microscope; V200e camera, from Invent Vision; ThorLabs camera, DCU223C; picomotor actuator from New Port, 8302 and the xy manual stage, from Thor Labs.

[0037] The interface between the operator and the equipment was made using software containing the automatic method of positioning by visual feedback.

[0038] Figure 7 shows a chronological sequence of images, numbered from 1 to 12, which illustrates the automatic positioning performed by the equipment prototype shown in Figure 6, showing the automatic approach of the probe gradually to the LASER point (in color) green).

1/3

Claims (6)

1. Automatic positioning method for probe scanning microscopy and optical spectroscopy in situ, characterized by comprising the following steps: a) acquire images through a camera controlled by an image acquisition system capable of controlling the cadence and resetting it during the execution of the method, timing the activities of the camera and requesting the acquisition of images, in addition to counting images and use it to coordinate the method instructions; b) recognize the laser point in an acquired image, isolating the channel with the most intense brightness, applying a filter, preferably the Gaussian one, intensifying the points of greater brightness and eliminating the points of less brightness, binarizing the image is extracted, the most relevant contour is extracted within the region of the most intense brightness, the relevant contour is circumscribed by a circle and its center is used as the local centroid; c) detect in an image the nanoantenna and its reflection and determine the positions of both, applying a segmentation method that separates the regions that correspond to the first and second plan of the image by comparing the pixel intensity between the first and the second plane of the image in relation to the threshold value obtained by the adaptive method, preferably the Otsu method; after applying the threshold, the most relevant contours are obtained, which are subjected to a verification that examines pairs of contours and look for those that are spaced and aligned vertically and have similar and pertinent sizes, finally the selected contours are circumscribed; the intermediate circle is obtained by calculating the arithmetic mean of the coordinates of the other two circles obtained from the probe and its reflection and from the contour points closest to the circle, the innermost points that will be used as coordinates of the nanoantenna and its reflection; 2/3 d) check and update data regarding the positions of the LASER point and the probe set and its reflection by submitting the images to a low-pass filter to provide more stability to the procedure and prevent the sending of commands to the actuators based on erroneous measurements , analyze the coherence of the variations of the images, checking if the changes are physically possible and expected before updating them; e) control the movements of the probe based on the calculation of the relative distances, horizontal (x) and vertical (Y), between the elements of interest (laser point, probe and its reflection) that are calculated through the images obtained after the step “d”; define, based on the images obtained after step “d”, the directions and directions of the movements of the probe approaching the LASER point, define the type of movement based on the calculated distances, and the movement of the probe is carried out by means of motors and it can be subdivided into three scales: movements of greater amplitude, moderate and steps, in accordance with the increasing order of precision intrinsic to the positioning; send the resulting commands to the actuators containing all the movement information and perform the steps iteratively until the necessary conditions for positioning are reached. 2. Automatic positioning method for probe scanning microscopy and optical spectroscopy in situ, according to claim 1, step “b”, characterized by the use of filters, such as the mean and median filter, preferably the Gaussian. 3. Automatic positioning method for probe scanning microscopy and optical spectroscopy in situ, according to claim 1, step “c”, characterized by the use of segmentation methods such as model matching (template matching), corner detection Harris (Harris corner detection), background subtraction, preferably the Otsu method. 3/3 4. Automatic positioning equipment for probe scanning microscopy and optical spectroscopy in situ, characterized by comprising a tuning fork (1), probe (2), scanning unit (3) containing (1) and (2), picomotor (4 ), computer (5), telescope controller (6), picomotor controller (7), XY stage positioning system controller (8), X stage positioning system (9), Y stage positioning system (10 ), microscope (11), optical system for image capture (12), telescope camera (13), microscope camera (14) and XY stage (15). 5. Automatic positioning equipment for probe scanning microscopy and optical spectroscopy in situ, according to claim 4, characterized in that the elements (8) to (15) may involve stepper motors or piezoelectric systems, optical telescopes or objective lenses. 1/6 FIGURES FIGURE 1 2/6 Contour Manipulation LASER Point Outline Outlines of the Probe and its Reflection Outlines are circumscribed with a circle The Center of the Circle is Considered the Center of the LASER Point Find the Points Nearest to the Circle Both outlines are circumscribed with a circle Elects, among the Points Found, those that are more Internal to the Contours Use these Points as Coordinates FIGURE 2 3/6 Confirms Alignment Start Evaluates the Consistency of Position Data Calculates the Average Position of the LASER Point Updates the Final Position of the LASER Point Probe and Reflex Calculates Average Probe and Reflex Position Evaluates the Consistency of Position Data Updates the Final Position of the Probe and Reflex FIGURE 3 4/6 o Send Commands to Actuators ---O.............................................. .................................................. . Check Distances and Signal End of Alignment FIGURE 4 5/6 Ο Checks the Count of Images Captured by the Camera Checks the Count of Images Captured by the Camera Sets the Speed and Direction of Movement of the Telescope Send Commands to Controllers Send Commands to Controllers FIGURE 5 «V 6/6 FIGURE 6 FIGURE 7 1/1