TW580580B - Method and apparatus for determining the spatial position and angular orientation of an object - Google Patents

Abstract

This invention relates generally to the method and apparatus for determining the spatial position and angular orientation of an object. The apparatus consists of two CMOS image sensors to capture the image of three markers on the target which is attached to an object to be measured. The markers are three LEDs which emit IR light. A central control unit not only periodically transmits activation commands through microwave transceiver to light up these three LEDs, but also sends enable signals to the image capture unit to capture the image of the three markers on two CMOS image sensors. An image processing unit calculates the positions of these markers spontaneously and the central control unit calculates the position and angular orientation of the center of the three markers i.e. the center of the target. The results are transmitted to the control center or PC through a serial communication interface. Then the next run starts again. This invention can be a position sensor for any virtual reality system, such as the position tracker for the hand or the head; it can be a distance measurement device for a 3D digitizing system; it can be an input device for PC too.

Description

5. Description of the invention d) and device. !! ： 明 ΐ :: Ϊ 之; Standing: Method for measuring attitude, taking units, image processing, driving source of single target source & image capture, where the target consists of two targets, central control early element and radio transceiver Device. Unit-driven; the scene is composed of a combination of light-emitting human diodes and bodies, and the sensor is driven by a light source. Jiu Bai Gu's early days contain a pair of two CMOS image light images side by side. The time gate of the three infrared light emitting diodes on the target is the marked light source. + The central control unit fixes the specific "= by the microwave receiver to the ♦ order to the light source drive unit so that the record is two: ⑨; and? Enable signal to enable the image manipulation unit to capture the target sensor unit can estimate immediately The two deleted images are marked according to the light source image of the Zn 4 light source. The "i :: marked light source and the corresponding two cm0s images of the central control unit are sensed by a series of i;: surfaces: Position and position the center point. Most of the time, it was transmitted to the console or personal computer for supply, and the next cycle of processing was started. ί: t1 bar is tied with an object, then the object can be used as a position and a device in the virtual reality, and when combined with the hand or head, it can be measured to determine the position of the hand or head ^ Vertical device (ρ 〇siti ο η raCker); can also be used as a three-dimensional digitizing system (3D digitizing -asure ^ nt devlce); computer and input settings (computer input device) to make up for the lack of keyboard or mouse functions . lce), a method and device for measuring the position and attitude of a private object

Page 6 580580 V. Description of the invention (2) The device is marked with passive retro-reflective film attached to the surface of the object, while other devices are marked with an active light source. Both types of technology ^ project this light source onto a position-sensitive sensor, and then use mathematical operations to determine the three-dimensional coordinates of each light source mark. If the relative position of these marker light sources is known, the position and attitude of the center points of these marker light sources can be known from their three-dimensional coordinates (ie, six-dimensional coordinates X, y, z, pitch, yaw, roi). In an active light source system, multiple CCDs are often used to detect the emitted energy of a labeled light source. Only one marker light source emits infrared light per sensing cycle. After the TJ Xuan infrared light is collected by the C C D sensors, they are processed and calculated by the sensing processing circuit.

The US Patent No. 4,672,562 of Egli et al., "Method and Apparatus for Determining Location and Orientation of Objects", is a method of placing multiple marked points on an object to form a vertical line. And imaged on the image plane of the camera, and detected by the method and device described in the patent. The space information of this object is calculated from the coordinates of these marked points. However, Egl i et al. Did not describe any hardware architecture to Explains how to detect the image points of the marked points.

Dementhon US 5,227,985 Patent Computer Vision System for Position Monitoring in Three Dimensions Using Non-Cop 1 anar Light Sources

Page 7 580580 V. Description of the invention (3)

Attached to a Monitored Object) ’’ uses a camera and more than four non-coplanar light sources on the object under test. When the number of marked light sources on the object is sufficient and the arrangement is complicated, this system can obtain the position and attitude of the object from some light points formed by the marked light source on the camera image plane. Leis US 5,8 2 8,7 7 0 patent "System for Determining the Spatial Position and Angular Orientation of an Object" " is based on a sensing unit and several objects on the object A system that actively measures the position and attitude of an object in real time. Its sensing unit has two cameras to capture the image of the marker light source. The system is in the initial marker 1 dent 1 f icat ion mode stage first. Then, in each sensing cycle, these several active marker light sources are emitted at the same time, and are detected by other sensors. In this system, 'because the marker light source has completed a 1 in the initial mark recognition mode phase, Second recognition, and its relative geometric position is also known. In the subsequent Marker-tracking mode stage, the light of the marker ft f, debt measurement and tracking can be performed simultaneously. This method is more suitable Inexpensive and relatively slow sensor cycle. A, = Γ ^ The application of the camera to stereo measurement technology is a conventional technology. It is obtained by triangulation at the image position to obtain the three-dimensional object.

"Behold / two-dimensional volume" technology, please refer to the MIT publication, • π machine vision by Horn "(" R〇b〇t

580580 V. Description of Invention (4)

Vision ,,). Most computer vision systems for 3D positioning require more than two CCD cameras. 5, 2 27, 98 5 The patent uses only one CCD camera in an attempt to reduce costs, but its assumption of orthographic projection also reduces the measurement accuracy (accuraCy). The price of a CMOS image sensor is lower than that of a CCD with the same resolution. The invention uses two CMOS image sensors to form an image capture unit (i m a g i n g u n i t), which can greatly reduce the cost while maintaining the advantage of high accuracy. The 5,8 2 8,7 70 patent divides the system operation into an identification mode and a tracking mode. The system is first in the identification mode, during which many marks are actuated in a sequential manner (activati). Allows the system to identify each mark. Then the system enters the lock mode. At this time, many `` marks '' are actuated at the same time to keep the sensing rate at the CC Hz frame rate of 60 Hz. Very ... The central control unit of the present invention. Because the markers are actuated at the same time, if the target may cause errors in the identification of the markers, the light sources on the markers are sequentially illuminated, and each frame cycle (frame period) to light up only one marker light source

Increase in the number of sub-standard six sources

580580 V. Description of the invention (5) Generally, computer vision systems use video cameras (the video signal output by video camera is standard NTSC standard. It is difficult to make the two cameras fully synchronized with the exposure and acquisition.) High, the present invention uses an integrated CM0S image sensor with a full digital interface ^^: The door switch and image acquisition are fully controlled by the central control unit, which can easily achieve complete synchronization between the two and the image sensor, eliminating the lack of synchronization The measurement error caused by the measurement error. The present invention proposes a parallel pipeline (paraUel pipe_Hne) image processing circuit architecture, which includes a paired one-dimensional median wavelet, dynamic threshold processing circuit, and instant peak detection circuit, and = Take the exchange circuit and microcomputer processor to achieve the second force of high-speed real-time processing. At the same time, when the image sensor element such as CMOS image sensor damages $ points, it can be eliminated by a one-dimensional median filter. Dead pixels (by pixe 1), which improves the system's fault tolerance.… A 5, 227, 9 85 and 5, 828, 770 patents all use active light sources, but When the light source is in a complicated environment, because the system does not process background noise such as daylight, glass reflection, etc., it is easy to misidentify the situation of the marker light source. The present invention proposes a method for instant division. The same daylight surface is taken out of the simple foreground and birth and the deduction is performed. At the same time with the use of the filter, there can be interference problems with the background light source of the two lambs, and it can meet the immediate needs.

580580 V. Description of the invention (6) 5, 227, 9 85 and 5, 828, 770 The active marker light sources used in the patent are all operated in a wired manner. When the target is far away or there is a wide range of position changes, control Too long a cord will cause inconvenience to use. The invention wirelessly controls the driving and triggering sequence of the marker light source through the microwave transceiver, which can increase the maneuverability and convenience during measurement. Schematic description · As shown in Figure- #, the present invention proposes a target light source driving unit 3, a target light source driving unit 3, a consisting of three marker light sources 1 (ie, LED 0, LED 1 and LED 2) The microwave transceiver 4, the image capturing unit 5, the image processing unit 6, and the central control unit 7 are configured. The related parameters of the optical-mechanical structure in the image capturing unit 5 are known, and at the same time, the marked light source = position on the W is arranged in a relatively geometric shape, # regular triangle or right-angled triangle is dedicated to facilitate the calculation of the target's attitude. The standard light source 1 can use infrared light emitting diodes (IR LEDs) to achieve its point light source, and the image capture unit 5 contains two "⑽ cry (while ge cai ') can capture the image of the marked light source. Central ~! The control unit 7 sends the command = moving order through the microwave transceiver 4 at a fixed time interval to make a specific marker light source! Light up; at this time, the central control order = re-enables the signal to enable the image capturing unit 5 to capture the marker The light source 丨 completion. The image processing unit 6 can estimate the position of the light source imaging point of the shirt image captured by the two CM0s image sensors in real time. The central control unit 7 then *

Page 11 ^ 80580 V. Description of the invention (7) '" 一 " ~-11 Any structure, the position of the target 1 relative to the system body as the origin can be distinguished, and finally through the serial communication interface Send it to the console / PC for provisioning and start the next cycle process. Gongkou〆 The present invention uses two integrated CMOS image sensors, which have a lower market price than traditional CCDs, and include an analog digital converter and exposure imaging. !! 1 circuit is available, so in addition to effectively reducing system cost, it can simplify circuit design and reduce system size. The integrated CMOS shirt image sensing is a full digital interface. Its shutter switch and image capture are fully controlled by the central control unit, which can easily achieve complete synchronization of the two image sensors. Measurement error.本本 明明 々; f shows that the marker light source 1 on the mark is sequntia 1 in a verbose manner. The parent frame period (frame per iod) only lights one marker light source to eliminate discrimination errors. opportunity. The image processing unit 6 calculates the target's six-dimensional coordinate change once in each frame period. In this way, the data update rate can be kept comparable to the frame rate, and will not be reduced due to the increase in the number of marker light sources. The invention proposes a method for real-time background light source elimination, which takes out the simple foreground and background light source images on the same screen and performs a subtraction operation, so that the data update rate and the frame rate can be made equal to the real-time requirement. At the same time, the use of optical filters can effectively improve the interference of background light sources. The invention also proposes wireless control via the microwave transceiver 4

580580

It can increase the mobility during measurement and the convenience of driving and triggering of the marker light source 1. Figure 2 is a schematic diagram of the optical-mechanical structure of the image capture unit. The front end of the image sensor is coupled with light attenuation; = image, sensor ... External (IR) filter 'optical mirror 9 to increase the light source and the backing film. Filter, light mirror) 8 and the red surface have an angle with the system reference coordinate axis, and /, i are different. The sensor is flat. The size of the month is determined by the circuit structure of the device to be tested. The device contains (1) an image sensor J 〇 It can take an image of the target and output an image. Zhanshan, the enlightenment place where the image was taken ... + The sound of the rear circuit of the machine is triggered by the outside of the image sensor. Ding (2) Image acquisition timing of microprocessor i i 10. (3) Command the translator to translate the result of the processing or control the image sensing. The operator 3 will translate the control command from the micro processor to the image sensing number and connect it to the translator. Capture trigger signal of two sensors Ensure that two image sensing = unified control by the microprocessor 11 can be taken at 10 at the same time

580580 V. Description of invention (9) Image. In addition to this source, two sets of image sensors 1 Q can make their data readout also fully synchronized. /. Use the same clock (4) Linear median filter 14 Linear median filter 14 The main power value is a bad pixel (Bad pixel), which eliminates the independent and its grayscale is not a marked light source 1 image Image point to avoid errors in peak detector detection. The linear intermediate value of the present invention is estimated from the position of the private light source 1 according to the wave wave method. The field wave method is changed. Each pixel is compared with the adjacent / secondary intermediate value. Binary. It can filter out the image points that are independent and whose gray value is maximum or minimum. (5) The dynamic threshold processor 1 5 generally uses a fixed threshold to remove the background, but when the ring is crossed, the background gray will change with it, so the fixed threshold has the defect of / N to 70 to completely separate the background. At the dynamic threshold used in the present invention, Zhai 15 can record the total gray value of the previous day surface. The microprocessor 11 can use this as a reference to dynamically adjust the threshold (6) peak position detector 16 used in the current picture. The peak detection of the system is completed by a hardware circuit, which can accelerate the marking light. Page 14 580580

Source 1 location search. The position with the highest degree is read back and the following peak detector 16 records the pixel gray of the current day. Microprocessor 11 converts the position through the system bus 12 to obtain the row and column coordinates. Line coordinates Peak address mod Total number of lines on the day and time, value address (7) Frame memory 1 7 Memory that can temporarily store the complete picture, there are two pairs. (8) The access switching circuit 18 accesses the parent switching circuit to provide u paths, and the frame remembers # 17 processing the four sets of image pixel data for the 11 switching control, and a system bus 12 connected. The two stored image anomalies / times twenty-six offenses were stored in the frame memory 17 respectively. In Gansu, one pixel m like the pixel stream is accessed by the system bus 12: the processor 11 can take the contents of another frame memory 17. (9) The absence of the background memory j 9 / there are two light points: point ... ㈣ ㈣ ((1 0) the microwave transceiver 2 0 contains the transmitting and receiving antennas and can convert the baseband signal to 580580. 5. Invention Note (11) modulates to a microwave frequency signal or demodulates the microwave signal (demodu 1 ate) to a fundamental frequency signal. The micro processor 11 wirelessly communicates with the light source driver 21 through the microwave transceiver 20. (1 1) Light source driver 2 1 and length 1 for lighting the current required by the micro processor to interpret the command sent by the microprocessor 11 and the LED designated by the processor 11. (12) LED module 22 contains three LED non-reflective materials on the panel. Light source 1. 'And arrange in the positive LED that is the mark of the triangle vertex position of the system. The color system is first in the identification process pair ~ identification, the system follows the two lock modes can be described in Figure 4 describes the system operation process 2 3, waiting for all marks Once the light source is completed, it will enter the lock process. The system has the following options: (1) High speed lock mode uses the hardware calculated peak position as the weight 2 4 (Threshold is still required to deduct the virgin and do not do Background deduction of King Wangli) (2) Robust lock mode 2 ^

Page 16 580580 V. Description of the invention (12) Use the position of the center of gravity of the vehicle body's area of interest (A0I) and make a background subtraction (three threshold deduction processing is still required). (Such as after 10 seconds) automatically re-enter the identification process.

Figure 5 is a flowchart of the identification process. After the three marker light sources (LEDs 0, 1, 2) are extinguished, the background image is stored in the frame memory 0, and the marker light source LED 0 is turned on. Box memory i. The microprocessor starts to perform the operation of subtracting the difference. The result image is stored in the frame memory, and then the peak position of the result image is searched. Taking this point as the center, a square window (hereinafter called the center of gravity window) is taken into account. The center of gravity is where the light source LED 〇 is located. Take the two light sources of the CMOS image sensor 1 ^ 1) 〇 the position of the center of gravity, and obtain the three-dimensional (x, y, z) coordinates of the light source UD 〇 after the algorithm. Mark the light source LED 1 and the light source 1 ^ 1) 2 according to the point shell, and record the center of gravity, position in the center of gravity, and position of the one-dimensional coordinates on the two CMOS shirt image sensing $ images. Finally, the center of gravity of the original labeled light source LED 〇 in the background image is W 0 ′ is stored in the background memory. A picture / is a flow chart of the Shaanxi speed-locking program. After the three marker light sources have completed the identification of one or two, only one marker light source is lit on each day in turn. The P-day item takes the peak register and converts the components. Do not use two CMOS images to reduce the two sets of f standards and update the center of gravity table. If you get rMnt 払 ’, you can count the three-dimensional coordinates of the marked light source in the system (with the center of the two image sensors as the origin), and then record according to the center of gravity table.

Page 17 580580 5. In the description of the invention (13), the peak coordinates of the two labeled light sources can be used to calculate the six-dimensional coordinates of the target ^. Seven is a flowchart of the robust locking procedure. Three marker light sources can obtain three marker light source windows after two identifications. If you make each daylight surface light only & σ σ light source 'Brother 1 daylight surface light mark light source LED 0' according to § in the center of gravity table, the light source LED 〇 Gravity center capture window W0, with W0 and W0, Deduction operation can get a simple foreground light source image. In the same day, the center of gravity of the marked light source D 1 recorded in the two tables f = f 1 is taken from the window w 1 and stored in $ 5 for future use. By analogy, the second screen lights up the marker light source LED ^ according to the marker light source LED 1 recorded in the center of gravity table 1 the center of gravity capture window W1, = 3 denier_surface point shell marker light source led 2 ... Take the foreground image from each side ^ A subtraction operation is performed on the background image of the day and day, and the background image of the light source is taken from the current frame as a backup. Finally, calculate the center of gravity by subtracting the background ... the result is stored in the center of gravity table.

And ^ are circuit architectures for high-speed real-time processing; software operations are more ^ = symmetrical design so that two CMOS images can be processed simultaneously. The individual pixel data is officialized. The first stage is the dead point management. The third stage is to write the frame memory. The present invention uses parallel pipelines to overlap the time-consuming process. It is implemented in hardware, and the image data stream of the sensor can be "IL (P i X e 1 stream) operation also eliminates the 'the second stage is the threshold at the J34JL ^^ 0 design an access switching circuit in the system' connecting the system bus and two

580580 V. Description of the invention (14) Frame memory, the microprocessor can switch the flow of pixel data. When a daytime surface is over, frame memory 0 is full. After switching the pixel data stream to frame memory 1, the microprocessor can completely access frame memory 0 for background subtraction and center of gravity. Calculation work. In this way, the microprocessor and the image pre-processing units can perform overlapping operations at the same time without interfering with each other.

Figure 1 System architecture diagram Figure 2 Optical-mechanical structure of the image capture unit and imaging diagram of the image sensor Figure 3 Circuit architecture diagram Figure 4 System operation flowchart Figure 5 Identification procedure flowchart Figure 6 Fast mode locking procedure flowchart Flowchart of Locking Procedure for Seven Robust Modes

Description of figure number: 1 Mark light source LED 0, LED 1, LED 2 2 Target

Page 19 580580 V. Description of the invention (15) 3 Light source driving unit 4 Microwave transceiver 5 Image manipulation unit 6 Image processing unit 7 Central control unit 8 ND filter 9 IR filter I 0 Image sensor II Micro Processor 1 2 System bus 1 3 Command translator 1 4 Median filter 1 5 Dynamic threshold 1 6 Peak position detector 1 7 Frame memory 1 8 Access switching circuit 1 9 Background memory 2 0 Microwave Transceiver 2 1 Light source drive unit 22 LED module 2 3 Identification program 24 High speed lock mode 2 5 Robust (r 〇bust) lock mode

Page 20

Claims (1)

580580 Six-pointer, patent application details ^ ———— month ~-a kind of rigid device for the position and attitude of the target, including a · target, which is inlaid with three · b ·-light source drive unit for : °°, > source, C. An image capture unit; contains the emission of the marker light source; used to receive the CM0S image of the sub-row when the marker light source is emitted d.-Image processing sheet s; t of ^ Own image; e ·-central control unit; f · A pair of radio transceivers, respectively, τ, τ ^ ^ ⑺ placed on the light source drive unit 亓 and control early, use the radio frequency of the specific frequency band and 2. If the amount of the first patent application scope Χ Μ, anti-interaction The H source a can be approximated as a point light source that is broken, and the standard central recording light described therein. 3. The measurement device according to item i of the patent application range, wherein the low reflection coefficient material is used. τ 艳 a 2 ^ The measurement device of the scope of the patent application, wherein the phototherapy drive: early b can click a single or all labeled light sources, and the lighting time length and driving current intensity of the marked light source are programmable Adjustment. ~ 5. ^ The measurement device for the magic item in the scope of the patent application, wherein the light% drive f 7Gb drives the marker light source according to the information received by the hair device f in the radio described in the measurement device in the item 丨 of the patent application scope and After receiving the correct message, an approval signal is sent to the source. 6. The measurement device according to the first patent claim, wherein the CMOS image sensor of the image capturing unit c has a condenser lens in front and a filter that allows only the wavelength of the marked light source to pass through. 7 · The measuring device according to item 1 of the scope of patent application, wherein the image processing Page 21 6. The patent application scope management unit d, which contains a • line-type median filter circuit, can independently noise pixels; in addition to the high-intensity b · image peak position detection circuit in the brother ’s image, the address of the pixel with the highest degree And its intensity value · Shou Huan's strong path in the image 'is used to remove the marked image. D. Access the parent circuit to switch the moon hall, miscellaneous ring, and path; Frame memory for storing passing images; this image processing unit can target / image. Then _ processed mark image The two original mark images are processed in order by the principle generated by °, the value position detection, and the real-time parallel processing at the value between the frame marks. There are four actions for U-body writing. 8 · If the amount of item 7 in the scope of patent application is applied σ 9 · If the month of item 7 in the scope of patent application is measured, the office will be redundant and tragic. . One of the frame memory e, A, becomes "-Chinese image processing unit" l from a threat to eight, and into two sets of memory, each group has a memory can store two steam at the same time Write two §Buy; and this frame is marked 10. If the measurement of item 1 of the scope of patent application is measured through the "5 JI" " imaging system e of Jingan f, it is described in the patent application ^ 8 The centrally controlled radio transceiver f transmits a signal from the 1 test device, where the information content is set to light the door ^ 7; the light source drive unit is notified to light according to the signal light source; and generates: the dynamic current intensity will be specified Target trigger k 旒 to drive the image capture unit to image Page 22 580580 11. According to the measurement of the dream strike unit e in the scope of the patent application, "the circuit containing the execution calculation process" is set: where the center refers to the structure position to calculate the center of gravity and time of the target < and image capture Units of the unit: 2. A kind of target position and attitude. The target is composed of two marked light sources. The light source is driven by a measuring system consisting of three devices, an image acquisition unit, and an image processing unit. Among them, the “central control unit first lights up the image capture unit sent by the taxi control unit to capture the marker image. Find out the position of the marker light source relative to the image coordinates. Then: calculate the spatial position of the marker light source relative to the image capture unit according to the known geometric positions of the two image sensors. Out: After locating the three marker light sources, every time you update any x: y: the position of the light source, the central control unit will then center the target: The calculation mode of the system is divided into high-speed mode and stable mode. The operation mode can be selected according to the complexity of the background environment. 13. According to the method described in item 12 of the scope of the patent application, the three marker light sources marked with "^" are sequentially lit, and only one marker light source is lit during each frame period to eliminate discrimination errors. opportunity. '~ 1 4. According to the method described in item 12 of the scope of patent application, the central processing unit contains a center of gravity table, which can record the relative peak coordinates of the three labeled light sources measured by two image sensors; each graph After the end of the frame period, the central control early element term takes the peak temporary sister and converts it into corresponding coordinates to update the center of gravity table. Page 23 580 ^ 8 ° Announce the patent next to the claim, and use the relative peak coordinates in the center of gravity table to dimension VI: If the method described in the scope of patent application No. 14 is used, the central control unit will calculate 15 · ^---• 丨 · to calculate Mark the three light sources, and then calculate the six-dimensional coordinates of the center of the handle according to the three labeled light source coordinate library structures recorded in the center of gravity table. " 6 The method as described in item 12 of the scope of the patent application ', wherein the system's high-speed mode described in the measurement 1 mode, the background noise is only removed by the threshold circuit, and β reduces the calculation time. i 7 The method as described in item 12 of the scope of the patent application, wherein the robust mode described in the measurement mode of the system 'its background noise removal method is to turn on the image when the marker light source is lit and the marker light source is not spotted. The bright images are subtracted, and the pixels with negative shame values are replaced with zero intensity values. The fixed background light source and noise can be eliminated 'while the reliable positioning of the Tienan marker light source 18 • The method described in item 17 of the scope of patent application, where the robust mode described in the system's measurement mode can be used in In the same day frame, the image with the lighted mark and the image with the unlighted mark are taken out respectively. i 9. According to the method described in item 12 of the scope of patent application, the central processing unit calculates the target's six-dimensional coordinate change once every frame period, and the data update rate will not decrease due to the increase in the number of marked light sources. Page 24