CN104155006B - A kind of hand-held thermal infrared imager and its method to the range finding of Small object quick lock in - Google Patents
A kind of hand-held thermal infrared imager and its method to the range finding of Small object quick lock in Download PDFInfo
- Publication number
- CN104155006B CN104155006B CN201410426808.6A CN201410426808A CN104155006B CN 104155006 B CN104155006 B CN 104155006B CN 201410426808 A CN201410426808 A CN 201410426808A CN 104155006 B CN104155006 B CN 104155006B
- Authority
- CN
- China
- Prior art keywords
- target
- module
- image
- information
- video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Studio Devices (AREA)
- Gyroscopes (AREA)
Abstract
The present invention relates to a kind of hand-held thermal infrared imager, including infrared imaging sensor, visible light camera, laser range finder, bluetooth module, magnetic compass module, MEMS microthrust test, GPS module, wireless transport module, linux system embedded signal panel and LED display, can be to the data fusion of multisensor, to the rational distribution of resources, realize complicated algorithm, improve handheld device performance, improve the systematicness of unit；Also disclose a kind of Small object quick lock in location algorithm of the hand-held thermal infrared imager based on linux system, principle according to MEMS microthrust test, using infrared and visible ray digital picture, fast and accurately lock onto target is reached by image processing algorithm, and follow the tracks of target, effectively reach quick lock in distance measurement function using laser range finder, the present invention can eliminate video jitter in real time, improve video quality, quick lock in target, automatic identification simultaneously detects target range, the feature of abundant handheld device.
Description
Technical field
The invention belongs to infrared thermal imaging technique field is and in particular to a kind of hand-held infrared thermal imagery based on linux system
Instrument, and its method to the range finding of Small object quick lock in.
Background technology
With developing rapidly of electronic technology and embedded technology, rapidly the carrying of various mobile terminal hardware platform performance
Rise, the development of technology it is also proposed new demand, hand-held thermal infrared imager just court to the product of hand-held portable infrared thermal imaging system
One-of-a-kind system, miniaturization, the direction of multifunction is developed.Require, on the basis of infrared imaging, to be integrated with CCD shooting
The peripheral hardwares such as machine, laser ranging, bluetooth, GPS, magnetic compass, radio communication.
Each single-sensor is all widely used it is necessary to by many peripheral hardwares, the overall management of many data, this is just badly in need of one at present
Kind of operating system allocating, laser, infrared, visible ray is integrated, and makes full use of the advantage of each sensor, merges each biography
The data of sensor, realizes rapid, effectively observation and monitoring, is accurately positioned target location.This technology monitors to front, personnel search
The fields such as rope, forest fires monitoring are even more important, and will be widely applied to the fields such as military surveillance, civilian detection.
In the application of hand-held thermal infrared imager, how view data and laser ranging function sufficiently to be utilized, and reduce
Power consumption, is urgent need to solve the problem.
Wherein, laser ranging is that generating laser is charged first, then passes through laser transmitter projects modulated optical signal, arrives
After reaching measurement target, return in reception eyepiece through diffuse-reflectance, receptor converts optical signals to the signal of telecommunication, then calculate mesh
Target distance.For the angle of energy, it is to charge first, the process finally discharged.If Small object can fast and accurately be measured
Be conducive to lowering power consumption.And its Small object is difficult to aim at, the often center of laser beam axis because of jitter deviation, it is difficult to receive unrestrained
The signal of reflection.On the one hand this cause the difficulty of object ranging, so that the ease for use of handheld device is greatly reduced, also results in simultaneously
During laser range finder is always at charging and discharging, power consumption is caused to raise.
Therefore, reduction shake brings image to rock and causes target especially Small object to be difficult to lock, be the pass having to solve
Key technology.By providing stable image, target quick and precisely can be positioned it is possible to reduce the work(of laser ranging process
Consumption.Thus the quick lock in ranging technology tool of research Small object has very important significance.
Content of the invention
It is an object of the present invention to provide a kind of hand-held thermal infrared imager based on linux system, increase hand-held infrared
Feature, to enrich hand-held infrared peripheral hardware and purposes, provide better image quality and target acquisition ability.
The technical solution adopted for the present invention to solve the technical problems is：A kind of hand-held thermal infrared imager, including for obtaining
Take visible light camera, the infrared imaging for obtaining target scene Infrared Image Information of target scene visible light image information
Sensor, laser range finder, the GPS for providing the latitude and longitude coordinates information of current device for obtaining target range information
Module, be used for providing short distance control interface and Digital Transmission, support the bluetooth modules of terminal control such as mobile phone, are used for being given currently
The magnetic compass module of the attitude information of equipment, for being wirelessly transferred video, digital signal and characteristic quantity interested, carry out many
The wireless transport module of apparatus interconnection and the LED display for image data presentation of information, also include linux system embedding
Enter formula signal panel, for controlling to the module in whole equipment, complete two-way video collection, laser range finder transmitting control
System, and to bluetooth module, magnetic compass module, MEMS microthrust test, GPS module, wireless transport module function control, and number
According to transmitting and sharing, also include MEMS microthrust test, for measuring angular displacement and the angular velocity of thermal imaging system, by SPI, signal is passed
Pass single-chip microcomputer, single-chip microcomputer is sent to linux system embedded signal panel by serial ports timing.
Described a kind of hand-held thermal infrared imager, its linux system embedded signal panel include AD conversion module,
VPIF interface, memory module, ARM chip, DSP, video conversion module, usb interface module, UART interface, I2C bus, ether
Network interface, fpga chip and serial ports, described AD conversion module input and visible light camera, infrared imaging sensor and
I2C bus connects, and AD conversion module outfan is connected with VPIF interface, and the described outfan of VPIF interface is with memory module even
Connect, the described input of memory module is connected with ARM chip and DSP, the outfan of memory module is with video conversion module even
Connect, described ARM chip also respectively with usb interface module, bluetooth module, wireless transport module, UART interface, I2C bus, with
Too network interface and fpga chip connect, and described DSP is connected with memory module, the described input of video conversion module and
I2C bus connects, and the outfan of video conversion module is connected with LED display, and described usb interface module is with ARM chip even
Connect, described UART interface is connected with ARM chip and fpga chip respectively, the described outfan of I2C bus and AD conversion mould
Block and video conversion module, described fpga chip and laser range finder, GPS module, MEMS microthrust test, magnetic compass module are even
Connect, fpga chip is connected with serial ports.
A further object of the present invention is to provide a kind of quick lock in range finding using above-mentioned thermal infrared imager to Small object
Method, the method is quick, energy consumption is low, can obtain the three-dimensional information of target further.
The technical solution adopted for the present invention to solve the technical problems is：One kind is based on hand-held infrared described in claim 1
The method that thermal imaging system is found range to Small object quick lock in, comprises the steps：
One）Two-way video Real-time Collection, and it is converted into digital video, obtain scene information
VPIF module is controlled manage by ARM end primary control program, for the collection of analog video, by AD conversion
Analog video is changed into digital video, the compressed encoding being completed the H.264 form of algorithm process and video using DSP end is regarded
Frequency stream file, obtains infrared and visible ray original video data, the acquisition for next step target and electronic steady image simultaneously；
Two）Positioning to interesting target, obtains the information of target
By image information direct interpretation current handheld equipment translational speed, or judged using MEMS microthrust test data
Current accekeration, by these data interpretations currently in quick coarse localization still in being accurately positioned；
Three）The quick lock in of Small object
Provide the particular location of Small object and the position of image pixel by image, by the position instruction of pixel, move
The dynamic heart in the picture, triggers lock image by button, and the information Store by Small object, simultaneously in the target lock-on stage, complete
The triggering of laser charging process, is charged process to laser range finder, carries out laser preparation pending in pairs；
Four）The quick distance measurement of target, obtains measurement result
According to the positional information of target, Digital Image Processing is carried out by DSP and obtains the position from optical axis center for the target, give
Go out the side-play amount of target, and by calculating the angle of divergence of laser, obtain obtaining the angular range of target range, when target is can
During the scope hit, rapid fire laser, calculate target range, close Laser Power Devices, reach reduction power consumption purpose.
A kind of described method to the range finding of Small object quick lock in, its step 2）In steady as under state target accurate
Acquisition comprises the steps：
a）Analog video is gathered by infrared imaging sensor 102 or visible light camera 101, is converted to by A/D
Whether digital video, be that the first two field picture judges to present frame, if previous frame image is then assigned to equally by the first frame
Numerical value, if not the first two field picture, be directly assigned to present frame；
b）Medium filtering is carried out to present frame, and marginal information is obtained using Sobel operator；
c）Process the enhanced image in edge, constant to the gray value of the point being more than specific threshold in image, less than feature
The point of threshold value is assigned to zero, carries out piecemeal process to current region, carries out decile N block for 768 × 576, selects suitable block
Join region, more rich to scene information, obtain higher weight, in image, marginal information amount is low, obtain relatively low weight.
d）By to the angular displacement of MEMS microthrust test and the calculating of angular acceleration, obtaining the direction of motion of image and probably inclined
Shifting amount；
e）By being calculated the side-play amount of the relative reference frame in each region, global motion is obtained by weighted value correction
Vector；
f）By using global motion vector, motion compensation is carried out to every two field picture, using the compensation side of bilinear interpolation
Method, is corrected to dither image, obtains stable video image；
g）After obtaining stable image sequence, the information of object observing.
A kind of described method to the range finding of Small object quick lock in, its step d）And e）Small Target quick lock in is found range
Method concrete steps comprise the steps：
A）First target is observed, find suspicious object；
B）Select the size of suitable target lock-on frame, target is locked, the information of target is stored, be used for
Recognition and tracking to target；
C）After finding target, laser range finder is charged, when discovery suspicious object passes through linux system embedded signal control
Making sheet sends charging process instruction to laser range finder；
D）In next two field picture, by the departure of MEMS microthrust test and steady picture, carry out the prediction of target, target is entered
Row correlation tracking, the position of interpretation Small object；
E）By target position in the picture, provide the positional information that target deviates laser center axle, by positional information
It is superimposed upon on the image of display；
F）By target from the side-play amount of optical axis center, it is adjusted round about, carry out relevant matches in real time, obtain
Target deviates the deviation of optical axis, when target reaches the scope that can hit, automatically launches laser；
G）Close whole object ranging process, close During laser emission.
The invention has the beneficial effects as follows：This thermal infrared imager has merged electronic steady image, target following scheduling algorithm, combines
GPS information, magnetic compass information simultaneously combine the range information that laser range finder exports, and target location are completed position, and pass in conjunction with more
Sensor, and the independent control advantage using linux system, complete the fusion of multiline message, employ ARM+DSP framework hand-held
The process of many algorithms is completed on thermal infrared imager, make use of spatial alternation principle, realize target is fast and accurately positioned,
The use in field such as facilitate look at, monitor.
Brief description
Fig. 1 is the structural representation of thermal imaging system of the present invention；
Fig. 2 is the structural representation of linux system embedded signal panel；
Fig. 3 is the workflow diagram of thermal imaging system of the present invention；
Fig. 4 is the algorithm workflow diagram of electronic steady image；
Fig. 5 is Small object quick lock in range finding flow chart.
Each reference is：001 target scene, 101 visible light cameras, 102 infrared imaging sensors,
103 laser range finders, 104 GPS module, 105 bluetooth modules, 106 MEMS microthrust test, 107 magnetic compass modules,
108 wireless transport modules, 109 linux system embedded signal panels, 109.1 AD conversion module, 109.2
VPIF interface, 109.3 memory modules, 109.4 ARM chips, 109.5 DSP, 109.6 video conversion module,
109.7 usb interface modules, 109.8 UART interface, 109.9 I2C buses, 109.10 Ethernet interfaces, 109.11
Fpga chip, 109.12 serial ports, 110 LED displays.
Specific embodiment
Below in conjunction with the accompanying drawings the present invention is described in further detail.
With reference to shown in Fig. 1, the invention discloses a kind of hand-held thermal infrared imager, including can for obtaining target scene 001
See the visible light camera 101 of light image information, be used for providing Infrared Image Information to obtain target scene 001 Infrared Image Information
Infrared imaging sensor 102, for by launch and receive laser obtain target range information laser range finder 103, use
In the latitude and longitude coordinates information providing current device GPS module 104, be used for providing short distance control interface and Digital Transmission,
Hold the bluetooth module 105 of the terminal controls such as mobile phone, MEMS microthrust test 106, magnetic sieve for providing the attitude information of current device
Disk module 107, wireless transport module 108, linux system embedded signal panel 109 and for image data information show
The LED display 110 showing, described MEMS microthrust test 106 is used for providing rotation and the offset information of equipment, and that is, attitude information is used
In angular displacement and the angular velocity of measurement thermal imaging system, by SPI by signal transmission to single-chip microcomputer, single-chip microcomputer is sent by serial ports timing
To linux system embedded signal panel 109, described wireless transport module 108 is used for providing and is wirelessly transferred and controls
Method is used for being wirelessly transferred video, digital signal and characteristic quantity interested, carries out many apparatus interconnections, described Linux system
System embedded signal panel 109 is ARM+DSP framework, for being controlled to each sensor by serial ports and USB, receives
Each sensing data, completes signal processing, for controlling to the module in whole equipment, completes two-way video collection（Infrared
Video and visible light video）, laser range finder 103 emission control, and to bluetooth module 105, magnetic compass module 107, MEMS
Microthrust test 106, GPS module 104, the function control of wireless transport module 108, and data transfer and shared, can be in Linux
Under system software platform, realize visible ray and the imaging of infrared two-way, to GPS, laser ranging（LRF）, MEMS gyro controls, soft
On part platform linux system, complete resource allocation, by completing process and the target following scheduling algorithm of algorithm with reference to peripheral hardware.
As shown in Fig. 2 linux system embedded signal panel 109 includes AD conversion module 109.1, VPIF interface
109.2nd, memory module 109.3, ARM chip 109.4, DSP109.5, video conversion module 109.6, usb interface module 109.7,
UART interface 109.8, I2C bus 109.9, Ethernet interface 109.10, fpga chip 109.11 and serial ports 109.12, described
AD conversion module 109.1 input be connected with visible light camera 101, infrared imaging sensor 102 and I2C bus 109.9,
AD conversion module 109.1 outfan is connected with VPIF interface 109.2, for receiving the visible ray of visible light camera 101 transmission
The infrared simulation picture signal that analog picture signal and infrared imaging sensor 102 send, the input of VPIF interface 109.2 with
AD conversion module 109.1 connects, and the described outfan of VPIF interface 109.2 is connected with memory module 109.3, for transmitting number
To in the middle of memory module 109.3, the input of memory module 109.3 is connected word video signal with VPIF interface 109.2, described
The input of memory module 109.3 is connected with ARM chip 109.4 and DSP109.5, the outfan of memory module 109.3 and video
Modular converter 109.6 connects, for storing infrared digital image signal and visible ray data image signal, for ARM chip 109.4
With DSP109.5 access, simultaneously also store (SuSE) Linux OS, ARM chip 109.4 except being connected with memory module 109.3, institute
The ARM chip 109.4 stated also respectively with usb interface module 109.7, bluetooth module 105, wireless transport module 108, UART interface
109.8th, I2C bus 109.9, Ethernet interface 109.10 and fpga chip 109.11 connect, for completing linux system
Run, and described each sensor of hand-held thermal infrared imager is controlled, complete digital video and video data（Laser ranging
The range data that instrument 103 obtains, attitude data that longitude and latitude degrees of data that GPS module 104 obtains, magnetic compass module 107 obtain, fast
Speed locking distance finder counts evidence）With the superposition of display interface, complete to be communicated with external equipment simultaneously, described DSP109.5
Be connected with memory module 109.3, be used for completing Digital Image Processing, the described input of video conversion module 109.6 with deposit
Storage module 109.3 and I2C bus 109.9 connect, and the outfan of video conversion module 109.6 is connected with LED display 110, use
In infrared image digital signal and visible images digital signal are respectively converted into analog video signal, export LED and show
Complete on screen 110 to show, described usb interface module 109.7 is connected with ARM chip 109.4, for being connected with external equipment,
Described UART interface 109.8 is connected with ARM chip 109.4 and FPGA109.11 chip respectively, is used for completing serioparallel exchange,
The input of I2C bus 109.9 is connected with ARM chip 109.4, the described outfan of I2C bus 109.9 and AD conversion module
109.1 and video conversion module 109.6, for configuring to AD conversion module 109.1 and video conversion module 109.6, with
Too network interface 109.10 and ARM chip 109.4 are connected to provide network interface, described fpga chip 109.11 and Laser Measuring
Distance meter 103, GPS module 104, MEMS microthrust test 106, magnetic compass module 107 connect, for receiving and processing the sensor
Data message, inside linux system embedded signal panel 109, fpga chip 109.11 and ARM chip 109.4, UART
Interface 109.8, serial ports 109.12 connects, and for being sent to the sensor information after processing in ARM chip 109.4, and completes
The selection of UART interface 109.8, serial ports 109.12 is connected with fpga chip 109.11, logical for being connected with external control deviceses
Letter.Due to employing the chip of ARM+DSP Duo-Core Architecture, and by DSP module, the association as ARM chip 109.4 processes mould
Block, in linux system running background, and can be carried out the interaction of data and realizes cooperated computing, this core by the way of shared drive
Piece completes most of signal processing work, undertakes the process of video signal, the process of view data, the control of communication, peripheral hardware mould
The calculating of the control of block, target following and electronic steady image.Infrared imaging sensor 102, visible light camera 101, laser ranging
Instrument 103, magnetic compass module 107, MEMS microthrust test 106 and GPS module 104, by serial ports 109.12 and usb interface module
Signal is passed to ARM chip 109.4 by 109.7, and data is passed to DSP109.5 by ARM chip 109.4, and DSP109.5 is to GPS module
104 and magnetic compass module 107 information superposition on image, DSP109.5 also completes to infrared image and visible images simultaneously
Image procossing, and assist the range finding to target.
It is illustrated in figure 3 linux system embedded signal panel 109 workflow：It is initial that system electrification carries out hardware
Change, read and run linux system in ARM chip 109.4 from memory module 109.3, DSP109.5 runs DSP/BIOS system；
System controls to be completed by the linux system of ARM chip 109.4, is taken the photograph by the infrared imaging sensor 102 and visible ray of leading portion
Camera 101, to obtain scene 001 information, is converted to digital video signal by AD conversion module 109.1, by digital video signal
Memory module 109.3 is passed to by VPIF interface 109.2；Dsp chip 109.5 access memory module 109.3 obtains numeral and regards
Frequency signal, completes Digital Image Processing and target recognition；Fpga chip 109.11 obtain laser range finder 103, GPS module 104,
MEMS microthrust test 106, the information of magnetic compass module 107, complete data processing, then by the data transfer after processing to ARM chip
109.4；ARM chip 109.4 accesses memory module 109.3, the visible ray digital picture after acquisition process and infrared digital simultaneously
Image, and by the range information of laser range finder 103, the latitude and longitude coordinates information of GPS module 104, magnetic compass module 107 appearance
State information is superimposed with digital image information；Digital image information after superposition is sent in video conversion module 109.6, completes mould
Intend video conversion, re-send to and complete on LED display 110 to show.
With reference to Fig. 4 and Fig. 5, the method for the Small object quick lock in range finding using thermal imaging system is done further details of
Explanation.
First step Real-time Collection two-way video, and it is converted into digital video, obtain scene information
VPIF module is controlled manage by ARM end primary control program, for the collection of analog video, by AD conversion
Analog video is changed into digital video, the compressed encoding being completed the H.264 form of algorithm process and video using DSP end is regarded
Frequency stream file, obtains infrared and visible ray original video data, the acquisition for next step target and electronic steady image simultaneously.
The two-way video collection that technical scheme adopts, wherein signal source infrared imaging sensor 102 and visible image capturing
Machine 101, by scene 001 through light path, is delivered on focal plane, by by the infrared radiation signal on focal plane and visible ray spoke
Penetrate signal and switch to analogue signal, then pass to the analog video interface of linux system embedded signal panel 109, by AD
The visible analog picture signal of acquisition and infrared simulation picture signal are converted into digital picture letter by modular converter 109.1 respectively
Number, sent to memory module 109.3 by VPIF interface 109.2, respectively obtain 768 × 576 videos, 25ms gathers the number of a frame
According to DSP109.5 accesses memory module 109.3, obtains data image signal, completes image procossing.
Second step positions interesting target, obtains target information
Scene 001 is mainly passed through sensor by the coarse localization of target, obtains infrared imaging sensor 102 and visible ray
The analogue signal of video camera 101, analog information is converted to digital signal, and then is shown on LED display 110.
Two-way switches in real time, selects to observe an image more clearly road video, and target characteristic is more apparent, by movement
Hand-held thermal infrared imager, adjustment different azimuth finds target, after basic determination target, enters the stage that is accurately positioned, enters steady
As pattern, this stage can clearly obtain scene information, target will be made blooming because of the shake of handss.
The acquisition of target includes the initial alignment of target and is accurately positioned two stages of stage, and the first stage, due to searching
Rope target occurs and significantly moves, image can bring certain blooming, second stage be in the first stage on the basis of, to figure
Stablizing of picture makes target stable in the central area of image.By image information direct interpretation current handheld equipment translational speed,
Or judge current accekeration using MEMS microthrust test 106 data, by these data interpretations currently quickly rough
Positioning is still in being accurately positioned.
Target, in coarse localization, using straight-through video, can switch current pixel value in real time, can be with hand-held infrared
The movement of thermal imaging system and change, target when being accurately positioned, steady as pattern under, can clearly obtain the information of target, no
Fuzzy image occurs, and the position from central shaft for the target can be given.
With reference to Fig. 4, to obtaining stable image, reduce shake and bring image blurring method to be described in detail：
1）The acquisition of scene, gathers analog video by infrared imaging sensor 102 or visible light camera 101, passes through
Whether A/D is converted to digital video, be that the first two field picture judges to present frame, if the first frame is then by former frame figure
As being assigned to same numerical value, if not the first two field picture, directly it is assigned to present frame.
2）Medium filtering is carried out to present frame, and marginal information is obtained using Sobel operator.
Image semantic classification, is filtered to noise by medium filtering, removes the noise of isolated point, using 3 × 3 mould
Plate, reduces amount of calculation.By arranging threshold value, carry out high-pass filtering, then by Sobel operator, edge is strengthened, obtain edge letter
Breath.
3）Process the enhanced image in edge, constant to the gray value of the point being more than specific threshold in image, less than feature
The point of threshold value is assigned to zero, carries out piecemeal process to current region, carries out decile N block for 768 × 576, selects suitable block
Join region, more rich to scene information, obtain higher weight, in image, marginal information amount is low, obtain relatively low weight.
Image-region divides, and every two field picture is carried out with the division in region, for different images and size, and target characteristic
The size of template is determining.For example, the sized images for 768 × 576, can be divided into 9 regions, and stencil-chosen 11 ×
11.The division in 4 regions equally can be selected, stencil-chosen is 15 × 15, meets real-time processing simultaneously.
4）By to the angular displacement of MEMS microthrust test 106 and the calculating of angular acceleration, obtaining the direction of motion of image and big
General side-play amount.
The selection of the most obvious feature, can select the Harris angle point in each region it is also possible to select feature in image
The most obvious block, is determined by block size.Employ the block of 15 × 15 sizes in embodiment, select feature edge the abundantest
Information.
5）By being calculated the side-play amount of the relative reference frame in each region, global motion is obtained by weighted value correction
Vector.
The selection of images match region of search, by the side-play amount of MEMS microthrust test 106 prognostic chart picture, first search offsets
Region is mated, and obtains departure by least absolute value difference.Can be quick in small range by using MEMS microthrust test 106
Block matching algorithm.
6）By using global motion vector, motion compensation is carried out to every two field picture, using the compensation side of bilinear interpolation
Method, is corrected to dither image, obtains stable video image.
This step calculates global motion vector, obtains stable image.After this process is mainly to images match, obtain each
The motion vector in region, by the distribution of weight, calculates the motion vector of the overall situation, obtains the side-play amount of entire image, to view picture
Image carries out motion compensation, and dither image is corrected, and obtains stablizing image.
7）After obtaining stable image sequence, the information of object observing.
Stable image is obtained by above method, both will not be brought by small size shake be affected, more visible observation mesh
Target position is it is also possible to be accurately positioned the position of Small object.
The quick lock in of the 3rd step Small object, to Small object real-time tracking
Quick lock in after accurately clearly obtaining Small object image information, to target especially Small object.
Provide the particular location of Small object and the position of image pixel by image, by the position instruction of pixel, move
The dynamic heart in the picture, triggers lock image by button, and the information Store by Small object, simultaneously in the target lock-on stage, complete
The triggering of laser charging process, is charged process to laser range finder, carries out laser preparation pending in pairs.
After small target auto-orientation, trigger press key locking Small object, the feature of Small object is preserved, real-time tracking Small object
Position, provides the side-play amount of target and optical axis, provides instruction for judging the time of transmitting laser.Laser emission is a kind of charging
The process of electric discharge, in the moment of Small object locking, the charge control to laser for the triggering, that is, laser is pending
In stable image/video, the position of Real Time Observation Small object, when Small object is in viewing area, touched by button
Send out and Small object is locked, be i.e. the information storage to the Small object of current interest, the position of real-time tracking Small object.Small object
Real-time tracking adopts correlation tracking, by calculating the similarity of target, calculates the particular location of Small object.Be given in real time in image
Target location.In target lock-on, that is, trigger laser ranging pattern, that is, the preparation of Laser emission to be done, quickly swashs
Light charging process, this avoid laser and is constantly in charging process, energy detects distant object enough simultaneously, be in laser and treat
Hair-like state.
The quick distance measurement of the 4th step target, obtains measurement result.
According to the positional information of target, Digital Image Processing is carried out by DSP and obtains the position from optical axis center for the target, give
Go out the side-play amount of target, and by calculating the angle of divergence of laser, obtain obtaining the angular range of target range, when target is can
During the scope hit, rapid fire laser, calculate target range, close Laser Power Devices, reach reduction power consumption purpose.Picture frame it
Between skew calculated, obtain the deviation from central shaft for the target, be adjusted by departure, and by calculate the angle of divergence,
Calculate target and deviate in much scopes of optical axis and can hit the mark.
According to the target location of display on LED display 110, laser in much regions is calculated by DSP and can hit mesh
Mark, that is, pass through to calculate the angle of divergence of laser, calculate and can hit the mark in the much radians in laser beam axis center.Pass through mesh simultaneously
Mark tracking data, obtains the position that target deviates lasing central, manually adjusts thermal imaging system, close to optical axis center,
The scope that arrival can be hit, rapid fire laser, by detecting whether to hit, continuously found range.Obtain target away from
From information, by target range information superposition on image, it is shown on LED display 110, exits target following pattern simultaneously,
Carry out target detection mode next time.Now software triggers closing Laser Power Devices automatically, is conducive to less power consumption.
The range finding realization of Small object quick lock in is being described in detail with reference to Fig. 5 for above step.This technology is base
On the basis of electronic steady image and target following technology, by the real-time tracking to target, in conjunction with the center of laser beam axis, obtain
Target deviates the center of optical axis；The scope automatic decision that can hit the mark in conjunction with laser, to Small object quick distance measurement
Process.Its step is：The locking of target, observes to target first, finds suspicious object, is triggered to target by button
Locking, and target information is preserved, finds Small object in the next frame；Select the size of suitable target lock-on frame, to mesh
Mark is locked, and the information of target is stored, for the recognition and tracking to target；After finding target, laser range finder is filled
Electricity, is conducive to reducing the charging interval, is conducive to reducing power consumption, rapid fire laser simultaneously, is conducive to energy sufficient, when discovery can
Doubtful target is passed through linux system embedded signal panel 109 and laser range finder is sent with charging process instruction；In next frame figure
In picture, by the departure of MEMS microthrust test 106 and steady picture, carry out the prediction of target, correlation tracking is carried out to target, interpretation is little
The position of target；By target position in the picture, provide the positional information that target deviates laser center axle.By positional information
It is superimposed upon on the image of display；By target from the side-play amount of optical axis center, it is adjusted round about, carry out correlation in real time
Coupling, obtains the deviation that target deviates optical axis, according to side-play amount and laser beam divergence, obtains much regions interior energy and accepts to be reflected back
The laser coming, judges Small object within the range, when target reaches the scope that can hit, automatically launches laser；Close whole
Individual object ranging process, closes During laser emission.
S1）Target information obtains and the stage of recognition.This process mainly has the initial alignment of target, that is, determine mesh interested
Target approximate location；Being accurately positioned of target, that is, obtain the accurate location of target；The locking of Small object, will target characteristic letter
The acquisition of breath.This process mainly passes through the analysis to image information for the observer, obtains target interested, then is touched by button
Send out and obtain target information.
S2）Target following and target location calculate.After target lock-on, according to the information of target storage, in next frame figure
As carrying out correlation tracking.Correlation tracking technology is the similarity principle based on image, by arranging the Similarity value of target characteristic,
In application region phase knowledge and magnanimity highest value as coupling target, using least absolute value Error Calculation.Wherein, object matching is base
In image inter-frame relation, matching speed is even more important to raising real-time and rapidity.Target search region and the size of target
Determine the time of search and the precision of coupling, target frame is bigger and hunting zone is bigger, and error rate is higher, and matching precision is got over
High.In this process embodiments, the angular displacement according to MEMS microthrust test 106 and the interval of angular velocity Optimizing Search, reduce target
Hunting zone.In 768 × 576 image-regions, select 11 × 11 target frame.
This process also completes Laser emission preparation.Right by linux system embedded signal panel 109 before starting
Laser range finder sends charging process instruction, completes the charging to laser range finder.Charging interval controls critically important, discovery target
Charge immediately, be conducive to reducing the charging interval and reducing power consumption.Rapid fire laser simultaneously, is conducive to energy sufficient.
S3）Calculate target from laser center shaft position, whether interpretation can hit the mark.By target sizes and target search
The selection in region, inclined between the position in the picture of target position and laser center optical axis in the picture by being calculated
Residual quantity, provides the departure of target in real time, and positional information is superimposed upon on the image of display.By target abaxial
Side-play amount, is adjusted round about, carries out relevant matches in real time, obtains the deviation that target deviates optical axis.Laser ranging
Scope calculates gained according to the size of the angle of divergence, target, adopts the size of target frame in previous step, should object observing big
Little, then deviation range can be calculated by the angle of divergence.Image shows in how many pixel coverages of deviation and can obtain mesh
Mark, tentatively obtains the scope hit.
S4）Laser ranging, this process is not manual triggering, but is based on last stage measurement result, automatically launches
Journey.Whether can detect what target obtained by interpretation, wait and launched laser, by the relation calculating between light path, obtain mesh
Target distance.When target reaches the scope that can hit, automatically launch laser, close whole object ranging process, close and swash
Light emission process.By the overlay distance information of target on image, intuitively can obtain from LED display 110 target away from
From information.
S5）Measure next target, continue to observe this target under surely as state.This process can automatically switch, and enter and see
Examine pattern, exit steady picture pattern.
, surely as technology, its step is for electronic flutter-proof technology in described target positioning and image：
（1）Obtain current frame image, pretreatment is carried out to image.Using medium filtering, remove isolated noise point.Pass through again
Image border strengthens, and obtains harsh image information characteristics.
（2）To picture portion domain, search the most obvious characteristic area in each region.And to each areal division of power weight, lead to
Cross weight size, calculate final side-play amount.
（3）Small range block matching algorithm, that is, pass through MEMS microthrust test, carry out pre- interpretation to global motion, reduce region
Selection, preferentially Block- matching is carried out to this region.
（4）Using the weight of regional, it is calculated global motion vector.By motion vector to jitter compensation, obtain
To stable image.
Above-described embodiment only principle of the illustrative present invention and its effect, and the embodiment that part is used, for
For those of ordinary skill in the art, without departing from the concept of the premise of the invention, can also make some deformation and
Improve, these broadly fall into protection scope of the present invention.
Claims (4)
1. a kind of hand-held thermal infrared imager it is characterised in that：Including for obtaining target scene (001) visible light image information
Visible light camera (101), it is used for obtaining the infrared imaging sensor (102) of target scene (001) Infrared Image Information, is used for
Obtain laser range finder (103), the GPS module for providing the latitude and longitude coordinates information of current device of target range information
(104), it is used for providing short distance control interface and Digital Transmission, support the bluetooth module (105) of mobile phone terminal control, be used for being given
The magnetic compass module (107) of the attitude information of current device, for being wirelessly transferred video, digital signal and feature interested
Amount, carries out wireless transport module (108) and the LED display for image data presentation of information of many apparatus interconnections
(110), also include linux system embedded signal panel (109), for controlling to the module in whole equipment, complete double
Road video acquisition, laser range finder (103) emission control, and micro- to bluetooth module (105), magnetic compass module (107), MEMS
Gyro (106), GPS module (104), the function control of wireless transport module (108), and data transfer and shared, also include
MEMS microthrust test (106), for measuring angular displacement and the angular velocity of thermal imaging system, by SPI by signal transmission to single-chip microcomputer, monolithic
Machine is sent to linux system embedded signal panel (109) by serial ports timing；Described linux system embedded signal
Panel (109) includes AD conversion module (109.1), VPIF interface (109.2), memory module (109.3), ARM chip
(109.4), DSP (109.5), video conversion module (109.6), usb interface module (109.7), UART interface (109.8), I2C
Bus (109.9), Ethernet interface (109.10), fpga chip (109.11) and serial ports (109.12), described AD conversion
Module (109.1) input is connected with visible light camera (101), infrared imaging sensor (102) and I2C bus (109.9),
AD conversion module (109.1) outfan is connected with VPIF interface (109.2), the described outfan of VPIF interface (109.2) with
Memory module (109.3) connection, the described input of memory module (109.3) and ARM chip (109.4) and DSP (109.5)
Connect, the outfan of memory module (109.3) is connected with video conversion module (109.6), described ARM chip (109.4) is also
Respectively with usb interface module (109.7), bluetooth module (105), wireless transport module (108), UART interface (109.8), I2C
Bus (109.9), Ethernet interface (109.10) and fpga chip (109.11) connect, described DSP (109.5) with deposit
Storage module (109.3) connects, and the described input of video conversion module (109.6) is connected with I2C bus (109.9), video
The outfan of modular converter (109.6) is connected with LED display (110), described usb interface module (109.7) and ARM chip
(109.4) connect, described UART interface (109.8) is connected with ARM chip (109.4) and FPGA (109.11) chip respectively,
The described outfan of I2C bus (109.9) and AD conversion module (109.1) and video conversion module (109.6), described
Fpga chip (109.11) and laser range finder (103), GPS module (104), MEMS microthrust test (106), magnetic compass module
(107) connect, fpga chip (109.11) is connected with serial ports (109.12).
2. a kind of method Small object quick lock in found range based on thermal infrared imager hand-held described in claim 1, its feature exists
In comprising the steps：
One) Real-time Collection two-way video, and it is converted into digital video, obtain scene information
VPIF module is controlled manage by ARM end primary control program, for the collection of analog video, by AD conversion by mould
Intend video and change into digital video, the compressed encoding of the H.264 form completing algorithm process and video using DSP (109.5) end obtains
To video stream file, obtain infrared and visible ray original video data, acquisition and electronics for next step target are steady simultaneously
Picture；
Two) positioning to interesting target, obtains the information of target
By image information direct interpretation current handheld equipment translational speed, or sentenced using MEMS microthrust test (106) data
Disconnected current accekeration, by these data interpretations currently in quick coarse localization still in being accurately positioned；
Three) quick lock in of Small object
Provide the particular location of Small object and the position of image pixel by image, by the position instruction of pixel, move
Picture centre, triggers lock image by button, and the information Store by Small object, and simultaneously in the target lock-on stage, it is right to complete
The triggering of laser charging process, is charged process to laser range finder, carries out laser preparation pending；
Four) quick distance measurement of target, obtains measurement result
According to the positional information of target, Digital Image Processing is carried out by DSP (109.5) and obtains the position from optical axis center for the target
Put, provide the side-play amount of target, and by calculating the angle of divergence of laser, obtain obtaining the angular range of target range, work as mesh
When being marked on the scope that can hit, rapid fire laser, calculate target range, close Laser Power Devices, reach reduction power consumption purpose.
3. a kind of method to the range finding of Small object quick lock according to claim 2 is it is characterised in that described step
Two) in, the accurate acquisition of target information comprises the steps：
A) pass through infrared imaging sensor (102) or visible light camera (101) gathers analog video, be converted to by A/D
Whether digital video, be that the first two field picture judges to present frame, if previous frame image is then assigned to equally by the first frame
Numerical value, if not the first two field picture, be directly assigned to present frame；
B) medium filtering is carried out to present frame, and marginal information is obtained using Sobel operator；
C) process the enhanced image in edge, constant to the gray value of the point being more than specific threshold in image, less than feature threshold value
Point be assigned to zero, piecemeal process is carried out to current region, carries out decile N block for 768 × 576, select suitable Block- matching area
Domain, more rich to scene information, obtain higher weight, in image, marginal information amount is low, obtain relatively low weight；
D) by the angular displacement of MEMS microthrust test (106) and the calculating of angular acceleration, obtaining the direction of motion and probably of image
Side-play amount；
E) by being calculated the side-play amount of the relative reference frame in each region, global motion arrow is obtained by weighted value correction
Amount；
F) by using global motion vector, motion compensation is carried out to every two field picture, using the compensation method of bilinear interpolation, right
Dither image is corrected, and obtains stable video image；
G) after obtaining stable image sequence, the information of object observing.
4. a kind of method to the range finding of Small object quick lock according to claim 3 is it is characterised in that described step
D) and e) Small Target quick lock in distance-finding method concrete steps comprise the steps：
A) first target is observed, find suspicious object；
B) select the size of suitable target lock-on frame, target is locked, the information of target is stored, for mesh
Target recognition and tracking；
C after) finding target, laser range finder is charged, when discovery suspicious object passes through linux system embedded signal panel
Laser range finder is sent with charging process instruction；
D) in next two field picture, by the departure of MEMS microthrust test (106) and steady picture, carry out the prediction of target, to target
Carry out correlation tracking, the position of interpretation Small object；
E) pass through target position in the picture, provide the positional information that target deviates laser center axle, positional information is superimposed
On the image of display；
F) pass through target from the side-play amount of optical axis center, be adjusted round about, carry out relevant matches in real time, obtain target
Deviate the deviation of optical axis, when target reaches the scope that can hit, automatically launch laser；
G) close whole object ranging process, close During laser emission.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410426808.6A CN104155006B (en) | 2014-08-27 | 2014-08-27 | A kind of hand-held thermal infrared imager and its method to the range finding of Small object quick lock in |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410426808.6A CN104155006B (en) | 2014-08-27 | 2014-08-27 | A kind of hand-held thermal infrared imager and its method to the range finding of Small object quick lock in |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104155006A CN104155006A (en) | 2014-11-19 |
| CN104155006B true CN104155006B (en) | 2017-03-01 |
Family
ID=51880569
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410426808.6A Active CN104155006B (en) | 2014-08-27 | 2014-08-27 | A kind of hand-held thermal infrared imager and its method to the range finding of Small object quick lock in |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104155006B (en) |
Families Citing this family (39)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104535186B (en) * | 2014-12-30 | 2016-08-17 | 华中科技大学 | A kind of moving platform infared spectrum association detection system and method |
| CN104808256A (en) * | 2015-04-02 | 2015-07-29 | 苏州华徕光电仪器有限公司 | Day and night universal photoelectric detection system based on graphene solar energy power source |
| CN104883481A (en) * | 2015-05-11 | 2015-09-02 | 杭州奕霖传感科技有限公司 | Double-spectrum infrared thermal imaging ball machine |
| CN105115431A (en) * | 2015-06-02 | 2015-12-02 | 绵阳东福高科技有限公司 | Novel CCD laser displacement sensor paper sheet thickness meter |
| CN105120129B (en) * | 2015-08-05 | 2018-01-30 | 西安电子科技大学 | The Infrared DIM-small Target Image sequence emulation mode of infrared area array cameras shake |
| CN105222999B (en) * | 2015-11-09 | 2017-07-25 | 北京理工大学 | A kind of portable visible ray/infrared double-waveband dynamic scene analogue means |
| CN105676884B (en) * | 2016-01-27 | 2018-09-25 | 武汉天木林科技有限公司 | A kind of device and method that infrared thermal imaging search and track aims at |
| CN105716721B (en) * | 2016-02-22 | 2018-09-11 | 国网江苏省电力公司电力科学研究院 | A kind of infrared temperature accuracy of detection bearing calibration |
| CN107543578A (en) * | 2016-06-24 | 2018-01-05 | 珠海多监测科技有限公司 | A kind of Multifunctional hand-held instrument |
| DE102016218295A1 (en) * | 2016-09-23 | 2018-03-29 | Robert Bosch Gmbh | Method for non-contact determination of a two-dimensional temperature information and infrared measurement system |
| CN108088565A (en) * | 2016-11-23 | 2018-05-29 | 杭州海康威视数字技术股份有限公司 | Transmit the method and device of scene of fire image, hand-held fire-fighting thermal imaging system |
| CN106647495A (en) * | 2016-12-23 | 2017-05-10 | 天津七六四通信导航技术有限公司 | Ultra-high-frequency range finder control system and realization method thereof |
| CN107095384B (en) * | 2017-04-26 | 2023-11-24 | 左志权 | Intelligent fire control helmet device based on WIFI transmission |
| CN107295230A (en) * | 2017-07-14 | 2017-10-24 | 广州紫川电子科技有限公司 | A kind of miniature object movement detection device and method based on thermal infrared imager |
| CN107728160B (en) * | 2017-08-21 | 2021-03-05 | 广州紫川电子科技有限公司 | Thermal infrared imager-based visual radar detection method and system |
| DE102017215783A1 (en) * | 2017-09-07 | 2019-03-07 | Robert Bosch Gmbh | Method for operating a laser rangefinder |
| CN107941347A (en) * | 2017-11-23 | 2018-04-20 | 广东天圣高科股份有限公司 | A kind of combined type temperature testing device |
| CN108307130A (en) * | 2018-01-09 | 2018-07-20 | 中国地质大学（武汉） | A kind of collecting method and system of infrared thermal imaging device |
| CN110413173B (en) * | 2018-04-28 | 2022-09-27 | 中国科学院沈阳自动化研究所 | Man-machine interaction device of infrared imaging tracking system |
| CN108839018B (en) * | 2018-06-25 | 2021-08-24 | 盐城工学院 | Robot control operation method and device |
| CN109060146B (en) * | 2018-06-28 | 2020-04-21 | 湖北久之洋红外系统股份有限公司 | Scanning galvanometer control system for area array infrared wide area imaging |
| TWI690718B (en) * | 2018-08-02 | 2020-04-11 | 李宏富 | Target position tracking system |
| CN109269675A (en) * | 2018-10-29 | 2019-01-25 | 深圳市度彼电子有限公司 | Temperature-measuring gun |
| CN109245823B (en) * | 2018-11-12 | 2020-08-18 | 维沃移动通信（杭州）有限公司 | Time of flight TOF module and mobile terminal |
| CN109859178B (en) * | 2019-01-18 | 2020-11-03 | 北京航空航天大学 | FPGA-based infrared remote sensing image real-time target detection method |
| CN110018485A (en) * | 2019-05-15 | 2019-07-16 | 中国计量大学 | A kind of unmanned plane remains hunting system and method based on polarization imaging and infrared distance measurement |
| CN110796655B (en) * | 2019-11-01 | 2023-09-12 | 南京申威光电技术研究院有限公司 | Photoelectric detection method and photoelectric detection device |
| CN112824933A (en) * | 2019-11-19 | 2021-05-21 | 北京小米移动软件有限公司 | Distance measuring method, distance measuring device and electronic equipment |
| CN110913116A (en) * | 2019-12-31 | 2020-03-24 | 黑龙江大学 | Small-sized infrared visible light multifunctional fusion camera |
| CN111369592B (en) * | 2020-03-13 | 2023-07-25 | 浙江工业大学 | Newton interpolation-based rapid global motion estimation method |
| CN112013963A (en) * | 2020-07-16 | 2020-12-01 | 南京莱斯电子设备有限公司 | Portable photoelectric information processing platform |
| CN112147639A (en) * | 2020-07-17 | 2020-12-29 | 中国工程物理研究院应用电子学研究所 | MEMS one-dimensional laser radar and digital camera surveying and mapping device and method |
| CN112197868B (en) * | 2020-10-12 | 2022-09-30 | 上海热芯视觉科技有限公司 | FPGA-based multi-sensor identification red thermal imager system and method |
| CN112565683B (en) * | 2020-11-19 | 2023-03-24 | 湖南宇正智能科技有限公司 | Portable photoelectric heel searching system and method |
| CN113433557B (en) * | 2021-01-29 | 2024-04-23 | 安徽科技学院 | Laser for field search and rescue |
| CN113608232B (en) * | 2021-06-30 | 2024-06-04 | 湖南宏动光电有限公司 | Laser dynamic scanning ranging method and system |
| CN114199464A (en) * | 2021-12-09 | 2022-03-18 | 湖北久之洋信息科技有限公司 | SF6 gas leakage detection handheld equipment for realizing video display control |
| CN114862828B (en) * | 2022-05-30 | 2024-08-23 | Oppo广东移动通信有限公司 | Light spot searching method device, computer readable medium and electronic equipment |
| CN116566358B (en) * | 2023-05-19 | 2024-07-12 | 西安现代控制技术研究所 | Hypersonic rocket bomb filtering method |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102128683A (en) * | 2010-12-29 | 2011-07-20 | 南京武大卓越科技有限公司 | Wireless intelligent thermodetector |
| CN202793593U (en) * | 2011-09-29 | 2013-03-13 | 中国人民解放军海军兵种指挥学院训练部 | Infrared thermal imaging device |
| US9228838B2 (en) * | 2011-12-20 | 2016-01-05 | Fluke Corporation | Thermal imaging camera with compass calibration |
| EP2974277A1 (en) * | 2012-11-30 | 2016-01-20 | Robert Bosch GmbH | Methods to combine radiation-based temperature sensor and inertial sensor and/or camera output in a handheld/mobile device |
| CN103278246B (en) * | 2013-05-03 | 2016-08-10 | 广州飒特红外股份有限公司 | A kind of thermal infrared imager based on Android system |
-
2014
- 2014-08-27 CN CN201410426808.6A patent/CN104155006B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN104155006A (en) | 2014-11-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104155006B (en) | A kind of hand-held thermal infrared imager and its method to the range finding of Small object quick lock in | |
| CN106802658B (en) | Method for rapidly positioning in a kind of full-automatic high precision room | |
| US10242454B2 (en) | System for depth data filtering based on amplitude energy values | |
| CN105940429B (en) | For determining the method and system of the estimation of equipment moving | |
| CN109737981B (en) | Unmanned vehicle target searching device and method based on multiple sensors | |
| CN109212545A (en) | Multiple source target following measuring system and tracking based on active vision | |
| CN103017730A (en) | Single-camera ranging method and single-camera ranging system | |
| CN106605154B (en) | A kind of monitoring method of moving target, wearable device and server | |
| CN107122770A (en) | Many mesh camera systems, intelligent driving system, automobile, method and storage medium | |
| CN110370273B (en) | Robot obstacle avoidance method, device and system | |
| CN101922894A (en) | Anti-sniper laser active detection system and method | |
| CN112629519B (en) | Target positioning handheld observer and navigation method thereof | |
| CN110398258A (en) | A kind of performance testing device and method of inertial navigation system | |
| CN108734655A (en) | The method and system that aerial multinode is investigated in real time | |
| EP3005238A1 (en) | Method and system for coordinating between image sensors | |
| CN112991439A (en) | Method, apparatus, electronic device, and medium for positioning target object | |
| JP2002199382A (en) | Moving image processing camera and image processing system using the camera | |
| CN111103899A (en) | Holder positioning method and device | |
| CN113743286A (en) | Target monitoring system and method for multi-source signal fusion | |
| CN111652276A (en) | All-weather portable multifunctional bionic positioning, attitude determining and viewing system and method | |
| CN110413836A (en) | A kind of panorama search system | |
| CN106156764B (en) | Realize the optical tracking system and its control method of high speed tracking | |
| RU108136U1 (en) | COMPLEX OF PANORAMIC VIDEO SURVEILLANCE AND TERRITORY CONTROL | |
| KR101856151B1 (en) | An portable apparatus for gathering indoor positioning infra information | |
| CN109211185A (en) | A kind of flight equipment, the method and device for obtaining location information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |