CN110223223A - Street scan method, device and scanner - Google Patents
Street scan method, device and scanner Download PDFInfo
- Publication number
- CN110223223A CN110223223A CN201910349750.2A CN201910349750A CN110223223A CN 110223223 A CN110223223 A CN 110223223A CN 201910349750 A CN201910349750 A CN 201910349750A CN 110223223 A CN110223223 A CN 110223223A
- Authority
- CN
- China
- Prior art keywords
- data
- point cloud
- street
- cloud data
- image data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 96
- 230000008569 process Effects 0.000 claims abstract description 51
- 230000001360 synchronised effect Effects 0.000 claims abstract description 36
- 230000004927 fusion Effects 0.000 claims abstract description 26
- 230000002123 temporal effect Effects 0.000 claims abstract description 16
- 238000000605 extraction Methods 0.000 claims description 16
- 238000012216 screening Methods 0.000 claims description 12
- 238000013139 quantization Methods 0.000 claims description 8
- 238000010030 laminating Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 abstract description 22
- 238000005259 measurement Methods 0.000 abstract description 10
- 238000003860 storage Methods 0.000 description 12
- 238000004590 computer program Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 9
- 238000012545 processing Methods 0.000 description 9
- 239000000284 extract Substances 0.000 description 8
- 230000006870 function Effects 0.000 description 7
- 230000008901 benefit Effects 0.000 description 4
- 238000013480 data collection Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000007423 decrease Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000009825 accumulation Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000001052 transient effect Effects 0.000 description 2
- 206010039203 Road traffic accident Diseases 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000002844 melting Methods 0.000 description 1
- 230000008018 melting Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000010408 sweeping Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/176—Urban or other man-made structures
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/04—Synchronising
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Electromagnetism (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Traffic Control Systems (AREA)
Abstract
The embodiment of the invention discloses a kind of street scan method, device and scanners, are related to technical field of mapping, and main purpose is to solve the problems, such as that measurement efficiency is low in existing street scanning process.Main technical schemes of the embodiment of the present invention are as follows: obtain point cloud data, image data and location parameter, the point cloud data, image data and location parameter are respectively the data obtained after the street moving process Zhong Dui is scanned；The point cloud data is synchronized with image data according to temporal information；By the point cloud data after synchronizing and rear image data is synchronized, is merged according to corresponding location parameter, fused image information is obtained, described image information is the street image that fusion has point cloud data and image data.The embodiment of the present invention is mainly used for avenue and is scanned and surveys and draws.
Description
Technical field
The method, apparatus and scanner scanned the present embodiments relate to technical field of mapping more particularly to a kind of street.
Background technique
With being constantly progressive for science and technology, smart city is the trend of current era development, wherein in smart city structure
During building, for accurate location information, and intuitive Map Expression is to realize the digitized important link in smart city,
Therefore in the building process for realizing digitized city map, street scanning has in terms of mapping, Urban renewal, urban planning
Significance.
Avenue mapping at present is mainly surveyed and drawn by mapping type laser radar, is being scanned and is being surveyed to street
By the chosen position in street during drawing, and sweeping for target area is carried out in this location arrangements mapping type laser radar
It retouches and surveys and draws.However, in practical applications, in existing measurement process, such mapping type laser radar is arranged in tripod
On, do not have locomotivity when in use, manpower is needed to be carried, influences to survey and draw efficiency, and survey the street of large area
It when amount, needs that scanning element is arranged in different positions, by the setting of mapping type laser radar at the scanning element, repeatedly be surveyed
Amount, then multiple measurement is spliced again, time loss is more.Therefore, exist during existing avenue measurement and survey
Draw the lower problem of efficiency.
Summary of the invention
It is to mention in view of this, the embodiment of the present invention provides a kind of street scan method, device and scanner, main purpose
Measurement efficiency in high avenue mapping process solves the problems, such as that measurement efficiency is low in existing street scanning process.
In order to achieve the above object, the embodiment of the present invention mainly provides the following technical solutions:
In a first aspect, the embodiment of the invention provides a kind of street scanners, comprising: data acquisition components, data processing
Component and mobile vehicle, wherein
The data acquisition components are set at the top of the mobile vehicle, for acquire point cloud data, image data and
Location parameter；
The data acquisition components are connected with the data handling component.
Optionally, the data acquisition components include: laser radar, panorama camera, Differential Global Positioning System and inertia
Sensor and synchronous mould group；
Wherein, the laser radar is for acquiring point cloud data, and the panorama camera is for acquiring image data, the difference
Divide global positioning system, the Differential Global Positioning System and inertial sensor are used for acquisition position and posture information, described to swash
Optical radar, panorama camera, Differential Global Positioning System are connected with the synchronous mould group respectively with inertial sensor, the synchronous mould
Group is for synchronous with the position and posture information respectively by the point cloud data, image data.
Second aspect, the embodiment of the invention provides a kind of street scan method, this method is based on described in first aspect
Scanner realize this method comprises:
Obtain point cloud data, image data and location parameter, the point cloud data, image data and location parameter point
Not Wei the street moving process Zhong Dui be scanned after obtained data；
The point cloud data is synchronized with image data according to temporal information, the temporal information is in a cloud number
It is determined simultaneously according to, image data in acquisition；
By the point cloud data after synchronizing and rear image data is synchronized, is merged, is melted according to corresponding location parameter
Image information after conjunction, described image information are the street image that fusion has point cloud data and image data.
Optionally, the point cloud data is acquired by laser radar；
Described image data are acquired by panorama camera；
The location parameter is specially position and posture information, and the position and posture information are positioned by difference global
What system and inertial sensor acquired.
Optionally, in image data by the point cloud data after synchronizing and after synchronizing, melted according to corresponding location parameter
It closes, after obtaining fused image information, which comprises
The fused image information is subjected to vectoring operations, obtains vectorial images, the vectorial images are
To eliminating the lines image for obtaining street after lines extract after non-targeted object in street.
Optionally, described that the fused image information is subjected to vectoring operations, obtain vectorial images, comprising:
The fused image information is subjected to feature extraction；
The identification of object in street is carried out to the fused image information according to the feature of extraction；
According to default screening conditions, the object in the street is classified according to target object and non-targeted object,
Include in the default screening conditions user setting target object and corresponding feature；
The non-targeted object is deleted from the fused image information, and extract obtain after lines it is described
Vectorial images.
Optionally, described by the point cloud data after synchronizing and to synchronize rear image data, it is carried out according to corresponding location parameter
Fusion, obtaining fused image information includes:
Image data according to the point cloud data after synchronizing and after synchronizing is overlapped, and obtains the list of corresponding different moments
Frame image；
The relativeness between single-frame images is determined according to the location information in location parameter, and according to the relativeness
Whole single-frame images are spliced, obtain include point cloud data and image data the fused image information.
The third aspect, the embodiment of the invention also provides a kind of street scanning means, which includes:
Acquiring unit, for obtaining point cloud data, image data and location parameter, the point cloud data, image data
And location parameter is respectively the data obtained after the street moving process Zhong Dui is scanned；
Synchronization unit, for synchronizing the point cloud data with image data according to temporal information；
Integrated unit, for by the point cloud data after synchronizing and synchronizing rear image data, according to corresponding location parameter into
Row fusion, obtains fused image information, and described image information is the street image that fusion has point cloud data and image data.
Optionally, the point cloud data is acquired by laser radar；
Described image data are acquired by panorama camera；
The location parameter is specially position and posture information, and the position and posture information are positioned by difference global
What system and inertial sensor acquired.
Optionally, described device further include:
Vectoring operations unit obtains vector quantization figure for the fused image information to be carried out vectoring operations
Picture, the vectorial images are to eliminating the lines figure for obtaining street after lines extract after non-targeted object in street
Picture.
Optionally, the vectoring operations unit includes:
Extraction module, for the fused image information to be carried out feature extraction；
Identification module carries out the knowledge of object in street for the feature according to extraction to the fused image information
Not；
Categorization module, for according to screening conditions are preset, by the object in the street according to target object and non-targeted
Object is classified, include in the default screening conditions user setting target object and corresponding feature；
Vector quantization module, for the non-targeted object to be deleted from the fused image information, line of going forward side by side
Item extracts the vectorial images.
Optionally, the integrated unit, comprising:
Laminating module is overlapped for the image data according to the point cloud data after synchronizing and after synchronizing, is corresponded to
The single-frame images of different moments；
Splicing module is obtained for the point cloud data to be overlapped and splice with image data comprising there is a cloud number
According to the image information in the fused entire street with image data.
A kind of street scan method, device and scanner provided in an embodiment of the present invention, with existing street mapping process
Middle have mapping efficiency be lower compares, the embodiment of the present invention can first by obtaining point cloud data, image data with
And location parameter, then the point cloud data is synchronized with image data according to temporal information, the point after finally synchronizing
Cloud data and rear image data is synchronized, is merged according to corresponding location parameter, obtain fused image information.Wherein,
The point cloud data, image data and location parameter are respectively the data obtained after the street moving process Zhong Dui is scanned,
To realize in moving process carry out street scanning realization, avoid it is existing during need every time choose deployment position
It sets, and the process of time loss caused when mobile mapping type laser radar, so that scan efficiency is improved, meanwhile, pass through
It is scanned in moving process, and location parameter and accessed point cloud data, image data is synchronized, can moved
It is realized when moving to the function of scanning and survey and draw in street, can then be swept to avoid needing to be arranged every time in existing mapping process
The process that image scanned at position or acquisition carries out subsequent splicing is retouched, improves scan efficiency on the whole.In addition, being based on
The fusion that the fusion of point cloud data, image data can be realized to point cloud data collection image data is realized in scanning process,
Image data to make street scan is more fully.
Above description is only the general introduction of technical solution of the embodiment of the present invention, in order to better understand the embodiment of the present invention
Technological means, and can be implemented in accordance with the contents of the specification, and in order to allow above and other mesh of the embodiment of the present invention
, feature and advantage can be more clearly understood, the special specific embodiment for lifting the embodiment of the present invention below.
Detailed description of the invention
By reading the following detailed description of the preferred embodiment, various other advantages and benefits are common for this field
Technical staff will become clear.The drawings are only for the purpose of illustrating a preferred embodiment, and is not considered as to the present invention
The limitation of embodiment.And throughout the drawings, the same reference numbers will be used to refer to the same parts.In the accompanying drawings:
Fig. 1 shows a kind of street scanner provided in an embodiment of the present invention；
Fig. 2 shows a kind of street scan method flow charts provided in an embodiment of the present invention；
Fig. 3 shows another street scan method flow chart provided in an embodiment of the present invention；
Fig. 4 shows a kind of composition block diagram of street scanning means provided in an embodiment of the present invention；
Fig. 5 shows the composition block diagram of another street scanning means provided in an embodiment of the present invention.
Specific embodiment
The exemplary embodiment that the present invention will be described in more detail below with reference to accompanying drawings.Although showing the present invention in attached drawing
Exemplary embodiment, it being understood, however, that may be realized in various forms the present invention without should be by embodiments set forth here
It is limited.It is to be able to thoroughly understand the present invention on the contrary, providing these embodiments, and can be by the scope of the present invention
It is fully disclosed to those skilled in the art.
The embodiment of the present invention provides a kind of street scanner first, as shown in Figure 1, comprising:
Data acquisition components 11, data handling component 12 and mobile vehicle 13,
Wherein,
In embodiments of the present invention, the data acquisition components 11 are set to 13 top of mobile vehicle, for acquiring
Point cloud data, image data and location parameter.
The data acquisition components 11 are connected with the data handling component 12, and the data handling component 12 is used for institute
The data for stating the output of acquisition component 11 are handled；The data handling component 12 is set in the mobile vehicle 13, is needed
Illustrate, in embodiments of the present invention, the data handling component 12 can be shown in Fig. 1, to be arranged in mobile vehicle 13,
It is, of course, also possible to by data handling component setting and other regions, for example, the data handling component is set to remote machine
Fang Zhong is remotely connect between data acquisition components 11, here, set-up mode described in the embodiment of the present invention, is example
Property, the specific position that is arranged can be chosen according to actual needs.
In this way, can be moved by its mobile vehicle based on the street scanner, then by being carried out in moving process
Scanning, solves in existing scanning process and manpower is needed to be carried, and influences to survey and draw efficiency, and for the street of large area
When measurement, is measured it is not necessary that scanning element is arranged in multiple positions, then will repeatedly measure the process spliced again, improve survey
Amount efficiency.
Wherein, in above-mentioned street scanner, the data acquisition components 11 include: laser radar 111, panorama camera
112, Differential Global Positioning System and inertial sensor 113 and synchronous mould group 114；
Wherein, the laser radar 111 is used to acquire image data for acquiring point cloud data, the panorama camera 112,
The Differential Global Positioning System, the Differential Global Positioning System and inertial sensor 113 are believed for acquisition position and posture
Breath, the laser radar 111, panorama camera 112, Differential Global Positioning System and inertial sensor 113 respectively with it is described synchronous
Mould group 114 is connected, and the synchronous mould group is used for the point cloud data, image data is same with the position and posture information respectively
Step.In addition, Differential Global Positioning System and inertial sensor 113 are additionally provided with antenna, connected with will pass through antenna and external satellite
It connects.
Further, the embodiment of the present invention also provides a kind of street scan method, as shown in Figure 2, which comprises
201, point cloud data, image data and location parameter are obtained.
Wherein, the point cloud data, image data and location parameter are respectively that the street moving process Zhong Dui is scanned
The data obtained afterwards.
It in embodiments of the present invention, when carrying out street scanning, is realized based on scanner described in previous embodiment.
Specifically, its point cloud data can be acquired by laser radar, and image data can then pass through default camera acquisition, position
Setting parameter then can be by locating module, for example, global positioning system.Specifically, its acquisition modes may include but be not limited to
Above-mentioned any number of mode carries out, it is not limited here.
202, the point cloud data and image data are synchronized according to temporal information.
Wherein, the temporal information is when point cloud data, image data are in acquisition while to determine.When abovementioned steps obtain
It based on above-mentioned data is obtained in moving process after having got point cloud data, image data and location parameter, because
This, in order to carry out the generation of street image, it is also necessary to above-mentioned data be synchronized, that is, determine the point cloud obtained in the same time
Data, image data and corresponding relationship, in addition, a in the actual operation process, adopting based on point cloud data and image data
Acquisition means can have distance on the actual position, i.e., distinct between coordinate system of the two between the data of acquisition, be
Ensure accuracy when subsequent fusion, in embodiments of the present invention can also obtain point cloud data collection image data it
Before, the coordinate system of the acquisition device of the two is initialized, is located at when so that the point cloud data is with image data
In same coordinate system, so that it is guaranteed that accuracy.
203, by the point cloud data after synchronizing and rear image data is synchronized, is merged, is obtained according to corresponding location parameter
To fused image information.
Wherein, described image information is the street image that fusion has point cloud data and image data.
After point cloud data, image data are synchronized in abovementioned steps 202, point cloud data after being synchronized and
Synchronize rear image data.In order to both data are merged, obtain include point cloud data and image data road map
Picture.It then by the point cloud data after synchronizing and rear image data can be synchronized merges according to the method for this step, wherein melting
The process of conjunction can carry out the fusion of single-frame images by the position coordinates between data, and when carrying out the image co-registration of multiframe
Multiple single-frame images can then be spliced according to positional relationship according to location parameter.
Certainly, it is referred in its specific fusion but is not limited to following manner progress: determining each moment point cloud first
Corresponding relationship and the corresponding location parameter of each moment image data between the corresponding location parameter of data, and according to position
Parameter splices the point cloud data of synchronization with image data, to obtain corresponding street scan image.
A kind of street scan method provided in an embodiment of the present invention surveys and draws efficiency with existing in existing street mapping process
Lower problem is compared, and the embodiment of the present invention can be first by obtaining point cloud data, image data and location parameter, then
The point cloud data is synchronized with image data according to temporal information, is finally schemed by the point cloud data after synchronizing and after synchronizing
As data, is merged according to corresponding location parameter, obtain fused image information.Wherein, the point cloud data, image
Data and location parameter are respectively the data obtained after the street moving process Zhong Dui is scanned, are being moved through to realize
The realization that street scanning is carried out in journey, avoid it is existing during need to choose deployed position, and mobile mapping type laser every time
The process of caused time loss when radar, so that scan efficiency is improved, meanwhile, by being scanned in moving process,
And synchronize location parameter and accessed point cloud data, image data, it can realize and be swept in street when moving
The function of retouching and survey and draw, then can to avoid needed in existing mapping process will every time setting scan position at scanned or
The image of acquisition carries out the process of subsequent splicing, improves scan efficiency on the whole.In addition, based on realize in scanning process by
The fusion of point cloud data, image data, can be realized the fusion of point cloud data collection image data, so that street be made to scan
Image data is more fully.
It is specific such as Fig. 3 the embodiment of the invention provides another street scan method below in order to be explained in more detail
It is shown, this method comprises:
301, point cloud data, image data and location parameter are obtained.
Wherein, the point cloud data, image data and location parameter are respectively that the street moving process Zhong Dui is scanned
The data obtained afterwards.Specifically, the point cloud data is acquired by laser radar；Described image data are by panorama phase
Machine acquisition；The location parameter is specially position and posture information, and the position and posture information are fixed by difference global
What position system and inertial sensor acquired.
In embodiments of the present invention, the laser radar can use the current common machinery rotating type of intelligent driving automobile
Multi-thread 3D laser radar.This laser radar has 360 ° of horizontal direction field angle, 0.1 °~0.2 ° of horizontal direction angle
The field angle of resolution ratio and 20~40 ° of vertical direction, 0.3~2 ° of vertical direction angular resolution and centimetres
Range measurement accuracy can depict the accurate 3D point cloud data of scene around.
The panorama camera can carry out field stitching using the industrial camera of four millions pixels and obtain, at present
It is widely used in streetscape acquisition.This panorama camera also have 360 ° horizontal direction field angle and 270 °~
The field angle of 300 ° of vertical direction, the high definition RGB image in available street.
(Global Navigation Satellite System, full name are for the Differential Global Positioning System, i.e. GNSS
Global Navigation Satellite System, guide number SS) system uses difference modes to obtain high-precision location information.But in complexity
Dynamic environment in, it is especially very significant the big city, Multipath reflection the problem of, cause the GNSS location information obtained to be held very much
It is also easy to produce several meters of error.In addition, since the renewal frequency of GNSS is low (renewal frequency 10Hz), very in vehicle fast running
Difficulty provides accurately positioning in real time.The simple navigation for relying on GNSS is likely to result in traffic accident.Therefore, GNSS is usually assisted
With inertial sensor described in the embodiment of the present invention, i.e. INS (Inertial Navigation System, inertial navigation system,
Abbreviation INS) it is used to enhance the precision of positioning.INS is high frequency (1KHz) sensor for detecting acceleration and rotary motion, but INS
The problems such as itself also having deviation accumulation and noise, influences result.By using the sensor fusion techniques based on Kalman filtering,
GNSS and INS data can be merged, in conjunction with GNSS positioning accuracy height and error without accumulation the characteristics of, with the independence of INS and
The advantages of real-time, to obtain higher positioning accuracy.When GPS signal is preferable, the location information of data acquisition unit and
Posture information is subject to the fused data of GNSS and INS.
But in some special screnes, the case where based on being lost for a long time there may be GNSS signal, for example, narrow street
Etc. blocking more serious region, and as time increases, the positioning accuracy of INS can gradually decrease, to make positioning accurate
Degree decline.It is then positioned immediately using laser point cloud at this time with INS and GNSS is built in conjunction with diagram technology SLAM and carries out merging positioning.Its
In, SLAM (simultaneous localization and mapping, synchronous to position and build figure, abbreviation SLAM).It utilizes
As endpoint constraint, intermediate position carries out ground with laser SLAM for position before GNSS differential signal loss and the position after recovery
Scheme building, in order to avoid the cumulative errors problem of SLAM, then it is excellent to SLAM progress using the method for the nonlinear optimizations such as figure optimization
Change.
302, the point cloud data and image data are synchronized according to temporal information.
Wherein, the temporal information is when point cloud data, image data are in acquisition while to determine, is implemented in the present invention
Its implementation procedure can utilize synchronization after when sensor-triggered in the scanner of street during synchronizing in example
Module synchronizes operation, which mainly can produce the trigger signal of panorama camera, for laser radar, panorama phase
Machine, GNSS+INS data carry out millisecond magnitude synchronization, i.e., by this step point cloud data, image data according to the time into
Row simultaneously operating carries out specifically, the method for synchronization can choose existing any number of mode, it is only necessary to guarantee what this step synchronized
As a result under the premise of accuracy, the method for synchronization of selection is not specifically limited.
303, by the point cloud data after synchronizing and rear image data is synchronized, is merged, is obtained according to corresponding location parameter
To fused image information.
Wherein, it can be the fusion for first carrying out single-frame images when being merged, be based on fused multiple single frames figures
As carrying out multiple image splicing.Specifically, by the process of the fusion of single-frame images to multiframe may include: first according to synchronizing after
The corresponding relationship of obtained point cloud data and position and posture information, and the image data and position and posture that are obtained after synchronizing
The corresponding relationship of information, determines point cloud data under same position coordinate system and corresponding image data, and by the two
It is overlapped to obtain single-frame images；Then, by the point cloud data and image data according to the position between the image in location parameter
Relationship is set, multiple single-frame images are spliced, then obtains including the described fused of point cloud data and image data
Image information.In this way, then may insure to guarantee to be carried out point cloud data and image data according to location parameter in fusion
Fusion, it is ensured that the accuracy of fused image.
304, the fused image information is subjected to vectoring operations, obtains vectorial images.
The vectorial images are to obtaining the lines image in street after eliminating non-targeted object in street.Wherein, originally
Step may include: that the fused image information is carried out feature extraction；According to the feature of extraction to described fused
Image information carries out the identification of object in street；According to default screening conditions, by the object in the street according to target object
And non-targeted object is classified, include in the default screening conditions user setting target object and corresponding feature；
The non-targeted object is deleted from the fused image information, and extracts to go out lines extraction after lines and obtain the arrow
Quantized image.Specifically, its implementation procedure can include but is not limited to following manner progress: firstly, in high-precision map
Object correlation is identified and is classified, for example is built, vehicle, trees, road, pedestrian etc..Then, by uninterested data
It rejects, only retains the point cloud of interesting part.Such as, it is intended that the Vector Message in available street needs to build, road
The information of equal fixation means, and to vehicle, trees, the information such as pedestrian are lost interest in, then can be rejected.Then on the one hand may be used in this way
To reduce data volume, the pressure of later data processing is reduced, data processing speed is improved, on the other hand decreases relevant interference
Data are conducive to the characteristic information for extracting needs.In addition, the image data resolution of panorama camera acquisition is high, color and line
The detailed information such as reason are abundant, are easier to extract the features such as relation line, face, in conjunction with the 3D location information of cloud, it can
To relation line, the specific equation in face, i.e., vector quantization equation and shown, obtain corresponding vectorial images, convenient for it is subsequent
It opens and edits in the tools such as CAD.
Further, as the realization to method shown in above-mentioned Fig. 2, Fig. 3, the embodiment of the invention provides a kind of streets to sweep
Imaging apparatus.The Installation practice is corresponding with preceding method embodiment, and to be easy to read, present apparatus embodiment is no longer to preceding method
Detail content in embodiment is repeated one by one, it should be understood that the device in the present embodiment, which can correspond to, realizes aforementioned side
Full content in method embodiment.Specifically as shown in figure 4, the device includes:
Acquiring unit 41 can be used for obtaining point cloud data, image data and location parameter, the point cloud data, figure
As data and location parameter are respectively the data obtained after the street moving process Zhong Dui is scanned；
Synchronization unit 42, the location parameter that can be used for obtaining the acquiring unit 41 respectively with the point cloud data,
Image data synchronizes；
Integrated unit 43, point cloud data after can be used for synchronizing synchronization unit 42 and synchronizes rear image data, according to
Corresponding location parameter is merged, and fused image information is obtained, and described image information is that fusion has point cloud data and figure
As the street image of data.
Further, as shown in figure 5, the point cloud data is acquired by laser radar；
Described image data are acquired by panorama camera；
The location parameter is specially position and posture information, and the position and posture information are positioned by difference global
What system and inertial sensor acquired.
Further, as shown in figure 5, described device further include:
Vectoring operations unit 44 can be used for the fused image information of the integrated unit 43 carrying out vector quantization behaviour
Make, obtain vectorial images, the vectorial images be to eliminated in street after non-targeted object after lines extract
To the lines image in street.
Further, as shown in figure 5, the vectoring operations unit 44 includes:
Extraction module 441 can be used for the fused image information carrying out feature extraction；
Identification module 442, can be used for according to extraction module 441 extract feature to the fused image information into
The identification of object in row street；
Categorization module 443 can be used for according to default screening conditions, in the street that the identification module 442 is identified
Object classify according to target object and non-targeted object, include the target of user setting in the default screening conditions
Object and corresponding feature；
Vector quantization module 444 can be used for the sorted non-targeted object of the categorization module 443 after the fusion
Image information in delete, and extract and go out the vectorial images after lines.
Further, as shown in figure 5, the integrated unit 43, comprising:
Laminating module 431 can be used for the image data according to the point cloud data after synchronizing and after synchronizing and be overlapped,
Obtain the single-frame images of corresponding different moments；
Splicing module 432 can be used for determining the opposite pass between single-frame images according to the location information in location parameter
System, and splices whole single-frame images according to the relativeness, obtain include point cloud data and image data institute
State fused image information.
A kind of street scan method, device and scanner provided in an embodiment of the present invention, with existing street mapping process
Middle have mapping efficiency be lower compares, the embodiment of the present invention can first by obtaining point cloud data, image data with
And location parameter, then the point cloud data is synchronized with image data according to temporal information, the point after finally synchronizing
Cloud data and rear image data is synchronized, is merged according to corresponding location parameter, obtain fused image information.Wherein,
The point cloud data, image data and location parameter are respectively the data obtained after the street moving process Zhong Dui is scanned,
To realize in moving process carry out street scanning realization, avoid it is existing during need every time choose deployment position
It sets, and the process of time loss caused when mobile mapping type laser radar, so that scan efficiency is improved, meanwhile, pass through
It is scanned in moving process, and location parameter and accessed point cloud data, image data is synchronized, can moved
It is realized when moving to the function of scanning and survey and draw in street, can then be swept to avoid needing to be arranged every time in existing mapping process
The process that image scanned at position or acquisition carries out subsequent splicing is retouched, improves scan efficiency on the whole.In addition, being based on
The fusion that the fusion of point cloud data, image data can be realized to point cloud data collection image data is realized in scanning process,
Image data to make street scan is more fully.
The street scanning means includes processor and memory, above-mentioned acquiring unit, synchronization unit and integrated unit
Deng be used as program unit storage in memory, above procedure unit stored in memory is executed by processor to realize
Corresponding function.
Include kernel in processor, is gone in memory to transfer corresponding program unit by kernel.Kernel can be set one
Or more, it solves the problems, such as that measurement efficiency is low in existing street scanning process by adjusting kernel parameter, improves city street
Measurement efficiency in road mapping process,.
Memory may include the non-volatile memory in computer-readable medium, random access memory (RAM) and/
Or the forms such as Nonvolatile memory, if read-only memory (ROM) or flash memory (flash RAM), memory include that at least one is deposited
Store up chip.
The embodiment of the invention provides a kind of non-transient computer readable storage medium, the non-transient computer is readable to be deposited
Storage media stores computer instruction, and the computer instruction makes the computer execute street scanning side described in above-described embodiment
Method.
Present invention also provides a kind of computer program products, when executing on data processing equipment, are adapted for carrying out just
The program code of beginningization there are as below methods step: obtaining point cloud data, image data and location parameter, the point cloud data,
Image data and location parameter are respectively the data obtained after the street moving process Zhong Dui is scanned；It will according to temporal information
The point cloud data is synchronized with image data；By the point cloud data after synchronizing and rear image data is synchronized, according to corresponding
Location parameter is merged, and fused image information is obtained, and described image information is that fusion has point cloud data and image data
Street image.
It should be understood by those skilled in the art that, embodiments herein can provide as method, system or computer program
Product.Therefore, complete hardware embodiment, complete software embodiment or reality combining software and hardware aspects can be used in the application
Apply the form of example.Moreover, it wherein includes the computer of computer usable program code that the application, which can be used in one or more,
The computer program implemented in usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) produces
The form of product.
The application is referring to method, the process of equipment (system) and computer program product according to the embodiment of the present application
Figure and/or block diagram describe.It should be understood that every one stream in flowchart and/or the block diagram can be realized by computer program instructions
The combination of process and/or box in journey and/or box and flowchart and/or the block diagram.It can provide these computer programs
Instruct the processor of general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices to produce
A raw machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute for real
The device for the function of being specified in present one or more flows of the flowchart and/or one or more blocks of the block diagram.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates,
Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or
The function of being specified in multiple boxes.
These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting
Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or
The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one
The step of function of being specified in a box or multiple boxes.
In a typical configuration, calculating equipment includes one or more processors (CPU), input/output interface, net
Network interface and memory.
Memory may include the non-volatile memory in computer-readable medium, random access memory (RAM) and/
Or the forms such as Nonvolatile memory, such as read-only memory (ROM) or flash memory (flash RAM).Memory is computer-readable Jie
The example of matter.
Computer-readable medium includes permanent and non-permanent, removable and non-removable media can be by any method
Or technology come realize information store.Information can be computer readable instructions, data structure, the module of program or other data.
The example of the storage medium of computer includes, but are not limited to phase change memory (PRAM), static random access memory (SRAM), moves
State random access memory (DRAM), other kinds of random access memory (RAM), read-only memory (ROM), electric erasable
Programmable read only memory (EEPROM), flash memory or other memory techniques, read-only disc read only memory (CD-ROM) (CD-ROM),
Digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape magnetic disk storage or other magnetic storage devices
Or any other non-transmission medium, can be used for storage can be accessed by a computing device information.As defined in this article, it calculates
Machine readable medium does not include temporary computer readable media (transitory media), such as the data-signal and carrier wave of modulation.
It should also be noted that, the terms "include", "comprise" or its any other variant are intended to nonexcludability
It include so that the process, method, commodity or the equipment that include a series of elements not only include those elements, but also to wrap
Include other elements that are not explicitly listed, or further include for this process, method, commodity or equipment intrinsic want
Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including element
There is also other identical elements in process, method, commodity or equipment.
It will be understood by those skilled in the art that embodiments herein can provide as method, system or computer program product.
Therefore, complete hardware embodiment, complete software embodiment or embodiment combining software and hardware aspects can be used in the application
Form.It is deposited moreover, the application can be used to can be used in the computer that one or more wherein includes computer usable program code
The shape for the computer program product implemented on storage media (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.)
Formula.
The above is only embodiments herein, are not intended to limit this application.To those skilled in the art,
Various changes and changes are possible in this application.It is all within the spirit and principles of the present application made by any modification, equivalent replacement,
Improve etc., it should be included within the scope of the claims of this application.
Claims (12)
1. a kind of street scanner characterized by comprising data acquisition components, data handling component and mobile vehicle,
In,
The data acquisition components are set at the top of the mobile vehicle, for acquiring point cloud data, image data and position
Parameter；
The data acquisition components are connected with the data handling component, and the data handling component is used for the acquisition component
The data of output are handled.
2. scanner according to claim 1, which is characterized in that the data acquisition components include: laser radar, panorama
Camera, Differential Global Positioning System and inertial sensor and synchronous mould group；
Wherein, the laser radar is for acquiring point cloud data, and for the panorama camera for acquiring image data, the difference is complete
Ball positioning system and inertial sensor are used for acquisition position and posture information, and the laser radar, panorama camera, difference global are fixed
Position system is connected with the synchronous mould group respectively with inertial sensor, and the synchronous mould group is used for the point cloud data, image
Data are synchronous with the position and posture information respectively.
3. a kind of street scan method, the method is realized based on street scanner such as claimed in claims 1-2, special
Sign is, comprising:
Point cloud data is obtained, image data and location parameter, the point cloud data, image data and location parameter are respectively
The data that the street moving process Zhong Dui obtains after being scanned；
The point cloud data is synchronized with image data according to temporal information, the temporal information is in point cloud data, figure
As data determine simultaneously in acquisition；
By the point cloud data after synchronizing and rear image data is synchronized, is merged according to corresponding location parameter, after obtaining fusion
Image information, described image information is that fusion has the street image of point cloud data and image data.
4. according to the method described in claim 3, it is characterized in that,
The point cloud data is acquired by laser radar；
Described image data are acquired by panorama camera；
The location parameter is specially position and posture information, and the position and posture information are to pass through Differential Global Positioning System
And inertial sensor acquisition.
5. according to the method described in claim 4, it is characterized in that, in picture number by the point cloud data after synchronizing and after synchronizing
According to being merged according to corresponding location parameter, after obtaining fused image information, which comprises
The fused image information is subjected to vectoring operations, obtains vectorial images, the vectorial images are to street
The lines image in street is obtained after eliminating non-targeted object in road.
6. according to the method described in claim 5, it is characterized in that, described carry out vector quantization for the fused image information
Operation, obtains vectorial images, comprising:
The fused image information is subjected to feature extraction；
The identification of object in street is carried out to the fused image information according to the feature of extraction；
According to default screening conditions, the object in the street is classified according to target object and non-targeted object, it is described
Preset the target object in screening conditions including user setting and corresponding feature；
The non-targeted object is deleted from the fused image information, and obtains the vector quantization figure after extracting lines
Picture.
7. according to the method described in claim 6, it is characterized in that, described by the point cloud data after synchronizing and synchronize rear picture number
According to being merged according to corresponding location parameter, obtaining fused image information includes:
Image data according to the point cloud data after synchronizing and after synchronizing is overlapped, and obtains the single frames figure of corresponding different moments
Picture；
The relativeness between single-frame images is determined according to the location information in location parameter, and according to the relativeness to complete
Portion's single-frame images is spliced, obtain include point cloud data and image data the fused image information.
8. a kind of street scanning means, which is characterized in that described device includes:
Acquiring unit, for obtaining point cloud data, image data and location parameter, the point cloud data, image data and
Location parameter is respectively the data obtained after the street moving process Zhong Dui is scanned；
Synchronization unit, for synchronizing the point cloud data with image data according to temporal information, the temporal information is
It is determined simultaneously when point cloud data, image data are in acquisition；
Integrated unit is melted for by the point cloud data after synchronizing and synchronizing rear image data according to corresponding location parameter
It closes, obtains fused image information, described image information is the street image that fusion has point cloud data and image data.
9. device according to claim 8, which is characterized in that
The point cloud data is acquired by laser radar；
Described image data are acquired by panorama camera；
The location parameter is specially position and posture information, and the position and posture information are to pass through Differential Global Positioning System
And inertial sensor acquisition.
10. device according to claim 9, which is characterized in that described device further include:
Vectoring operations unit obtains vectorial images, institute for the fused image information to be carried out vectoring operations
Stating vectorial images is to obtaining the lines image in street after eliminating non-targeted object in street.
11. device according to claim 10, which is characterized in that the vectoring operations unit includes:
Extraction module, for the fused image information to be carried out feature extraction；
Identification module carries out the identification of object in street for the feature according to extraction to the fused image information；
Categorization module presets screening conditions for basis, by the object in the street according to target object and non-targeted object
Classify, include in the default screening conditions user setting target object and corresponding feature；
Vector quantization module, for deleting the non-targeted object from the fused image information, and after extracting lines
Obtain the vectorial images.
12. device according to claim 11, which is characterized in that the integrated unit, comprising:
Laminating module is overlapped for the image data according to the point cloud data after synchronizing and after synchronizing, and obtains corresponding difference
The single-frame images at moment；
Splicing module, for determining the relativeness between single-frame images according to the location information in location parameter, and according to institute
Relativeness is stated to splice whole single-frame images, obtain include point cloud data and image data the fused figure
As information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910349750.2A CN110223223A (en) | 2019-04-28 | 2019-04-28 | Street scan method, device and scanner |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910349750.2A CN110223223A (en) | 2019-04-28 | 2019-04-28 | Street scan method, device and scanner |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110223223A true CN110223223A (en) | 2019-09-10 |
Family
ID=67820150
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910349750.2A Pending CN110223223A (en) | 2019-04-28 | 2019-04-28 | Street scan method, device and scanner |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110223223A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110531377A (en) * | 2019-10-08 | 2019-12-03 | 北京邮电大学 | Data processing method, device, electronic equipment and the storage medium of radar system |
| CN110942514A (en) * | 2019-11-26 | 2020-03-31 | 三一重工股份有限公司 | Method, system and device for generating point cloud data and panoramic image |
| CN113678136A (en) * | 2019-12-30 | 2021-11-19 | 深圳元戎启行科技有限公司 | Obstacle detection method and device based on unmanned technology and computer equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140253689A1 (en) * | 2013-03-08 | 2014-09-11 | Kabushiki Kaisha Topcon | Measuring Instrument |
| CN104268935A (en) * | 2014-09-18 | 2015-01-07 | 华南理工大学 | Feature-based airborne laser point cloud and image data fusion system and method |
| CN107121064A (en) * | 2017-04-27 | 2017-09-01 | 上海华测导航技术股份有限公司 | A kind of laser scanner |
| CN107421507A (en) * | 2017-04-28 | 2017-12-01 | 上海华测导航技术股份有限公司 | Streetscape data acquisition measuring method |
| CN109297510A (en) * | 2018-09-27 | 2019-02-01 | 百度在线网络技术（北京）有限公司 | Relative pose scaling method, device, equipment and medium |
-
2019
- 2019-04-28 CN CN201910349750.2A patent/CN110223223A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140253689A1 (en) * | 2013-03-08 | 2014-09-11 | Kabushiki Kaisha Topcon | Measuring Instrument |
| CN104268935A (en) * | 2014-09-18 | 2015-01-07 | 华南理工大学 | Feature-based airborne laser point cloud and image data fusion system and method |
| CN107121064A (en) * | 2017-04-27 | 2017-09-01 | 上海华测导航技术股份有限公司 | A kind of laser scanner |
| CN107421507A (en) * | 2017-04-28 | 2017-12-01 | 上海华测导航技术股份有限公司 | Streetscape data acquisition measuring method |
| CN109297510A (en) * | 2018-09-27 | 2019-02-01 | 百度在线网络技术（北京）有限公司 | Relative pose scaling method, device, equipment and medium |
Non-Patent Citations (1)
| Title |
|---|
| 谢波等: "车载移动激光扫描测绘系统设计", 《压电与声光》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110531377A (en) * | 2019-10-08 | 2019-12-03 | 北京邮电大学 | Data processing method, device, electronic equipment and the storage medium of radar system |
| CN110531377B (en) * | 2019-10-08 | 2022-02-25 | 北京邮电大学 | Data processing method and device of radar system, electronic equipment and storage medium |
| CN110942514A (en) * | 2019-11-26 | 2020-03-31 | 三一重工股份有限公司 | Method, system and device for generating point cloud data and panoramic image |
| CN113678136A (en) * | 2019-12-30 | 2021-11-19 | 深圳元戎启行科技有限公司 | Obstacle detection method and device based on unmanned technology and computer equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Agüera-Vega et al. | Accuracy of digital surface models and orthophotos derived from unmanned aerial vehicle photogrammetry | |
| US8571265B2 (en) | Measurement apparatus, measurement method, and feature identification apparatus | |
| CN111912417B (en) | Map construction method, map construction device, map construction equipment and storage medium | |
| Verhoeven et al. | Computer vision‐based orthophoto mapping of complex archaeological sites: The ancient quarry of Pitaranha (Portugal–Spain) | |
| KR100912715B1 (en) | Method and apparatus of digital photogrammetry by integrated modeling for different types of sensors | |
| CN104280036B (en) | A kind of detection of transport information and localization method, device and electronic equipment | |
| CN113424232A (en) | Three-dimensional point cloud map construction method, system and equipment | |
| Wang et al. | Automated road sign inventory system based on stereo vision and tracking | |
| CN111540048A (en) | Refined real scene three-dimensional modeling method based on air-ground fusion | |
| KR102200299B1 (en) | A system implementing management solution of road facility based on 3D-VR multi-sensor system and a method thereof | |
| JPWO2006114955A1 (en) | Shooting position analysis method | |
| CN110223223A (en) | Street scan method, device and scanner | |
| CN112465970A (en) | Navigation map construction method, device, system, electronic device and storage medium | |
| Wang et al. | Vision-based productivity analysis of cable crane transportation using augmented reality–based synthetic image | |
| CN110986888A (en) | Aerial photography integrated method | |
| Zhao et al. | Updating a digital geographic database using vehicle-borne laser scanners and line cameras | |
| Eugster et al. | Integrated georeferencing of stereo image sequences captured with a stereovision mobile mapping system–approaches and practical results | |
| Zhao et al. | Alignment of continuous video onto 3D point clouds | |
| CN108195359A (en) | The acquisition method and system of spatial data | |
| Koppanyi et al. | Experiences with acquiring highly redundant spatial data to support driverless vehicle technologies | |
| Niskanen et al. | Using a 2D profilometer to determine volume and thickness of stockpiles and ground layers of roads | |
| Thomas et al. | A new software/hardware architecture for real time image processing of wide area airborne camera images | |
| Randeniya et al. | Calibration of inertial and vision systems as a prelude to multi-sensor fusion | |
| Roncella et al. | Photogrammetric bridging of GPS outages in mobile mapping | |
| Lee et al. | Semi-automatic framework for traffic landmark annotation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |