CN110298921A - The construction method and processing equipment of three-dimensional map with personage's semantic information - Google Patents
The construction method and processing equipment of three-dimensional map with personage's semantic information Download PDFInfo
- Publication number
- CN110298921A CN110298921A CN201910602327.9A CN201910602327A CN110298921A CN 110298921 A CN110298921 A CN 110298921A CN 201910602327 A CN201910602327 A CN 201910602327A CN 110298921 A CN110298921 A CN 110298921A
- Authority
- CN
- China
- Prior art keywords
- personage
- information
- processing equipment
- dimensional
- thread
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/05—Geographic models
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Remote Sensing (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Processing Or Creating Images (AREA)
Abstract
This application provides the construction methods and processing equipment of a kind of three-dimensional map with personage's semantic information, for creating the three-dimensional map with personage's semantic information.The construction method of three-dimensional map provided by the present application with personage's semantic information includes: that processing equipment acquisition camera shoots the image sequence that scene to be reconstructed obtains under the different positions and pose of scene to be reconstructed；The three-dimensional structure information of the personage of scene to be reconstructed and personage's posture information of personage are calculated by personage's reconstruction thread on the basis of image sequence for processing equipment；The three-dimensional environment information for stating scene to be reconstructed is calculated by geometrical reconstruction thread on the basis of image sequence for processing equipment；Processing equipment imports three-dimensional structure information and personage's posture information in three-dimensional environment information, obtains the corresponding three-dimensional map of scene to be reconstructed.
Description
Technical field
This application involves data processing field more particularly to a kind of building sides of the three-dimensional map with personage's semantic information
Method and processing equipment.
Background technique
In the application of three-dimensional map, the application of semantic three-dimensional map can be by adding semanteme abundant in three-dimensional map
Information can provide more function choosing-items than geometrical constraint is used alone, and semantic three-dimensional map provides a powerful semanteme
Information database, the database are that three-dimensional map carries out information inquiry, therefore in the application of specific scene, robot be would know that
The attribute of object in scene so as to be responded, such as avoids touching or interaction.
In the prior art, semantic three-dimensional map is especially by the position object in identification scene, and assigns its semantic label,
To when using the semanteme three-dimensional map, identify that the attribute of object carries out corresponding response according to the semantic label.
However it is found in further practical application, existing semanteme three-dimensional map, for dynamic change
Object is then difficult to establish the geometrical model of topological coherence, can only meet the positioning and navigation feature on basis at this stage, this is one
Determine to limit the intelligent Application of robot in degree.
Summary of the invention
This application provides the construction methods and processing equipment of a kind of three-dimensional map with personage's semantic information, are used for
Create the three-dimensional map with personage's semantic information.
In a first aspect, the application provides a kind of construction method of three-dimensional map with personage's semantic information, method includes:
Processing equipment obtains camera and shoots the image sequence that scene to be reconstructed obtains under the different positions and pose of scene to be reconstructed；
Processing equipment is calculated the personage's of scene to be reconstructed on the basis of image sequence, through personage's reconstruction thread
Three-dimensional structure information and personage's posture information of personage；
The three-dimensional ring of scene to be reconstructed is calculated by geometrical reconstruction thread on the basis of image sequence for processing equipment
Border information；
Processing equipment imports three-dimensional structure information and personage's posture information in three-dimensional environment information, obtains field to be reconstructed
The corresponding three-dimensional map of scape.
Optionally, processing equipment rebuilds thread by personage and three-dimensional structure letter is calculated on the basis of image sequence
Breath includes:
Image sequence is imported personage and rebuilds thread by processing equipment；
Processing equipment is rebuild in thread in personage, extracts the key frame images of image sequence；
Processing equipment divides key frame images by Mask R-CNN parted pattern, obtains personage's semantic label of personage；
Processing equipment generates three-dimensional structure information on the basis of the profile information of personage's semantic label.
Optionally, processing equipment rebuilds thread by personage and personage's pose letter is calculated on the basis of image sequence
Breath includes:
Geometrical reconstruction thread image sparse point cloud information obtained in calculating is imported personage and rebuilds thread by processing equipment；
Processing equipment is rebuild in thread in personage, is believed according to the position of image sparse point cloud information and personage's semantic label
Breath, is calculated personage's posture information, personage's posture information of personage includes the central point three-dimensional coordinate of personage.
Optionally, processing equipment imports three-dimensional structure information and personage's posture information in three-dimensional environment information, obtains
The corresponding three-dimensional map of scene to be reconstructed includes:
Processing equipment merges three-dimensional structure information and personage's posture information, obtains the three-dimensional information of personage；
Processing equipment imports three-dimensional information in three-dimensional environment information, and according to timestamp carries out synchronizing information and by three
The point cloud for tieing up same position in the point cloud covering three-dimensional environment information of information, obtains three-dimensional map.
Optionally, scene to be reconstructed is calculated by geometrical reconstruction thread on the basis of image sequence in processing equipment
Three-dimensional environment information include:
Processing equipment imports image sequence in geometrical reconstruction thread；
Processing equipment will by Scale invariant features transform (Scale-Invariant Feature Transform,
SIFT) algorithm extract image sequence characteristic point, and by Loop constraint remove characteristic point in wrong epipolar geometry point and
With K-means level tree representation characteristic point；
Constraint of the processing equipment according to epipolar geometry, the corresponding epipolar geometry image of construction feature point；
Processing equipment estimates camera pose according to epipolar geometry image；
Processing equipment minimizes the re-projection of camera pose by bundle adjustment (Bundle Adjustment, BA) algorithm
Error；
Processing equipment passes through motor reorgan algorithm, the corresponding image sparse point cloud information of construction feature point；
Processing equipment passes through quantum evolutionary algorithm (Quantum-Inspired Evolutionary Algorithm, QEA)
For each width image selection adjacent image, stereo-picture group is constructed；
Processing equipment calculates the depth map of each image by PatchMatch algorithm；
Processing equipment is merged multiple depth maps by variational algorithm, obtains depth image；
Processing equipment obtains three-dimensional environment information by poisson algorithm constructing environment surface.
The application provides a kind of processing equipment in second aspect, and equipment includes:
Acquiring unit shoots the image that scene to be reconstructed obtains for obtaining camera under the different positions and pose of scene to be reconstructed
Sequence；
Computing unit, for rebuilding thread by personage and scene to be reconstructed being calculated on the basis of image sequence
The three-dimensional structure information of personage and personage's posture information of personage；
Computing unit is also used on the basis of image sequence, and scene to be reconstructed is calculated by geometrical reconstruction thread
Three-dimensional environment information；
Import unit, for three-dimensional structure information and personage's posture information to be imported in three-dimensional environment information, obtain to
Rebuild the corresponding three-dimensional map of scene.
Optionally, computing unit is specifically used for:
Image sequence is imported into personage and rebuilds thread；
It is rebuild in thread in personage, extracts the key frame images of image sequence；
Divide key frame images by Mask R-CNN parted pattern, obtains personage's semantic label of personage；
On the basis of the profile information of personage's semantic label, three-dimensional structure information is generated.
Optionally, computing unit is specifically used for:
Geometrical reconstruction thread image sparse point cloud information obtained in calculating is imported into personage and rebuilds thread；
It rebuilds in thread in personage, according to image sparse point cloud information and the location information of personage's semantic label, calculates
Personage's posture information is obtained, personage's posture information of personage includes the central point three-dimensional coordinate of personage.
Optionally, computing unit is specifically used for:
Geometrical reconstruction thread image sparse point cloud information obtained in calculating is imported into personage and rebuilds thread；
It rebuilds in thread in personage, according to image sparse point cloud information and the location information of personage's semantic label, calculates
Personage's posture information is obtained, personage's posture information of personage includes the central point three-dimensional coordinate of personage.
Optionally, import unit is specifically used for:
Three-dimensional structure information and personage's posture information are merged, the three-dimensional information of personage is obtained；
Three-dimensional information is imported in three-dimensional environment information, and synchronizing information is carried out according to timestamp and by three-dimensional information
The point cloud of same position, obtains three-dimensional map in point cloud covering three-dimensional environment information.
Optionally, computing unit is specifically used for:
Image sequence is imported in geometrical reconstruction thread；
The characteristic point of image sequence will be extracted by SIFT algorithm, and it is outer that the mistake removed in characteristic point is constrained by Loop
Pole geometric point and with K-means level tree representation characteristic point；
According to the constraint of epipolar geometry, the corresponding epipolar geometry image of construction feature point；
According to epipolar geometry image, camera pose is estimated；
By BA algorithm, the re-projection error of camera pose is minimized；
By motor reorgan algorithm, the corresponding image sparse point cloud information of construction feature point；
It is each width image selection adjacent image by QEA, constructs stereo-picture group；
The depth map of each image is calculated by PatchMatch algorithm；
Multiple depth maps are merged by variational algorithm, obtain depth image；
By poisson algorithm constructing environment surface, three-dimensional environment information is obtained.
The third aspect, the application provide a kind of processing equipment, including processor, and processor is stored for executing in memory
Computer program when realize as above-mentioned first aspect the three-dimensional map with personage's semantic information construction method any
Step.
Fourth aspect, the application provide a kind of readable storage medium storing program for executing, are stored thereon with computer program, computer program quilt
Any step of the construction method of the three-dimensional map with personage's semantic information such as above-mentioned first aspect is realized when processor executes
Suddenly.
As can be seen from the above technical solutions, the application has the following advantages:
Processing equipment shoots the image sequence that scene to be reconstructed obtains in acquisition camera under the different positions and pose of scene to be reconstructed
After column, on the one hand the basis based on image sequence is believed by the three-dimensional environment that scene to be reconstructed is calculated in geometrical reconstruction thread
Breath constructs the initial three-dimensional map of scene to be reconstructed, on the other hand also rebuilds thread by personage and field to be reconstructed is calculated
The three-dimensional structure information of the personage of scape and personage's posture information of personage, complete to the identification of personage in scene to be reconstructed and
The imparting of personage's semanteme, so that three-dimensional structure information and personage's posture information are imported three-dimensional environment information again by processing equipment
Afterwards, the corresponding three-dimensional map with personage's semantic information of scene to be reconstructed can be obtained, can dynamic personage in capturing scenes with
And the semantic information of dynamic personage is provided, more abundant data can be provided for the intelligent Application of robot and supported.
Detailed description of the invention
Fig. 1 shows a kind of flow diagram of the construction method of three-dimensional map of the application with personage's semantic information；
Fig. 2 shows the application personages to rebuild a kind of flow diagram that thread calculates the three-dimensional structure information of personage；
Fig. 3 shows the application personage and rebuilds a kind of flow diagram that thread calculates personage's posture information of personage；
Fig. 4 shows a kind of stream that three-dimensional structure information and personage's posture information are imported three-dimensional environment information by the application
Journey schematic diagram；
Fig. 5 shows a kind of structural schematic diagram of the application processing equipment；
Fig. 6 shows another structural schematic diagram of processing equipment provided by the present application.
Specific embodiment
This application provides the construction methods and processing equipment of a kind of three-dimensional map with personage's semantic information, are used for
Create the three-dimensional map with personage's semantic information.
Below in conjunction with the attached drawing in the embodiment of the present application, technical solutions in the embodiments of the present application carries out clear, complete
Site preparation description, it is clear that described embodiments are only a part of embodiments of the present application, instead of all the embodiments.It is based on
Embodiment in the application, those skilled in the art's every other implementation obtained without creative efforts
Example, shall fall in the protection scope of this application.
The description and claims of this application and term " first " in above-mentioned attached drawing, " second " etc. are for distinguishing
Similar object, without being used to describe a particular order or precedence order.It should be understood that the data used in this way are in appropriate feelings
It can be interchanged under condition, so that the embodiments described herein can be real with the sequence other than the content for illustrating or describing herein
It applies.In addition, term " includes " and " having " and their any deformation, it is intended that cover it is non-exclusive include, for example, packet
The process, method, system, product or equipment for having contained series of steps or module those of be not necessarily limited to be clearly listed step or
Module, but may include other steps being not clearly listed or intrinsic for these process, methods, product or equipment or
Module.Occur in this application to step carry out name perhaps number be not meant to must according to name or number
Indicated time/logic sequencing executes the step in method flow, and named or number process step can be with
Execution order is changed according to the technical purpose to be realized, as long as identical or similar technical effect can be reached.
The division of module appeared in the application is a kind of division in logic, can have when realizing in practical application
Other division mode, such as multiple modules can be combined into or are integrated in another system, or some features can be ignored,
Or do not execute, in addition, shown or discussion mutual coupling, direct-coupling or communication connection can be by one
A little interfaces, the indirect coupling or communication connection between module can be electrical or other similar form, do not make in the application
It limits.Also, module or submodule can be the separation that may not be physically as illustrated by the separation member, can be
It can not be physical module, or can be distributed in multiple circuit modules, portion therein can be selected according to the actual needs
Point or whole module realize the purpose of application scheme.
Firstly, before introducing the application, introduce first this application involves processing equipment.
In this application, processing equipment can carry out the analysis processing of data, be believed by application the application with personage's semanteme
The construction method of the three-dimensional map of breath can be created that three with personage's semantic information according to the image graphic sequence of scene to be reconstructed
Tie up map.
Processing equipment can be with independent operating, such as can be the robot in scene construction on site three-dimensional map to be reconstructed,
Or may be a part of the processing system of creation three-dimensional map, which may also include camera, cloud in addition to processing equipment
The elements such as platform；Alternatively, the processing equipment can also configure on other devices, such as robot, assist people is to weight
Build scene construction on site three-dimensional map.
In the following, the explanation then based on above-mentioned scene, starts the structure for the three-dimensional map for having personage's semantic information to the application
Construction method describes in detail.
Refering to fig. 1, Fig. 1 shows a kind of process of the construction method of three-dimensional map of the application with personage's semantic information
Schematic diagram, specifically, it may include following steps that the application, which has the construction method of the three-dimensional map of personage's semantic information:
Step S101, processing equipment obtain camera and shoot what scene to be reconstructed obtained under the different positions and pose of scene to be reconstructed
Image sequence；
It is appreciated that processing equipment itself may include that camera or processing equipment can also connect external camera, from camera
The middle camera that extracts shoots the image sequence that reconstruction scene scene obtains under the different positions and pose of scene to be reconstructed.
Alternatively, processing equipment can also be from local data base, user equipment (User Equipment, UE) or server etc.
It is stored with the equipment that camera shoots the image sequence that scene to be reconstructed obtains under the different positions and pose of scene to be reconstructed, extracts the figure
As sequence.
Wherein, UE is smart phone, tablet computer, laptop, desktop computer, computer all-in-one machine or individual digital
Assistant (Personal Digital Assistant, PDA) etc. can store the terminal device of image sequence.Further, user
Reconstruction tasks of the processing equipment about the three-dimensional map of scene to be reconstructed can be also triggered by UE.
It is appreciated that camera can provide original image for the application, which is made of image sequence, and camera can be with
Take RGB (Red-Green-Blue, RGB) image of scene to be reconstructed.
Certainly, in practical applications, camera is further, can also have the energy of shooting depth image and three-dimensional point cloud
Power.
Step S102, processing equipment rebuild thread by personage and field to be reconstructed are calculated on the basis of image sequence
The three-dimensional structure information of the personage of scape and personage's posture information of personage；
Processing equipment can start personage provided by the present application and rebuild thread and geometry after obtaining above-mentioned image sequence
Thread is rebuild, and image sequence is directed respectively into personage and is rebuild in thread and geometrical reconstruction thread, image sequence is carried out simultaneously
Row processing, improves the treatment effeciency to image sequence.
In this application, personage rebuilds thread and is used for from the personage identified in scene to be reconstructed in image sequence, and calculates
Obtain its three-dimensional structural information and personage's posture information.
Three-dimensional structure information is mainly used for indicating profile information of the personage in three-dimensional structure, and personage's posture information is mainly used
In the different positions and pose for indicating personage, to combine image of the personage under different positions and pose to carry out corresponding person recognition and personage
Modeling.
Field to be reconstructed is calculated on the basis of image sequence, through geometrical reconstruction thread in step S103, processing equipment
The three-dimensional environment information of scape；
On the other hand, geometrical reconstruction thread is used for the geometry environment rebuild in scene to be reconstructed, i.e., in static identification model
Interior background environment is enclosed, which can be restored by obtained three-dimensional environment information.
It is appreciated that in practical applications, which also can be used three-dimensional map in the prior art and rebuilds
Technology restores the geometry environment in scene to be reconstructed.
Three-dimensional structure information and personage's posture information are imported in three-dimensional environment information, are obtained by step S104, processing equipment
To the corresponding three-dimensional map of scene to be reconstructed.
In the people that the three-dimensional structure information of personage, personage are calculated by personage's reconstruction thread and geometrical reconstruction thread
When object posture information and three-dimensional environment information, but processing equipment the above two import three-dimensional environment information in, three-dimensional structure is believed
Personage's posture information of breath and personage are covered on the three-dimensional information of same position in three-dimensional environment, thus in three-dimensional map,
It can accurately identify personage and assign its personage's semantic label, so that realizing has the three-dimensional map of personage's semantic information.
From the foregoing it is appreciated that, processing equipment shoots scene to be reconstructed in acquisition camera under the different positions and pose of scene to be reconstructed
After obtained image sequence, on the one hand scene to be reconstructed is calculated by geometrical reconstruction thread in the basis based on image sequence
Three-dimensional environment information, construct the initial three-dimensional map of scene to be reconstructed, thread calculating also on the other hand rebuild by personage
The three-dimensional structure information of the personage of scene to be reconstructed and personage's posture information of personage are obtained, is completed to people in scene to be reconstructed
The identification of object and the imparting of personage's semanteme, so that three-dimensional structure information and personage's posture information are imported three again by processing equipment
After tieing up environmental information, the corresponding three-dimensional map with personage's semantic information of scene to be reconstructed can be obtained, it can be in capturing scenes
The semantic information of dynamic personage and offer dynamic personage, can provide more abundant data branch for the intelligent Application of robot
It holds.
In one embodiment, referring to Fig.2, Fig. 2 shows the application personages to rebuild the three-dimensional structure that thread calculates personage
A kind of flow diagram of information, specifically, the three-dimensional structure information that the application personage rebuilds thread calculating personage can be by such as
Lower step is realized:
Image sequence is imported personage and rebuilds thread by step S201, processing equipment；
Firstly, processing equipment head starts personage and rebuilds thread weight when carrying out personage's reconstruction tasks of personage's reconstruction thread
And image sequence is imported into the in running order personage and is rebuild in thread, it triggers the personage that it carries out scene to be reconstructed and rebuilds
Processing.
Step S202, processing equipment are rebuild in thread in personage, extract the key frame images of image sequence；
Image sequence includes multiple frame image, and for the processing workload for reducing frame image, personage rebuilds thread and working
In, key frame images are extracted from image sequence first.
Specifically, can be by the frame image recognition in default frame number or image with the biggish image of dynamic change amplitude
Key frame image.
Step S203, processing equipment divide key frame images by Mask R-CNN parted pattern, obtain the personage of personage
Semantic label；
After filtering out key frame images, personage rebuilds thread dispatching Mask R-CNN parted pattern, to the key frame figure
As being split processing, the personage in key frame images is identified, and assign its personage's semantic label.
The Mask R-CNN parted pattern includes multiple convolutional neural networks (Convolutional Neural
Networks, CNN), target detection, target classification and Pixel-level Target Segmentation can be carried out, it in this application, can be from key frame
Character image is identified and be partitioned into image, and assigns its personage's semantic label.
Step S204, processing equipment generate three-dimensional structure information on the basis of the profile information of personage's semantic label.
Further, personage rebuild thread on the basis of obtained personage's semantic label, using it includes profile letter
Breath, generates the three-dimensional structure information of personage.
In another embodiment, the application personage is shown refering to Fig. 3, Fig. 3 rebuild the personage position that thread calculates personage
A kind of flow diagram of appearance information, specifically, personage's posture information that the application personage rebuilds thread calculating personage can pass through
Following steps are realized:
Geometrical reconstruction thread image sparse point cloud information obtained in calculating is imported personage by step S301, processing equipment
Rebuild thread；
It is appreciated that image sequence pair can be obtained in calculating in the geometrical reconstruction thread that the other side carries out parallel
The image sparse point cloud information answered, the specific calculating process of geometrical reconstruction thread, sees subsequent content, specifically not first herein
It repeats.
The corresponding image sparse point cloud information of image sequence is being calculated in geometrical reconstruction thread, can directly rebuild to personage
Thread output；Alternatively, personage rebuilds thread when carrying out the calculating of personage's posture information, then image is transferred to collection occlusal reconstruction thread
Sparse point cloud information.
Step S302, processing equipment is rebuild in thread in personage, according to image sparse point cloud information and personage's semanteme mark
Personage's posture information is calculated in the location information of label.
Wherein, personage's posture information of personage includes the central point three-dimensional coordinate of personage.
Personage rebuilds thread after obtaining the sparse point cloud information that geometrical reconstruction thread is calculated, i.e., in combination with personage's language
The location information that adopted label includes carries out the calculating of the task posture information of personage.
In another embodiment, the application is shown refering to Fig. 4, Fig. 4 and believes three-dimensional structure information and personage's pose
Breath imports a kind of flow diagram of three-dimensional environment information, specifically, the application believes three-dimensional structure information and personage's pose
Breath imports three-dimensional environment information and can be achieved by the steps of:
Step S401, processing equipment merge three-dimensional structure information and personage's posture information, obtain the three of personage
Tie up information；
First by personage's three-dimensional structure information for being calculated of reconstruction thread and and personage's posture information import it is several
When what is rebuild in the three-dimensional environment information that thread is calculated, processing equipment can be first by three-dimensional structure information and and personage position
Appearance information is merged, and the three-dimensional information that can indicate three-dimensional structure and personage's pose is obtained.
Step S402, processing equipment import three-dimensional information in three-dimensional environment information, and same according to timestamp progress information
Step and the point cloud that the point cloud of three-dimensional information is covered to same position in three-dimensional environment information, obtain three-dimensional map.
After obtaining the three-dimensional information of personage, processing equipment can be conducted into three-dimensional environment information, that is, import dimension tool
In the three-dimensional map of someone's object semantic information, personage's semantic information is made it have.
During this, the timestamp of both processing equipment can also synchronize carrying carries out temporal synchronizing information, and root
The point cloud of three-dimensional information is covered on the point cloud of same position in three-dimensional environment information, completes to have by the distribution of strong point cloud
The creation of the three-dimensional map of personage's semantic information.
In another embodiment, processing equipment is calculated on the basis of image sequence by geometrical reconstruction thread
The three-dimensional environment information of scene to be reconstructed, can specifically be achieved by the steps of:
Processing equipment imports image sequence in geometrical reconstruction thread；
Processing equipment will extract the characteristic point of image sequence by SIFT algorithm, and be constrained in removal characteristic point by Loop
Wrong epipolar geometry point and with K-means level tree representation characteristic point；
Wherein, characteristic point is extracted, it is specific: to search for the picture position on all scales；Known by gaussian derivative function
Not potentially for the point of interest of scale and invariable rotary；On the position of each candidate, pass through the fine model of a fitting
To determine position and scale；Gradient direction based on image local distributes to each key point position one or more direction；?
In neighborhood around each key point, the gradient of image local is measured on selected scale；Finally state a kind of vector side
Formula expressing feature point.
Constraint of the processing equipment according to epipolar geometry, the corresponding epipolar geometry image of construction feature point；
Processing equipment estimates camera pose according to epipolar geometry image；
Processing equipment minimizes the re-projection error of camera pose by light BA algorithm；
Processing equipment passes through motor reorgan algorithm, the corresponding image sparse point cloud information of construction feature point；
Processing equipment is each width image selection adjacent image by QEA, constructs stereo-picture group；
Processing equipment calculates the depth map of each image by PatchMatch algorithm；
Processing equipment is merged multiple depth maps by variational algorithm, obtains depth image；
Processing equipment obtains three-dimensional environment information by poisson algorithm constructing environment surface.
It is the introduction of the construction method for the three-dimensional map that there is personage's semantic information to the application above, starts to introduce below
The processing equipment of the application.
A kind of structural schematic diagram of the application processing equipment is shown refering to Fig. 5, Fig. 5, specifically, processing equipment may include
Such as flowering structure:
Acquiring unit 501 shoots what scene to be reconstructed obtained under the different positions and pose of scene to be reconstructed for obtaining camera
Image sequence；
Computing unit 502, for rebuilding thread by personage and scene to be reconstructed being calculated on the basis of image sequence
Personage three-dimensional structure information and personage personage's posture information；
Computing unit 502 is also used on the basis of image sequence, and field to be reconstructed is calculated by geometrical reconstruction thread
The three-dimensional environment information of scape；
Import unit 503 is obtained for importing three-dimensional structure information and personage's posture information in three-dimensional environment information
The corresponding three-dimensional map of scene to be reconstructed.
In one embodiment, computing unit 502 are specifically used for:
Image sequence is imported into personage and rebuilds thread；
It is rebuild in thread in personage, extracts the key frame images of image sequence；
Divide key frame images by Mask R-CNN parted pattern, obtains personage's semantic label of personage；
On the basis of the profile information of personage's semantic label, three-dimensional structure information is generated.
In another embodiment, computing unit 502 are specifically used for:
Geometrical reconstruction thread image sparse point cloud information obtained in calculating is imported into personage and rebuilds thread；
It rebuilds in thread in personage, according to image sparse point cloud information and the location information of personage's semantic label, calculates
Personage's posture information is obtained, personage's posture information of personage includes the central point three-dimensional coordinate of personage.
In another embodiment, computing unit 502 is specifically used for:
Geometrical reconstruction thread image sparse point cloud information obtained in calculating is imported into personage and rebuilds thread；
It rebuilds in thread in personage, according to image sparse point cloud information and the location information of personage's semantic label, calculates
Personage's posture information is obtained, personage's posture information of personage includes the central point three-dimensional coordinate of personage.
In another embodiment, import unit 503 is specifically used for:
Three-dimensional structure information and personage's posture information are merged, the three-dimensional information of personage is obtained；
Three-dimensional information is imported in three-dimensional environment information, and synchronizing information is carried out according to timestamp and by three-dimensional information
The point cloud of same position, obtains three-dimensional map in point cloud covering three-dimensional environment information.
In another embodiment, computing unit 502 is specifically used for:
Image sequence is imported in geometrical reconstruction thread；
The characteristic point of image sequence will be extracted by SIFT algorithm, and it is outer that the mistake removed in characteristic point is constrained by Loop
Pole geometric point and with K-means level tree representation characteristic point；
According to the constraint of epipolar geometry, the corresponding epipolar geometry image of construction feature point；
According to epipolar geometry image, camera pose is estimated；
By BA algorithm, the re-projection error of camera pose is minimized；
By motor reorgan algorithm, the corresponding image sparse point cloud information of construction feature point；
It is each width image selection adjacent image by QEA, constructs stereo-picture group；
The depth map of each image is calculated by PatchMatch algorithm；
Multiple depth maps are merged by variational algorithm, obtain depth image；
By poisson algorithm constructing environment surface, three-dimensional environment information is obtained.
Another structural schematic diagram of processing equipment provided by the present application is shown refering to Fig. 6, Fig. 6, specifically, the application
The processing equipment of offer includes processor 601, real when processor 601 is for executing the computer program stored in memory 802
Now there is in corresponding any embodiment each step of the construction method of the three-dimensional map of personage's semantic information as shown in Figure 1 to Figure 4；Or
Person is realized when processor 601 is for executing the computer program stored in memory 602 such as each unit in Fig. 5 corresponding embodiment
Function.
Illustratively, computer program can be divided into one or more module/units, and one or more module/
Unit is stored in memory 602, and is executed by processor 601, to complete the application.One or more module/units can
To be the series of computation machine program instruction section that can complete specific function, which is calculating for describing computer program
Implementation procedure in machine device.
Processing equipment may include, but be not limited only to processor 601, memory 602.It will be understood by those skilled in the art that
Signal is only the example of computer installation, does not constitute the restriction to processing equipment, may include more more or less than illustrating
Component, perhaps combine certain components or different components, such as processing equipment can also include input-output equipment, net
Network access device, bus etc., processor 601, memory 602, input-output equipment and network access equipment etc. pass through bus
It is connected.
Processor 601 can be central processing unit (Central Processing Unit, CPU), can also be other
General processor, digital signal processor (Digital Signal Processor, DSP), specific integrated circuit
(Application Specific Integrated Circuit, ASIC), ready-made programmable gate array (Field-
Programmable Gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic,
Discrete hardware components etc..General processor can be microprocessor or the processor is also possible to any conventional processor
Deng processor is the control centre of computer installation, utilizes each portion of various interfaces and the entire computer installation of connection
Point.
Memory 602 can be used for storing computer program and/or module, and processor 601 is stored in by operation or execution
Computer program and/or module in memory 602, and the data being stored in memory 602 are called, realize computer dress
The various functions of setting.Memory 602 can mainly include storing program area and storage data area, wherein storing program area can store
Application program needed for operating system, at least one function (such as sound-playing function, image player function etc.) etc.；Store number
It can be stored according to area and created data (such as audio data, video data etc.) etc. are used according to mobile phone.In addition, memory can
It can also include nonvolatile memory, such as hard disk, memory, plug-in type hard disk, intelligence to include high-speed random access memory
Energy storage card (Smart Media Card, SMC), secure digital (Secure Digital, SD) card, flash card (Flash
Card), at least one disk memory, flush memory device or other volatile solid-state parts.
The application also provides a kind of readable storage medium storing program for executing, is stored thereon with computer program, computer program is by processor
The construction method of the three-dimensional map with personage's semantic information in corresponding any embodiment as shown in Figure 1 to Figure 4 is realized when execution.
If it is appreciated that integrated unit be realized in the form of SFU software functional unit and as the sale of independent product or
In use, can store in a computer readable storage medium.Based on this understanding, the technical solution sheet of the application
The all or part of the part that contributes to existing technology or the technical solution can be with software product in other words in matter
Form embodies, which is stored in a storage medium, including some instructions are used so that a meter
Calculate the whole that machine equipment (can be personal computer, server or the network equipment etc.) executes the application each method embodiment
Or part steps.And storage medium above-mentioned include: USB flash disk, mobile hard disk, read-only memory (Read-only Memory,
ROM), random access memory (Random Access Memory, RAM), magnetic or disk etc. are various can store program
The medium of code.
It is apparent to those skilled in the art that for convenience and simplicity of description, the processing of foregoing description
The specific work process of equipment and its unit, can be referring to figs. 1 to the three-dimensional in Fig. 4 corresponding embodiment with personage's semantic information
The explanation of the construction method of map, specific details are not described herein.
To sum up, the construction method of the three-dimensional map provided by the present application with personage's semantic information, processing equipment is readable deposits
Storage media, processing equipment shoot the image sequence that scene to be reconstructed obtains in acquisition camera under the different positions and pose of scene to be reconstructed
Afterwards, based on the basis of image sequence, the three-dimensional environment information of scene to be reconstructed is on the one hand calculated by geometrical reconstruction thread,
The initial three-dimensional map of scene to be reconstructed is constructed, thread is also on the other hand rebuild by personage, scene to be reconstructed is calculated
The three-dimensional structure information of personage and personage's posture information of personage complete identification and personage to personage in scene to be reconstructed
Semantic imparting, thus after three-dimensional structure information and personage's posture information are imported three-dimensional environment information again by processing equipment, it can
The corresponding three-dimensional map with personage's semantic information of scene to be reconstructed is obtained, it can dynamic personage in capturing scenes and offer
The semantic information of dynamic personage can provide more abundant data for the intelligent Application of robot and support.
In several embodiments provided herein, it should be understood that the deployment system of disclosed fictitious host computer and
Its unit, may be implemented in other ways.For example, the apparatus embodiments described above are merely exemplary, for example,
The division of unit, only a kind of logical function partition, there may be another division manner in actual implementation, such as multiple units
Or component can be combined or can be integrated into another system, or some features can be ignored or not executed.Another point, institute
Display or the mutual coupling, direct-coupling or communication connection discussed can be through some interfaces, device or unit
Indirect coupling or communication connection can be electrical property, mechanical or other forms.
Unit may or may not be physically separated as illustrated by the separation member, shown as a unit
Component may or may not be physical unit, it can and it is in one place, or may be distributed over multiple networks
On unit.It can some or all of the units may be selected to achieve the purpose of the solution of this embodiment according to the actual needs.
It, can also be in addition, each functional unit in each embodiment of the application can integrate in one processing unit
It is that each unit physically exists alone, can also be integrated in one unit with two or more units.Above-mentioned integrated list
Member both can take the form of hardware realization, can also realize in the form of software functional units.
More than, above embodiments are only to illustrate the technical solution of the application, rather than its limitations；Although referring to aforementioned reality
Example is applied the application is described in detail, those skilled in the art should understand that: it still can be to aforementioned each
Technical solution documented by embodiment is modified or equivalent replacement of some of the technical features；And these are modified
Or replacement, the spirit and scope of each embodiment technical solution of the application that it does not separate the essence of the corresponding technical solution.
Claims (10)
1. a kind of construction method of the three-dimensional map with personage's semantic information, which is characterized in that the described method includes:
Processing equipment obtains camera and shoots the image sequence that the scene to be reconstructed obtains under the different positions and pose of scene to be reconstructed；
The processing equipment rebuilds thread by personage and the scene to be reconstructed is calculated on the basis of described image sequence
Personage three-dimensional structure information and the personage personage's posture information；
The processing equipment is calculated by geometrical reconstruction thread on the basis of described image sequence and states scene to be reconstructed
Three-dimensional environment information；
The processing equipment imports the three-dimensional structure information and personage's posture information in the three-dimensional environment information,
Obtain the corresponding three-dimensional map of the scene to be reconstructed.
2. the method according to claim 1, wherein the processing equipment is on the basis of described image sequence,
The three-dimensional structure information, which is calculated, by personage reconstruction thread includes:
Described image sequence is imported the personage and rebuilds thread by the processing equipment；
The processing equipment is rebuild in thread in the personage, extracts the key frame images of described image sequence；
The processing equipment divides the key frame images by Mask R-CNN parted pattern, obtains personage's language of the personage
Adopted label；
The processing equipment generates the three-dimensional structure information on the basis of profile information of personage's semantic label.
3. according to the method described in claim 2, it is characterized in that, the processing equipment on the basis of described image sequence,
Personage's posture information, which is calculated, by personage reconstruction thread includes:
Geometrical reconstruction thread image sparse point cloud information obtained in calculating is imported the personage by the processing equipment
Rebuild thread；
The processing equipment is rebuild in thread in the personage, semantic according to the sparse point cloud information of described image and the personage
The location information of label, is calculated personage's posture information, and personage's posture information of the personage includes the personage
Central point three-dimensional coordinate.
4. according to the method described in claim 3, it is characterized in that, the processing equipment is by the three-dimensional structure information and institute
It states personage's posture information to import in the three-dimensional environment information, obtaining the corresponding three-dimensional map of the scene to be reconstructed includes:
The processing equipment merges the three-dimensional structure information and personage's posture information, obtains the personage's
Three-dimensional information；
The processing equipment imports the three-dimensional information in the three-dimensional environment information, and carries out synchronizing information according to timestamp
And the point cloud of the three-dimensional information is covered into the point cloud of same position in the three-dimensional environment information, obtain it is described dimensionally
Figure.
5. the method according to claim 1, wherein the processing equipment is on the basis of described image sequence,
Include: by the three-dimensional environment information that the scene to be reconstructed is calculated in geometrical reconstruction thread
The processing equipment imports described image sequence in the geometrical reconstruction thread；
The processing equipment will extract the characteristic point of described image sequence by Scale invariant features transform SIFT algorithm, and pass through
Loop constraint removes the wrong epipolar geometry point in the characteristic point and the characteristic point described in K-means level tree representation；
The processing equipment constructs the corresponding epipolar geometry image of the characteristic point according to the constraint of epipolar geometry；
The processing equipment estimates camera pose according to the epipolar geometry image；
The processing equipment minimizes the re-projection error of the camera pose by bundle adjustment BA algorithm；
The processing equipment constructs the corresponding image sparse point cloud information of the characteristic point by motor reorgan algorithm；
The processing equipment is each width image selection adjacent image by quantum evolutionary algorithm QEA, constructs stereo-picture group；
The processing equipment calculates the depth map of each image by PatchMatch algorithm；
The processing equipment is merged multiple depth maps by variational algorithm, obtains depth image；
The processing equipment obtains the three-dimensional environment information by poisson algorithm constructing environment surface.
6. a kind of processing equipment, which is characterized in that the equipment includes:
Acquiring unit shoots the image that the scene to be reconstructed obtains for obtaining camera under the different positions and pose of scene to be reconstructed
Sequence；
Computing unit, for rebuilding thread by personage and the field to be reconstructed being calculated on the basis of described image sequence
The three-dimensional structure information of the personage of scape and personage's posture information of the personage；
The computing unit is also used on the basis of described image sequence, by geometrical reconstruction thread be calculated it is described to
Rebuild the three-dimensional environment information of scene；
Import unit, for the three-dimensional structure information and personage's posture information to be imported the three-dimensional environment information
In, obtain the corresponding three-dimensional map of the scene to be reconstructed.
7. equipment according to claim 6, which is characterized in that the computing unit is specifically used for:
Described image sequence is imported into the personage and rebuilds thread；
It is rebuild in thread in the personage, extracts the key frame images of described image sequence；
Divide the key frame images by Mask R-CNN parted pattern, obtains personage's semantic label of the personage；
On the basis of the profile information of personage's semantic label, the three-dimensional structure information is generated.
8. the method according to the description of claim 7 is characterized in that the computing unit, is specifically used for:
Geometrical reconstruction thread image sparse point cloud information obtained in calculating is imported into the personage and rebuilds thread；
It rebuilds in thread in the personage, is believed according to the position of the sparse point cloud information of described image and personage's semantic label
Breath, is calculated personage's posture information, and personage's posture information of the personage includes that the central point three-dimensional of the personage is sat
Mark.
9. a kind of processing equipment, which is characterized in that the processing equipment includes processor, and processor is deposited for executing in memory
The method as described in any one of claims 1 to 5 is realized when the computer program of storage.
10. a kind of readable storage medium storing program for executing, which is characterized in that be stored thereon with computer program, computer program is held by processor
The method as described in any one of claims 1 to 5 is realized when row.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910602327.9A CN110298921B (en) | 2019-07-05 | 2019-07-05 | Method for constructing three-dimensional map with character semantic information and processing equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910602327.9A CN110298921B (en) | 2019-07-05 | 2019-07-05 | Method for constructing three-dimensional map with character semantic information and processing equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110298921A true CN110298921A (en) | 2019-10-01 |
| CN110298921B CN110298921B (en) | 2023-07-07 |
Family
ID=68030368
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910602327.9A Active CN110298921B (en) | 2019-07-05 | 2019-07-05 | Method for constructing three-dimensional map with character semantic information and processing equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110298921B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107564012A (en) * | 2017-08-01 | 2018-01-09 | 中国科学院自动化研究所 | Towards the augmented reality method and device of circumstances not known |
| CN109461208A (en) * | 2018-11-15 | 2019-03-12 | 网易（杭州）网络有限公司 | Three-dimensional map processing method, device, medium and calculating equipment |
| CN109658449A (en) * | 2018-12-03 | 2019-04-19 | 华中科技大学 | A kind of indoor scene three-dimensional rebuilding method based on RGB-D image |
| CN109815847A (en) * | 2018-12-30 | 2019-05-28 | 中国电子科技集团公司信息科学研究院 | A kind of vision SLAM method based on semantic constraint |
-
2019
- 2019-07-05 CN CN201910602327.9A patent/CN110298921B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107564012A (en) * | 2017-08-01 | 2018-01-09 | 中国科学院自动化研究所 | Towards the augmented reality method and device of circumstances not known |
| CN109461208A (en) * | 2018-11-15 | 2019-03-12 | 网易（杭州）网络有限公司 | Three-dimensional map processing method, device, medium and calculating equipment |
| CN109658449A (en) * | 2018-12-03 | 2019-04-19 | 华中科技大学 | A kind of indoor scene three-dimensional rebuilding method based on RGB-D image |
| CN109815847A (en) * | 2018-12-30 | 2019-05-28 | 中国电子科技集团公司信息科学研究院 | A kind of vision SLAM method based on semantic constraint |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110298921B (en) | 2023-07-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111243093B (en) | Three-dimensional face grid generation method, device, equipment and storage medium | |
| WO2021238595A1 (en) | Image generation method and apparatus based on artificial intelligence, and device and storage medium | |
| CN109242961A (en) | A kind of face modeling method, apparatus, electronic equipment and computer-readable medium | |
| CN111787242B (en) | Method and apparatus for virtual fitting | |
| US11928800B2 (en) | Image coordinate system transformation method and apparatus, device, and storage medium | |
| CN110020620A (en) | Face identification method, device and equipment under a kind of big posture | |
| CN110135455A (en) | Image matching method, device and computer readable storage medium | |
| US11087493B2 (en) | Depth-image processing device, depth-image processing system, depth-image processing method, and recording medium | |
| CN110705448A (en) | Human body detection method and device | |
| CN108108748A (en) | A kind of information processing method and electronic equipment | |
| US20110129118A1 (en) | Systems and methods for tracking natural planar shapes for augmented reality applications | |
| CN110298873A (en) | Construction method, construction device, robot and the readable storage medium storing program for executing of three-dimensional map | |
| KR20180035869A (en) | Method, device, terminal device and storage medium | |
| CN109035415B (en) | Virtual model processing method, device, equipment and computer readable storage medium | |
| CN114332374A (en) | Virtual display method, equipment and storage medium | |
| CN112308977B (en) | Video processing method, video processing device, and storage medium | |
| CN113469929A (en) | Training data generation method and device, electronic equipment and computer readable storage medium | |
| CN112102480A (en) | Image data processing method, apparatus, device and medium | |
| CN111784818A (en) | Method and device for generating three-dimensional human body model and computer readable storage medium | |
| CN112637665A (en) | Display method and device in augmented reality scene, electronic equipment and storage medium | |
| CN115018959A (en) | Drive processing method, device, equipment and storage medium for three-dimensional virtual model | |
| CN107092347B (en) | Augmented reality interaction system and image processing method | |
| CN114092670A (en) | Virtual reality display method, equipment and storage medium | |
| CN117422851A (en) | Virtual clothes changing method and device and electronic equipment | |
| CN111079535B (en) | Human skeleton action recognition method and device and terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |