Detailed Description
The following detailed description of specific embodiments of the present disclosure is provided in connection with the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the present disclosure, are given by way of illustration and explanation only, not limitation.
The positioning method provided by the disclosure can be applied to terminal equipment such as robots, unmanned planes, unmanned vehicles and the like, and also can be applied to servers such as positioning management servers, cloud servers and the like. When the positioning method is applied to the server, the terminal equipment can send the acquired data such as the image to the server, the server carries out subsequent processing to position the terminal equipment, and the positioning information is returned to the terminal equipment.
It should be noted that, since the camera is integrated in the terminal device such as the robot, in the following description of the present disclosure, when referring to the pose information of the camera, it is also understood as the pose information of the terminal device into which the camera is integrated, and when referring to the positioning of the camera, it is also understood as the positioning of the terminal device into which the camera is integrated.
Fig. 1 is a flow chart illustrating a positioning method according to an exemplary embodiment. As shown in fig. 1, the positioning method may include S101 to S105.
In S101, first posture information of the camera in the map coordinate system at the present time is acquired.
The map coordinate system is a coordinate system of a pre-established point cloud map, and the point cloud map can comprise a map of the current environment where the camera is located. In general, a point cloud map needs to be pre-established by equipment such as a robot to recognize the surrounding environment, and a path is planned according to the point cloud map, so that the current position of the equipment in the point cloud map needs to be acquired, and the equipment can continue to move according to the planned path. In one embodiment, the point cloud map may be pre-established by images acquired by the camera at historical times.
The first pose information of the camera under the map coordinate system at the current moment can comprise position information and pose information of the camera currently in a pre-established point cloud map. The position information may include three-dimensional coordinate information of the camera in a map coordinate system, and the posture information may include deflection angles of the camera with respect to three coordinate axes of the map coordinate system, respectively.
In S102, first point cloud data of a current image acquired by the camera at the current time in a map coordinate system is acquired.
The camera can acquire images in a visual field in real time, and the visual field of the camera is an environment range which can be shot by the camera. The current image acquired by the camera at the current moment is the image of the environment where the camera is currently located.
Wherein, the collection of feature points in the image can be called as point cloud. The point cloud data related to the present disclosure may be three-dimensional point cloud data, and the first point cloud data of the current image in the map coordinate system may include three-dimensional coordinate information of all feature points in the current image in the map coordinate system.
It should be noted that, the execution sequence of S101 and S102 is not specifically limited in the present disclosure, and for example, S102 may be executed first, then S101 may be executed, or both may be executed simultaneously. Fig. 1 only shows an example in which S101 is performed before S102, but does not constitute a limitation on the embodiments of the present disclosure.
In S103, a local point cloud map is extracted from the point cloud map according to the first pose information.
The first pose information acquired in S101 may include position information of the camera currently in the point cloud map. And extracting a local point cloud map from the point cloud map according to the first position information, wherein the local point cloud map refers to extracting a certain area around the current position of the camera. Because the range of the point cloud map is larger, the local point cloud map is extracted, and the interference of other areas on positioning can be avoided.
The range of the extracted local point cloud map may be preset, and the disclosure is not particularly limited as long as the local point cloud map includes the current position of the camera. Illustratively, for example, an area formed by taking the current position information of the camera in the point cloud map as the center and taking a preset distance as a radius is used as the local point cloud map.
In S104, the pose change information of the camera is determined according to the first point cloud data and the local point cloud map.
In the step, the first point cloud data and the local point cloud map can be geometrically registered, and the local point cloud map has a small range and only comprises a certain area around the current position of the camera, so that more accurate feature point registration can be realized. Illustratively, the pose change information of the camera is determined by means of ICP (Iterative Closest Point) geometric registration, for example. The pose change information may refer to an incremental change in pose relative to the first pose information for use in correcting the first pose information.
In S105, the first pose information is corrected according to the pose change information to obtain target pose information of the camera at the current time, and the camera is positioned according to the target pose information.
In this step, the obtained target pose information of the camera at the current time is obtained by correcting the first pose information, so that the target pose information is more accurate, that is, more accurate position information and pose information of the camera at the current time in a map coordinate system are obtained. Therefore, the camera is positioned according to the target pose information, the positioning precision is higher, and the safe and accurate operation of the robot and other equipment integrated with the camera can be ensured.
According to the technical scheme, the local point cloud map is extracted from the pre-established point cloud map according to the first position information of the camera under the map coordinate system at the current moment, and the local point cloud map can be a certain area around the current position of the camera, so that interference of other areas in the point cloud map on positioning is avoided. And then, determining pose change information of the camera according to the first point cloud data of the current image acquired by the camera in a map coordinate system and the local point cloud map. The target pose information of the camera at the current moment is obtained by correcting the first pose information through the pose change information, so that the target pose information is more accurate, the camera can be more accurately positioned according to the target pose information, the positioning accuracy is effectively improved, the current pose and the position of the robot and other equipment integrated with the camera can be accurately judged, and the safe and accurate operation of the equipment is ensured.
Fig. 2 is a flowchart illustrating a method of acquiring first pose information of a camera in a map coordinate system at a current time according to an exemplary embodiment. As shown in fig. 2, S101 may include S201 to S206.
In S201, second position information of the camera in the world coordinate system at the current time is acquired.
A Visual-inertial metrology (VIO) module is usually integrated in a terminal device such as a robot, and an Inertial Measurement Unit (IMU) is integrated in the VIO module. The VIO module may calculate second pose information of the camera in the world coordinate system at the current moment according to the current image acquired by the camera, where the second pose information may include position information and pose information of the camera in the world coordinate system.
In S202, visual repositioning is performed according to the current image and the point cloud map.
Because the VIO module only calculates the pose information of the camera according to the current image, errors generated by positioning at each moment are accumulated continuously, and an error accumulation process exists. In order to ensure the accuracy of the pose information of the camera, a vision repositioning module is usually integrated in the robot and other devices. The visual relocation module can perform visual relocation according to a current image acquired by the camera and a pre-established point cloud map, for example, a loop candidate frame consistent with the current image is detected from the point cloud map, and then the feature points in the loop candidate frame are matched with the feature points of the current image to obtain the current pose information of the camera, so that the phenomenon of error accumulation can be avoided to a certain extent.
However, the visual repositioning has certain limitations, for example, in an actual application scene, an image acquired by a camera is greatly influenced by environmental factors such as light, and if the acquired image has high noise or the number of feature points of the image is small, it is difficult to detect a loop candidate frame. Even if the loop candidate frame is detected, the number of feature points that can be matched is small. Thus, it may happen that the visual repositioning result is not obtained.
It should be noted that, the execution sequence of S201 and S202 is not specifically limited in the present disclosure, and fig. 2 is only an exemplary illustration, for example, S202 may also be executed before S201, or both may be executed at the same time.
In S203, it is determined whether a visual repositioning result is acquired. If not, executing S204; in the case of acquisition, S205 and S206 are executed.
In S204, the second pose information is subjected to coordinate transformation according to the currently stored coordinate transformation information of the world coordinate system and the map coordinate system, so as to obtain the first pose information.
In the case that the visual repositioning result is not obtained, since the second pose information calculated by the VIO module is pose information of the camera in the world coordinate system, in order to obtain the first pose information of the camera in the map coordinate system, coordinate conversion needs to be performed on the second pose information. In this step, the second pose information may be subjected to coordinate conversion according to coordinate system conversion information of the world coordinate system and the map coordinate system that are currently stored. The coordinate system transformation information can represent the pose relationship between the world coordinate system and the map coordinate system. For example, the first posture information may be obtained by the following formula (1):
T_G_I=T_M_I*T_G_M (1)
Wherein, T _ G _ I represents first pose information of the camera in the map coordinate system at the current time, T _ M _ I represents second pose information of the camera in the world coordinate system at the current time, and T _ G _ M represents coordinate system conversion information of the world coordinate system and the map coordinate system which are currently stored. Both T _ G _ I, T _ M _ I and T _ G _ M may be represented in the form of a matrix.
In S205, first pose information is determined according to the visual repositioning result.
Under the condition of obtaining the vision repositioning result, the vision repositioning module carries out vision repositioning according to the current image and the point cloud map, and can directly obtain the position information and the posture information of the camera in the point cloud map, so the vision repositioning result can comprise the first posture information of the camera in a map coordinate system at the current moment.
In S206, the coordinate system conversion information is updated based on the first and second attitude information.
In order to ensure the accuracy of the coordinate system conversion information, the coordinate system conversion information is not fixed, and in the case that the vision repositioning result can be obtained, the coordinate system conversion information can be updated after the first pose information is determined according to the vision repositioning result. The VIO module can output second position and posture information of the camera in a world coordinate system. For example, according to the first posture information and the second posture information, the updated coordinate system conversion information may be obtained by the following formula (2):
T_G_M'=T_G_I*T_M_I-1(2)
Where T _ G _ M' represents the updated coordinate system conversion information.
In the above technical solution, if the visual repositioning result is not obtained, the first pose information may be obtained by performing coordinate transformation on the second pose information of the camera currently in the world coordinate system. If the vision repositioning result can be obtained, the first pose information can be directly obtained according to the vision repositioning result, and the coordinate system conversion information of the world coordinate system and the map coordinate system can be updated, so that the accuracy of the coordinate system conversion information is ensured.
Fig. 3 is a flowchart illustrating a method of acquiring first point cloud data of a current image acquired by a camera at a current time in a map coordinate system according to an exemplary embodiment. As shown in fig. 3, S102 may include S301 and S302.
In S301, second point cloud data of the current image in the world coordinate system is acquired.
The VIO module can calculate second position and posture information of the camera in a world coordinate system according to the current image acquired by the camera, and can output second point cloud data of the current image in the world coordinate system. The second point cloud data may include three-dimensional coordinate information of all feature points in the current image in a world coordinate system.
In S302, the second point cloud data is projected to a map coordinate system to obtain first point cloud data.
In this step, for example, the second point cloud data may be subjected to coordinate conversion to project the second point cloud data into a map coordinate system. In this way, the first point cloud data of the current image acquired by the camera under the map coordinates can be obtained.
In this disclosure, in the above S105, an exemplary embodiment of correcting the first pose information according to the pose change information to obtain the target pose information of the camera at the current time may be:
And determining the target pose information according to the product of the pose change information and the first pose information.
For example, the product of the pose change information and the first pose information may be determined as the target pose information. By way of example, the target pose information may be determined by equation (3) as follows:
T_Gi_I=T_Gi_G*T_G_I (3)
Wherein, T _ Gi _ I represents object pose information, and T _ Gi _ G represents pose change information.
In the present disclosure, the pre-established point cloud map may be established in any manner of the related art, or may be established in the manner of establishing the point cloud map described below in the present disclosure. FIG. 4 is a flow diagram illustrating a method of pre-establishing a point cloud map, according to an example embodiment. As shown in FIG. 4, the method may include S401 to S404.
In S401, third posture information of the camera in the world coordinate system at the historical time is acquired.
The VIO module can calculate third posture information of the camera under a world coordinate system according to images collected by the camera at historical time. The third pose information may include position information and pose information of the camera in a world coordinate system at historical times.
In S402, third point cloud data of the image acquired by the camera at the historical time in the world coordinate system is obtained according to the third pose information.
In the process of establishing the point cloud map, the point cloud data of the image acquired by the camera is the point cloud data under the camera coordinate system. In the step, coordinate conversion can be performed on the point cloud data in the camera coordinate system according to the third pose information of the camera in the world coordinate system, so that third point cloud data of the image acquired by the camera at the historical moment in the world coordinate system can be obtained. The third point cloud data may include three-dimensional coordinate information of feature points in images acquired by the camera at historical times in a world coordinate system.
In S403, filtering the third point cloud data to obtain fourth point cloud data after filtering.
And filtering the third point cloud data to obtain more accurate filtered fourth point cloud data. The present disclosure does not specifically limit the manner of the filtering process. For example, a voxel filtering process may be adopted, and the fourth point cloud data after the voxel filtering process is obtained by creating a three-dimensional voxel grid and performing down-sampling and other processes on the third point cloud data in the three-dimensional voxel grid.
In S404, a point cloud map is generated according to the filtered fourth point cloud data.
in this step, a pre-established point cloud map may be generated according to the fourth point cloud data after the filtering processing by using a Simultaneous localization and Mapping (slam) technique in the related art.
Based on the same inventive concept, the disclosure also provides a positioning device. Fig. 5 is a block diagram illustrating a positioning apparatus according to an exemplary embodiment, and as shown in fig. 5, the positioning apparatus 500 may include:
A pose information acquisition module 501 configured to acquire first pose information of the camera in a map coordinate system at a current moment; the map coordinate system is a coordinate system of a pre-established point cloud map, and the point cloud map comprises a map of the current environment where the camera is located;
A point cloud data acquiring module 502 configured to acquire first point cloud data of a current image acquired by the camera at the current time in the map coordinate system;
An extracting module 503 configured to extract a local point cloud map from the point cloud map according to the first pose information;
A determining module 504 configured to determine pose change information of the camera according to the first point cloud data and the local point cloud map;
A positioning module 505 configured to modify the first pose information according to the pose change information to obtain target pose information of the camera at the current time, and position the camera according to the target pose information.
According to the technical scheme, the local point cloud map is extracted from the pre-established point cloud map according to the first position information of the camera under the map coordinate system at the current moment, and the local point cloud map can be a certain area around the current position of the camera, so that interference of other areas in the point cloud map on positioning is avoided. And then, determining pose change information of the camera according to the first point cloud data of the current image acquired by the camera in a map coordinate system and the local point cloud map. The target pose information of the camera at the current moment is obtained by correcting the first pose information through the pose change information, so that the target pose information is more accurate, the camera can be more accurately positioned according to the target pose information, the positioning accuracy is effectively improved, the current pose and the position of the robot and other equipment integrated with the camera can be accurately judged, and the safe and accurate operation of the equipment is ensured.
Optionally, the pose information acquiring module 501 may include: a first obtaining sub-module configured to obtain second pose information of the camera in a world coordinate system at the current time; a visual repositioning sub-module configured for visual repositioning according to the current image and the point cloud map; and the coordinate conversion sub-module is configured to, in a case that the vision repositioning sub-module does not obtain a vision repositioning result, perform coordinate conversion on the second pose information according to currently stored coordinate system conversion information of the world coordinate system and the map coordinate system to obtain the first pose information.
Optionally, the pose information acquiring module 501 may further include: a second obtaining sub-module configured to, in a case where the visual repositioning sub-module obtains the visual repositioning result, determine the first pose information according to the visual repositioning result.
Optionally, the apparatus 500 may further include: an update module configured to update the coordinate system conversion information according to the first pose information and the second pose information.
Optionally, the point cloud data obtaining module 502 is configured to obtain second point cloud data of the current image in a world coordinate system; and projecting the second point cloud data to the map coordinate system to obtain the first point cloud data.
Optionally, the positioning module 505 is configured to determine the target pose information according to a product of the pose change information and the first pose information.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Fig. 6 is a block diagram illustrating an electronic device 600 according to an example embodiment. As shown in fig. 6, the electronic device 600 may include: a processor 601 and a memory 602. The electronic device 600 may also include one or more of a multimedia component 603, an input/output (I/O) interface 604, and a communications component 605.
The processor 601 is configured to control the overall operation of the electronic device 600, so as to complete all or part of the steps in the positioning method. The memory 602 is used to store various types of data to support operation at the electronic device 600, such as instructions for any application or method operating on the electronic device 600 and application-related data, such as contact data, transmitted and received messages, pictures, audio, video, and so forth. The Memory 602 may be implemented by any type of volatile or non-volatile Memory device or combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk or optical disk. The multimedia components 603 may include a screen and audio components. Wherein the screen may be, for example, a touch screen and the audio component is used for outputting and/or inputting audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may further be stored in the memory 602 or transmitted through the communication component 605. The audio assembly also includes at least one speaker for outputting audio signals. The I/O interface 604 provides an interface between the processor 601 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 605 is used for wired or wireless communication between the electronic device 600 and other devices. Wireless communication, such as Wi-Fi, bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more of them, which is not limited herein. The corresponding communication component 605 may therefore include: Wi-Fi module, Bluetooth module, NFC module, etc.
in an exemplary embodiment, the electronic Device 600 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable logic devices (plds), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above-described positioning method.
In another exemplary embodiment, a computer readable storage medium comprising program instructions which, when executed by a processor, implement the steps of the positioning method described above is also provided. For example, the computer readable storage medium may be the memory 602 described above comprising program instructions that are executable by the processor 601 of the electronic device 600 to perform the positioning method described above.
Fig. 7 is a block diagram illustrating an electronic device 700 in accordance with another example embodiment. For example, the electronic device 700 may be provided as a server, such as a location management server, a cloud server, and the like, as described above. Referring to fig. 7, an electronic device 700 includes a processor 722, which may be one or more in number, and a memory 732 for storing computer programs that are executable by the processor 722. The computer programs stored in memory 732 may include one or more modules that each correspond to a set of instructions. Further, the processor 722 may be configured to execute the computer program to perform the above-described positioning method.
in addition, the electronic device 700 may also include a power component 726 that may be configured to perform power management of the electronic device 700 and a communication component 750 that may be configured to enable communication, e.g., wired or wireless communication, of the electronic device 700. in addition, the electronic device 700 may also include an input/output (I/O) interface 758. the electronic device 700 may operate based on an operating system stored in memory 732, e.g., Windows Server, Mac XTOSM, UnixTM, L inuxTM, and so on.
In another exemplary embodiment, a computer readable storage medium comprising program instructions which, when executed by a processor, implement the steps of the positioning method described above is also provided. For example, the computer readable storage medium may be the memory 732 described above including program instructions that are executable by the processor 722 of the electronic device 700 to perform the positioning method described above.
In another exemplary embodiment, a computer program product is also provided, which comprises a computer program executable by a programmable apparatus, the computer program having code portions for performing the above-mentioned positioning method when executed by the programmable apparatus.
The preferred embodiments of the present disclosure are described in detail with reference to the accompanying drawings, however, the present disclosure is not limited to the specific details of the above embodiments, and various simple modifications may be made to the technical solution of the present disclosure within the technical idea of the present disclosure, and these simple modifications all belong to the protection scope of the present disclosure.
It should be noted that, in the foregoing embodiments, various features described in the above embodiments may be combined in any suitable manner, and in order to avoid unnecessary repetition, various combinations that are possible in the present disclosure are not described again.
In addition, any combination of various embodiments of the present disclosure may be made, and the same should be considered as the disclosure of the present disclosure, as long as it does not depart from the spirit of the present disclosure.