WO2023036212A1 - 货架定位方法、货架对接方法以及装置、设备、介质 - Google Patents

货架定位方法、货架对接方法以及装置、设备、介质 Download PDF

Info

Publication number
WO2023036212A1
WO2023036212A1 PCT/CN2022/117683 CN2022117683W WO2023036212A1 WO 2023036212 A1 WO2023036212 A1 WO 2023036212A1 CN 2022117683 W CN2022117683 W CN 2022117683W WO 2023036212 A1 WO2023036212 A1 WO 2023036212A1
Authority
WO
WIPO (PCT)
Prior art keywords
shelf
key point
coordinate system
image data
mobile device
Prior art date
Application number
PCT/CN2022/117683
Other languages
English (en)
French (fr)
Inventor
程今
Original Assignee
灵动科技（北京）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 灵动科技（北京）有限公司 filed Critical 灵动科技（北京）有限公司
Priority to CN202280045980.XA priority Critical patent/CN117897348A/zh
Priority to JP2024513089A priority patent/JP2024532356A/ja
Publication of WO2023036212A1 publication Critical patent/WO2023036212A1/zh

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B65—CONVEYING; PACKING; STORING; HANDLING THIN OR FILAMENTARY MATERIAL
    • B65G—TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS
    • B65G1/00—Storing articles, individually or in orderly arrangement, in warehouses or magazines
    • B65G1/02—Storage devices
    • B65G1/04—Storage devices mechanical
    • B65G1/137—Storage devices mechanical with arrangements or automatic control means for selecting which articles are to be removed
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes

Definitions

  • the invention relates to the technical field of logistics, in particular to a shelf positioning method, a shelf docking method, a device, equipment, and a medium.
  • Movable equipment is one of the important components in the automated warehousing and logistics system.
  • the mobile equipment realizes the automatic scheduling and handling of warehousing goods by docking shelves, moving goods, and unloading goods.
  • the jacking rack needs to be positioned.
  • devices such as laser radar, solid-state radar, and depth image (Red Green Blue Depth, RGBD) camera are usually used to collect the depth information of the jacking shelf, and then the insertable side of the jacking shelf is positioned according to the depth information.
  • the depth information collected by the above-mentioned equipment can only reflect the geometric characteristics of the parallel plates installed in the jacking racks, so that there are certain errors in the positioning results for the jacking racks, and the accuracy of the positioning results is poor.
  • the invention provides a shelf positioning method, a shelf docking method, a device, equipment, and a medium, which are used to improve the positioning accuracy of movable equipment and improve the docking efficiency.
  • the present invention provides a shelf positioning method, the method comprising:
  • the image data is input into the key point detection network to extract the first position information of the shelf key point in the image coordinate system from the image data through the key point detection network;
  • the present invention provides a shelf positioning device, comprising:
  • An acquisition module configured to acquire image data in the environment through the image acquisition module in the mobile device
  • the key point detection module is used to input the image data into the key point detection network, so as to extract the first position information of the shelf key point in the image coordinate system from the image data through the key point detection network;
  • the relative pose determining module is used to determine the relative pose of the key points of the shelf relative to the movable device according to the first position information and the conversion relationship between the image coordinate system and the vehicle body coordinate system of the movable device.
  • the present invention provides a shelf docking method, the method comprising:
  • the image data is input into the key point detection network to extract the first position information of the shelf key point in the image coordinate system from the image data through the key point detection network;
  • the movable device determines the relative pose of the key point of the shelf relative to the movable device
  • the present invention provides a shelf docking device, comprising:
  • An acquisition module configured to acquire image data in the environment through the image acquisition module in the mobile device
  • the key point detection module is used to input the image data into the key point detection network, so as to extract the first position information of the shelf key point in the image coordinate system from the image data through the key point detection network;
  • the relative pose determining module is used to determine the relative pose of the shelf key point relative to the movable device according to the first position information and the conversion relationship between the image coordinate system and the vehicle body coordinate system of the movable device;
  • the docking module is configured to determine a docking route between the mobile device and the shelf according to the relative pose, so that the mobile device can execute a docking process based on the docking route.
  • the present invention provides an electronic device, which includes a processor and a memory, wherein executable code is stored in the memory, and when the executable code is executed by the processor, the processor At least the method in the first aspect or the third aspect can be implemented.
  • the present invention provides a non-transitory machine-readable storage medium, on which executable code is stored, when the executable code is executed by a processor of an electronic device , so that the processor can at least implement the method in the first aspect or the third aspect.
  • the image data in the environment is first obtained through the image acquisition module in the mobile device, and then the image data is input into the key point detection network, so as to extract from the image data through the key point detection network
  • the first position information of shelf key points in the image coordinate system is determined. Furthermore, according to the first position information and the conversion relationship between the image coordinate system and the vehicle body coordinate system of the mobile device, the relative pose of the key point of the shelf relative to the mobile device is determined.
  • the visual semantic feature of the shelf (that is, the position information of the key point of the shelf) is extracted from the image data of the environment where the mobile device is located through the key point detection network, and the position of the shelf relative to the mobile device is determined based on the visual semantic feature.
  • the relative pose greatly improves the detection accuracy of the position of the shelf and improves the accuracy of shelf positioning.
  • the embodiment of the present invention can identify the visual semantic features of the shelf through the key point detection network without collecting depth information, which expands the application range of shelf positioning and reduces the hardware cost of shelf positioning.
  • Fig. 1 is a schematic flow chart diagram of a shelf positioning method provided by the present invention
  • FIGS. 2 to 5 are schematic diagrams of a shelf positioning method provided by the present invention.
  • FIG. 6 shows a flow chart of another shelf positioning method provided by an embodiment of the present invention.
  • Fig. 7 is a schematic flow chart of a shelf docking method provided by the present invention.
  • Fig. 8 is a principle schematic diagram of a shelf docking method provided by the present invention.
  • FIG. 9 is a flow chart of another shelf docking method provided by an embodiment of the present invention.
  • Fig. 10 is a structural schematic diagram of a shelf positioning device provided by the present invention.
  • Fig. 11 is a structural schematic diagram of a shelf docking device provided by the present invention.
  • FIG. 12 is a schematic structural diagram of an electronic device provided by the present invention.
  • the words “if”, “if” as used herein may be interpreted as “at” or “when” or “in response to determining” or “in response to detecting”.
  • the phrases “if determined” or “if detected (the stated condition or event)” could be interpreted as “when determined” or “in response to the determination” or “when detected (the stated condition or event) )” or “in response to detection of (a stated condition or event)”.
  • the movable device is one of the important components in the automatic warehouse logistics system.
  • the mobile equipment realizes the automatic scheduling and handling of warehousing goods by docking shelves, moving goods, and unloading goods.
  • the jacking rack needs to be positioned before the movable equipment docks with the jacking rack.
  • laser radar, solid-state radar, RGBD camera and other equipment are usually used to collect the depth information of the jacking shelf, and then the insertable side of the jacking shelf is positioned according to the depth information.
  • the depth information collected by the above-mentioned equipment can only reflect the geometric characteristics of the parallel plates installed in the jacking racks, so that there are certain errors in the positioning results for the jacking racks, and the accuracy of the positioning results is poor.
  • the image data in the environment is acquired through the image acquisition module in the mobile device, and then the image data is input into the key point detection network to extract the shelf key points from the image data through the key point detection network in the image coordinates
  • the first position information in the system is determined. Furthermore, according to the first position information and the conversion relationship between the image coordinate system and the vehicle body coordinate system of the mobile device, the relative pose of the key point of the shelf relative to the mobile device is determined.
  • the visual semantic features of the shelf (such as the first position information of the key points of the shelf) are extracted from the image data of the environment where the mobile device is located through the key point detection network, and the position of the shelf relative to the mobile device is determined based on the visual semantic feature.
  • the relative pose greatly improves the detection accuracy of the position of the shelf and improves the accuracy of shelf positioning.
  • the scheme can identify the visual semantic features of the shelf through the key point detection network without collecting depth information, which expands the application range of shelf positioning and reduces the hardware cost of shelf positioning.
  • Fig. 1 is a flowchart of a shelf positioning method provided by an embodiment of the present invention. As shown in Fig. 1, the method includes the following steps:
  • the relative pose of the key point of the shelf relative to the movable device According to the first position information and the conversion relationship between the image coordinate system and the vehicle body coordinate system of the mobile device, determine the relative pose of the key point of the shelf relative to the movable device.
  • the mobile device may be an autonomous mobile robot (Automated Mobile Robot, AMR), a cargo vehicle, and the like.
  • AMR Automated Mobile Robot
  • an autonomous mobile robot refers to a device capable of highly autonomous spatial movement in a working environment. For example, warehousing collaborative robots, picking robots or handling robots.
  • the autonomous mobile robot is provided with a semantic map corresponding to its environment, and the semantic map refers to an environment map including semantic information of multiple objects in the environment.
  • the semantic information of an object refers to the information that can understand and explain what the object is or its category with the help of natural language, for example, it may include but not limited to the name, shape, location, etc. of the object.
  • the semantic map includes the location, type, and size of each shelf in the warehouse, as well as the location, type, and size of various obstacles (such as railings, steps, thresholds, etc.) in the warehouse.
  • the image acquisition module of the mobile device may be a sensor module capable of image acquisition.
  • the image acquisition module is a wide-angle monocular camera for capturing color (Red Green Blue, RGB) images. It can be understood that, in practical applications, RGB images include pictures or videos.
  • the method provided by the embodiment of the present invention may be realized by control and dispatching end devices of multiple mobile devices, or may be realized by multiple mobile devices.
  • the control and scheduling terminals of multiple mobile devices can be set in the cloud service center, or in one of the mobile devices, or in other forms of computing equipment, which is not limited by the present invention.
  • the method provided by the embodiments of the present invention can be applied to various scenarios, for example, it can be applied to warehousing scenarios, logistics sorting scenarios, material distribution scenarios, port freight scenarios, and the like.
  • the specific implementation manner of the embodiment of the present invention is introduced below by taking a storage scenario as an example. Other scenarios may be implemented with reference to the implementation manner of a storage scenario, and details are not repeated here.
  • the warehousing scene refers to the scene where goods are stored in spaces such as warehouses, warehouses, and warehouses.
  • the storage scenario it includes stored goods and multiple shelves for storing goods.
  • the shelf includes shelf laminates and shelf frames.
  • the image data of the environment where the mobile device is located can be collected through the image acquisition module.
  • the environment where the movable device is located refers to a part of the storage scene near the movable device.
  • the range in which the mobile device can collect image data in the storage scene is related to the field of view of the image collection module. The larger the field of view of the image collection module, the larger the range of image data it can collect.
  • the key point detection network can be used to detect the key points of the image data of the storage scene, so as to detect the key points of the shelf in the image data. Taking the storage scene shown in Figure 2 as an example, the mobile device shoots the shelf through the image acquisition module, passes the captured image data through the key point detection network, and then detects the shelf key contained in the image data through the key point detection network. point as shown in Figure 3.
  • the first position information of the shelf key points in the image coordinate system is extracted from the image data through the key point detection network.
  • the position of the upper left corner vertex, lower left corner vertex, lower right corner vertex, and upper right corner vertex of the shelf is marked through the key point detection network, and the corresponding key point 1 and key point 2 are respectively marked in the image data , key point 3, and key point 4 to obtain the first position information of the four shelf key points shown in FIG. 4 .
  • the first position information of the key points of the shelf according to the first position information of the key points of the shelf and the conversion relationship between the image coordinate system and the vehicle body coordinate system of the mobile device, determine that the key point of the shelf is relatively movable The relative pose of the device. That is, the relative pose of the shelf relative to the mobile device.
  • the position of the key point of the shelf is detected by the visual feature extraction technology, and the detection speed is fast, and the visual feature extracted by the key point detection network is not easily affected by other interference factors, and the depth information can also be avoided.
  • the false detection situation brought about provides a basis for the positioning scheme of the mobile device to get rid of the dependence on the depth information, which is conducive to improving the positioning accuracy and flexibility, and greatly expanding the application range of the shelf positioning method.
  • any model with image detection (or recognition) function can be used as the key point detection network involved in the above steps, which is not limited in the embodiment of the present invention.
  • various neural network models can be used as the key point detection network, such as convolutional neural network (Convolutional Neural Network, CNN).
  • CNN convolutional Neural Network
  • the key points of the shelf are marked in the image data containing the shelf, and then the convolutional neural network is trained using the marked image data to obtain a convolutional neural network for detecting key points of the shelf.
  • a mobile device is set in a specific application scenario.
  • the specific application scenario includes shelves for storing goods.
  • the shelf may be a jacked shelf as shown in FIG. 2 .
  • a wide-angle monocular camera is mounted on a mobile device.
  • the process of acquiring image data in the environment through the image acquisition module in the mobile device in 101 can be implemented as: acquiring RGB images containing shelf images through the wide-angle monocular camera of the mobile device.
  • the mobile device moves to the shelf, takes the shelf as a shooting target, and shoots with a wide-angle monocular camera to obtain an RGB image including the shelf.
  • the image data in the environment where the mobile device is located can also be acquired in other ways.
  • the surrounding area is scanned by a wide-angle monocular camera to obtain RGB video corresponding to the surrounding area.
  • the goal is to obtain two-dimensional image data including shelf images. So as to provide the basis for the subsequent detection of the key points of the shelf.
  • the image acquisition module is a wide-angle monocular camera, and the image data may be distorted due to the wide-angle shooting function.
  • each pixel in the image data is calculated according to the focal length parameter and the distortion parameter of the image acquisition module Coordinates are corrected for distortion.
  • step 102 input the distortion-corrected image data into the key point detection network, so as to extract the first position information of the shelf key point in the image coordinate system from the image data through the key point detection network.
  • the overall outline of the shelf can be constructed through the detected key points of the shelf, thereby providing a basis for subsequent acquisition of the relative pose of the key points of the shelf relative to the movable device where the image acquisition module is located.
  • the first position information is the pixel coordinates corresponding to the key points of the shelf in the two-dimensional image coordinate system.
  • the shelf key points of a certain shelf are recorded as key point 1, key point 2, key point 3, and key point 4.
  • the pixel coordinates of the above four shelf key points in the shelf picture are detected. Specifically, read the pixel coordinates of the key points of the shelf from the key point detection network in turn, and record them as the upper left corner vertex (x1, y1), the lower left corner vertex (x2, y2), the lower right corner vertex (x3, y3), the upper right corner vertex (x4, y4).
  • the output of the key point detection network can also be set according to the actual application.
  • the key points of the shelf in the embodiment of the present invention may be pre-marked according to the actual shape of the shelf.
  • the position of the key point of the shelf is pre-marked in the image data containing the shelf, and the calibrated image data is used as the training sample of the key point detection network.
  • a shelf picture and the two-dimensional coordinates of shelf key points in the shelf picture are used as a set of training samples.
  • the shelf included in the picture of the shelf can be complete or partial.
  • the number threshold corresponding to the key points of the shelf is set in advance. Furthermore, compare the key points to detect whether the number of shelf key points extracted by the network is consistent with the number threshold. If they are not consistent, it means that all the key points of the shelf have not been extracted. In this case, the position of the movable device can be adjusted so that the movable device can move to a position where all the key points of the shelf can be photographed. If they are consistent, it means that all the shelf key points have been extracted. In this case, step 103 can be executed.
  • the key point of the shelf relative to the mobile device determines the key point of the shelf relative to the mobile device.
  • the relative pose process of can be realized as:
  • the camera parameters of the image acquisition module and the first position information calculate the second position information of the key points of the shelf in the vehicle body coordinate system; based on the relationship between the image coordinate system and the vehicle body coordinate system of the movable device The conversion relationship and the second position information are used to calculate the relative pose of the key point of the shelf relative to the movable device.
  • the conversion relationship between the image coordinate system and the vehicle body coordinate system of the movable device is obtained based on the relative positional relationship between the image acquisition module and the movable device.
  • the vehicle body coordinate system of the mobile device refers to the three-dimensional coordinate system in the physical world with the vehicle body as the origin.
  • the shelf parameters include shelf dimensions.
  • the shelf size includes the length, width and height of the shelf.
  • the camera parameters include a focal length parameter.
  • the process of calculating the second position information of the key points of the shelf in the vehicle body coordinate system includes: according to the shelf size and The focal length parameter converts the two-dimensional coordinates of the key points of the shelf in the image coordinate system to the car body coordinate system, and obtains the three-dimensional coordinates of the key points of the shelf in the car body coordinate system.
  • O is the origin of the image coordinate system
  • O 1 is assumed to be the origin of the vehicle body coordinate system.
  • the two-dimensional coordinates P(u 1 , v 1 ) of the key points of the shelf can be mapped from the image coordinate system to the car body coordinate system, and transformed into the corresponding three-dimensional coordinates P(x c , y c , z c ).
  • the shelf dimensions including width and height are denoted as z c .
  • the focal length parameter is (f x , f v ).
  • the movable device further includes: 2.
  • the location information is corrected.
  • the inherent structural features of the shelf refer to the specific geometric features based on the shelf structure.
  • the inherent structural features of the shelf include but are not limited to: two parallel vertical frames of the shelf, and/or two parallel parallel horizontal frames of the shelf.
  • the relative pose of the key point of the shelf relative to the mobile device is calculated. This relative pose is used for subsequent positioning and insertion of the shelf.
  • the visual semantic features of the shelf are extracted from the image data of the environment where the mobile device is located through the key point detection network, and the relative pose of the shelf relative to the mobile device is determined based on the visual semantic feature, which greatly improves The detection accuracy of the position of the shelf is improved, and the accuracy of shelf positioning is improved.
  • the embodiment of the present invention can identify the visual semantic features of the shelf through the key point detection network without collecting depth information, greatly expand the application range of shelf positioning, and reduce the hardware cost of shelf positioning.
  • the above-mentioned image acquisition module can also be used to cooperate with the laser radar at the same time, so that the relative coordinates obtained by the image data acquired by the image acquisition module can be compared with the point cloud data acquired by the laser radar. fusion. Therefore, the point cloud data acquired by the lidar can be used as a supplement to the visual image data, so as to perform additional corrections to prevent excessive deviation of the positioning results of the visual data and improve the positioning accuracy of the shelves.
  • Fig. 6 shows a flow chart of another shelf positioning method provided by an embodiment of the present invention. As shown in FIG. 6 , in addition to steps 101 , 102 and 103 , the method may further include steps 104 , 105 and 106 . The detailed execution process and technical effect of steps 101, 102 and 103 have been described in detail above, and will not be repeated here.
  • the lidar point cloud data in the environment is acquired through the lidar in the mobile device.
  • the mobile device in addition to being equipped with an image acquisition module such as a wide-angle monocular camera, it may also be equipped with a laser radar for collecting point cloud data in the environment where the mobile device is located.
  • step 105 post-processing is performed on the coordinates of the shelf key points relative to the movable equipment.
  • the coordinates of the key points of the shelf relative to the movable device have been obtained through coordinate transformation.
  • the prior knowledge of the shelf can be used to further correct the 3D coordinates of the key points of the shelf relative to the movable device.
  • the vertical side of the shelf is perpendicular to the ground, and the horizontal side is horizontal to the ground.
  • the three-dimensional coordinates of the key points of the shelf in the car body coordinate system are recorded as the upper left corner vertex (x1, y1, z1), the lower left corner vertex (x2, y2, z2), the lower right corner vertex (x3, y3 , z3), upper right corner vertex (x4, y4, z4).
  • the coordinates after further correction in step 105 are:
  • the post-processed coordinates are fused with the lidar point cloud data to update the relative pose of the key points of the shelf relative to the movable device.
  • the width wid of the shelf can be calculated from the coordinates of the lower left corner and the lower right corner (or the upper left corner and the upper right corner) corrected in step 105 .
  • the point cloud of the single-line lidar since the point cloud of the single-line lidar is distributed on a plane parallel to the ground, it can be in a circle with (x2, z2) as the center and wid*0.1 as the radius Find the point cloud within the shape area.
  • using the laser point cloud data detected by the laser radar as a supplement to the image data acquired by the image acquisition module can prevent excessive deviations in the positioning results of the image data, thereby improving the positioning accuracy of the shelves.
  • Fig. 7 is a flow chart of a shelf docking method provided by an embodiment of the present invention. As shown in Fig. 7, the method includes the following steps:
  • the image data of the environment where the mobile device is located can be collected through the image acquisition module. Since the larger the field of view of the image acquisition module, the larger the range of the image data it collects, therefore, a wide-angle monocular camera is optionally used in the present invention.
  • the key point detection can be performed on the image data of the storage scene through the key point detection network, so as to detect the key points of the shelf in the image data.
  • the mobile device shoots the shelf through the image acquisition module, passes the captured image data through the key point detection network, and then detects the shelf key contained in the image data through the key point detection network. point as shown in Figure 3. Specifically, the first position information of the shelf key points in the image coordinate system is extracted from the image data through the key point detection network.
  • the position of the upper left corner vertex, lower left corner vertex, lower right corner vertex, and upper right corner vertex of the shelf is marked through the key point detection network, and the corresponding key point 1 and key point 2 are respectively marked in the image data , key point 3, and key point 4 to obtain the first position information of the four shelf key points shown in FIG. 4 .
  • the first position information of the key points of the shelf according to the first position information of the key points of the shelf and the conversion relationship between the image coordinate system and the vehicle body coordinate system of the mobile device, determine that the key point of the shelf is relatively movable
  • the relative pose of the device After determining the relative pose of the key points of the shelf relative to the mobile device, the docking route between the mobile device and the shelf is determined according to the relative pose, so that the mobile device can execute the docking process based on the docking route.
  • the visual semantic features of the shelf are extracted from the image data of the environment where the mobile device is located through the key point detection network, and the position information of the shelf relative to the mobile device is determined based on the visual semantic feature. relative pose. Furthermore, on the basis of the relative pose of the shelf relative to the mobile device, a docking route between the mobile device and the shelf is planned, so that the mobile device performs a docking process based on the docking route.
  • the docking between the movable device and the shelf is realized, the detection accuracy of the location of the shelf is greatly improved, and the positioning of the shelf and the accuracy of the docking of the shelf are improved.
  • the mobile device is further navigated to a docking position that matches the shelf to be docked, and the docking position is a position for collecting at least one key point of the shelf in the environment.
  • the docking position of each shelf can be calibrated in the warehouse environment map according to the detection results of historical key points, so that the follow-up can be quickly positioned, and the movable equipment can be navigated to the docking position, improving the efficiency of shelf positioning and docking.
  • the optional docking positions of each shelf are marked with triangle marks according to the detection results of historical key points.
  • the process of acquiring image data in the environment through the image acquisition module in the mobile device in 601 includes: capturing image data in the environment through the image acquisition module at the docking position.
  • the mobile device can reach the position where the key points of the shelf can be photographed faster, which can not only help improve the accuracy of shelf positioning, but also shorten the time spent in the shelf docking process and improve the shelf docking efficiency.
  • the preset condition is whether the number of key points extracted from the shelf reaches a set number threshold. Based on this, it is judged whether the number of shelf key points extracted by the key point detection network reaches the set number threshold. If the number of shelf key points does not reach the set number threshold, it means that the mobile device has not collected all the shelf key points. To put it simply, the angle at which the movable device captures the image of the shelf does not reach the preset angle, and the position of the movable device does not reach the preset position. In this case, the mobile device is controlled to move around the shelf to move to the preset set location. During the moving process of the mobile device, steps 601 to 602 are re-executed until the number of shelf key points extracted by the key point detection network reaches a set threshold, and it can be determined that the mobile device has reached the preset position.
  • the quality of image data, acquisition angle, etc. can be further screened through preset conditions, so as to improve the accuracy of shelf positioning results and docking results.
  • the movable device can be controlled to reach the preset position (that is, the position where the key points of the complete shelf can be collected), thereby assisting in improving the accuracy of shelf positioning and docking, shortening the time spent in the shelf docking process, and improving shelf docking. efficiency.
  • re-executing the step of determining the relative pose to correct the docking route includes: re-executing steps 602 to 603 to obtain the relative pose of the key point of the shelf relative to the mobile device; The calculated relative pose, re-determine the docking route between the mobile device and the shelf, so that the mobile device can correct the docking process based on the re-determined docking route.
  • the relative pose can be recalculated, so that when the above problems occur, the docking route can be re-determined to correct the docking process, thereby further improving the accuracy of shelf docking, shortening the time spent in the shelf docking process, and improving shelf docking efficiency.
  • the above-mentioned image acquisition module can also be used to cooperate with the laser radar at the same time, so that the relative coordinates obtained by the image data acquired by the image acquisition module can be compared with the point cloud data acquired by the laser radar. fusion.
  • Fig. 9 is a flow chart of another shelf docking method provided by the embodiment of the present invention. As shown in Fig. 9, in addition to steps 601, 602, 603 and 604, between steps 603 and 604, the method may also include Steps 605, 606 and 607. The detailed execution process and technical effect of steps 601, 602, 603 and 604 have been described in detail above, and will not be repeated here.
  • steps 605, 606, and 607 are similar to those of steps 104, 105, and 106 respectively, and will not be repeated here.
  • Fig. 10 is a schematic structural diagram of a shelf positioning device provided by an embodiment of the present invention. As shown in Fig. 10, the device includes:
  • An acquisition module 81 configured to acquire image data in the environment through the image acquisition module in the mobile device
  • a key point detection module 82 configured to input the image data into the key point detection network, so as to extract the first position information of the shelf key point in the image coordinate system from the image data through the key point detection network;
  • the relative pose determination module 83 is configured to determine the key point of the shelf relative to the movable device according to the first position information and the conversion relationship between the image coordinate system and the vehicle body coordinate system of the mobile device. The relative pose of the mobile device.
  • the relative pose determination module 83 determines the position of the shelf key point relative to the movable device according to the first position information and the conversion relationship between the image coordinate system and the vehicle body coordinate system. For relative pose, it is used for:
  • the conversion relationship is obtained based on the relative position relationship between the image acquisition module and the movable device.
  • the shelf parameter includes a shelf size
  • the camera parameter includes a focal length parameter
  • the relative pose determining module 83 calculates the second position information of the key points of the shelf in the vehicle body coordinate system according to the preset shelf parameters, the camera parameters of the image acquisition module and the first position information. , for:
  • the focal length parameter transform the two-dimensional coordinates of the key points of the shelf in the image coordinate system into the coordinate system of the car body, and obtain the three-dimensional coordinates of the key points of the shelf in the coordinate system of the car body coordinate.
  • the device further includes a correction module, configured to:
  • the inherent structural features include: the vertical frames of the shelves are parallel to each other and/or the horizontal frames of the shelves are parallel to each other.
  • the device also includes a distortion correction module for:
  • distortion correction is carried out to each pixel point coordinate in the image data according to the focal length parameter and the distortion parameter of the image acquisition module.
  • the shelf key points include any one or more of the following: the upper left vertex, the upper right vertex, the lower left vertex, and the lower right vertex of the shelf.
  • the device shown in FIG. 10 can execute the shelf positioning method provided in the embodiments shown in FIGS. 1 to 6 .
  • the device shown in FIG. 10 can execute the shelf positioning method provided in the embodiments shown in FIGS. 1 to 6 .
  • the detailed execution process and technical effects refer to the descriptions in the previous embodiments, which will not be repeated here.
  • Fig. 11 is a schematic structural diagram of a shelf docking device provided by an embodiment of the present invention. As shown in Fig. 11, the device includes:
  • An acquisition module 91 configured to acquire image data in the environment through the image acquisition module in the mobile device
  • a key point detection module 92 configured to input the image data into the key point detection network, so as to extract the first position information of the shelf key point in the image coordinate system from the image data through the key point detection network;
  • the relative pose determination module 93 is configured to determine the key point of the shelf relative to the movable device according to the first position information and the conversion relationship between the image coordinate system and the vehicle body coordinate system of the mobile device.
  • the relative pose of the mobile device is configured to determine the key point of the shelf relative to the movable device according to the first position information and the conversion relationship between the image coordinate system and the vehicle body coordinate system of the mobile device.
  • the docking module 94 is configured to determine a docking route between the mobile device and the shelf according to the relative pose, so that the mobile device executes a docking process based on the docking route.
  • the device also includes a navigation module for:
  • the docking position is a position in the environment for collecting at least one key point of the shelf;
  • the acquisition module 91 acquires the image data in the environment through the image acquisition module in the mobile device, it is used for:
  • the image data in the environment is captured by the image acquisition module.
  • the device also includes a judging module, configured to:
  • the image data in the environment is reacquired through the image acquisition module, and the key point detection network is used to extract the shelf key points from the image data in the image coordinate system In the step of the first position information, until the shelf key points extracted by the key point detection network meet the preset conditions.
  • the device also includes an orthotic module for:
  • the image data in the environment is reacquired through the image acquisition module, and the step of determining the relative pose is re-executed to correct the docking process.
  • the correction module re-executes the step of determining the relative pose to correct the docking route for:
  • the image acquisition module is a wide-angle monocular camera.
  • the device shown in FIG. 11 can execute the rack docking method provided in the embodiments shown in FIGS. 7 to 9.
  • the structure of the apparatus shown in FIG. 10 or FIG. 11 may be implemented as an electronic device.
  • the electronic device may include: a processor 1001 and a memory 1002 .
  • executable codes are stored on the memory 1002, and when the executable codes are executed by the processor 1001, the processor 1001 can at least implement Methods.
  • the electronic device may further include a communication interface 1003 for communicating with other devices.
  • an embodiment of the present invention provides a non-transitory machine-readable storage medium, the non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by the processor of the electronic device , so that the processor can at least implement the methods provided in the embodiments shown in FIGS. 1 to 9 .
  • the shelf positioning method provided by the embodiment of the present invention can be executed by a certain program/software, which can be provided by the network side, and the electronic device mentioned in the foregoing embodiments can download the program/software to a local non-volatile In the volatile storage medium, and when it needs to perform the aforementioned shelf positioning method, the program/software is read into the memory by the CPU, and then the program/software is executed by the CPU to realize the shelf positioning method provided in the aforementioned embodiments , the execution process can refer to the schematic diagrams in the aforementioned FIG. 1 to FIG. 6 .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Mechanical Engineering (AREA)
  • Image Analysis (AREA)
  • Warehouses Or Storage Devices (AREA)

Abstract

本发明提供一种货架定位方法、货架对接方法以及装置、设备、介质，该方法包括：通过可移动设备中的图像采集模块获取所处环境中的图像数据；将该图像数据输入到关键点检测网络中，以通过关键点检测网络从图像数据中提取货架关键点在图像坐标系中的第一位置信息；根据第一位置信息、以及图像坐标系与车体坐标系之间的转换关系，确定货架关键点相对可移动设备的相对位姿。该方法，通过关键点检测网络从可移动设备所处环境的图像数据中提取出货架视觉语义特征，并基于视觉语义特征确定货架相对可移动设备的相对位姿，大大提升对货架所处位置的检测精度，提高货架定位的准确性，扩展了货架定位的应用范围，降低了货架定位的硬件成本。

Description

货架定位方法、货架对接方法以及装置、设备、介质 技术领域
本发明涉及物流技术领域，尤其涉及一种货架定位方法、货架对接方法以及装置、设备、介质。
背景技术
可移动设备是自动化仓储物流系统中的重要组成部分之一。在仓储物流场景中，可移动设备通过对接货架、搬运货物、卸下货物来实现仓储货物的自动化调度搬运。
以顶升货架为例，可移动设备对接顶升货架之前，需要对顶升货架进行定位。相关技术中，通常是采用激光雷达、固态雷达、深度图像(Red Green Blue Depth，RGBD)相机等设备采集顶升货架的深度信息，进而根据深度信息对顶升货架可插取侧进行定位。但是，上述设备所采集的深度信息往往只能反映顶升货架中安装的平行板的几何特征，使得针对顶升货架的定位结果存在一定误差，定位结果准确性较差。
发明内容
本发明提供一种货架定位方法、货架对接方法以及装置、设备、介质，用以提高可移动设备的定位准确性，提升对接效率。
第一方面，本发明提供一种货架定位方法，该方法包括：
通过可移动设备中的图像采集模块获取所处环境中的图像数据；
将图像数据输入到关键点检测网络中，以通过关键点检测网络从图像数据中提取货架关键点在图像坐标系中的第一位置信息；
根据第一位置信息、以及图像坐标系与可移动设备的车体坐标系之间的转换关系，确定货架关键点相对可移动设备的相对位姿。
第二方面，本发明提供一种货架定位装置，包括：
获取模块，用于通过可移动设备中的图像采集模块获取所处环境中的图像数据；
关键点检测模块，用于将图像数据输入到关键点检测网络中，以通过关键点检测网络从图像数据中提取货架关键点在图像坐标系中的第一位置信息；
相对位姿确定模块，用于根据第一位置信息、以及图像坐标系与可移动设备的车体坐标系之间的转换关系，确定货架关键点相对可移动设备的相对位姿。
第三方面，本发明提供一种货架对接方法，该方法包括：
通过可移动设备中的图像采集模块获取所处环境中的图像数据；
将图像数据输入到关键点检测网络中，以通过关键点检测网络从图像数据中提取货架关键点在图像坐标系中的第一位置信息；
根据第一位置信息、以及图像坐标系与可移动设备的车体坐标系之间的转换关系，确定货架关键点相对可移动设备的相对位姿；
根据相对位姿确定可移动设备与货架之间的对接路线，以使可移动设备基于对接路线执行对接流程。
第四方面，本发明提供一种货架对接装置，包括：
获取模块，用于通过可移动设备中的图像采集模块获取所处环境中的图像数据；
关键点检测模块，用于将图像数据输入到关键点检测网络中，以通过关键点检测网络从图像数据中提取货架关键点在图像坐标系中的第一位置信息；
相对位姿确定模块，用于根据第一位置信息、以及图像坐标系与可移动设备的车体坐标系之间的转换关系，确定货架关键点相对可移动设备的相对位姿；
对接模块，用于根据相对位姿确定可移动设备与货架之间的对接路线，以使可移动设备基于对接路线执行对接流程。
第五方面，本发明提供一种电子设备，其中包括处理器和存储器，其中， 所述存储器上存储有可执行代码，当所述可执行代码被所述处理器执行时，使所述处理器至少可以实现第一方面或第三方面中的方法。
第六方面，本发明提供了一种非暂时性机器可读存储介质，所述非暂时性机器可读存储介质上存储有可执行代码，当所述可执行代码被电子设备的处理器执行时，使所述处理器至少可以实现第一方面或第三方面中的方法。
在本发明实施例中，首先通过可移动设备中的图像采集模块获取所处环境中的图像数据，然后将该图像数据输入到关键点检测网络中，以通过关键点检测网络从图像数据中提取货架关键点在图像坐标系中的第一位置信息。进而，根据第一位置信息、以及图像坐标系与可移动设备的车体坐标系之间的转换关系，确定货架关键点相对可移动设备的相对位姿。
本发明实施例中，通过关键点检测网络从可移动设备所处环境的图像数据中提取出货架视觉语义特征(即货架关键点的位置信息)，并基于视觉语义特征确定货架相对可移动设备的相对位姿，大大提升了对货架所处位置的检测精度，提高货架定位的准确性。并且，本发明实施例无需采集深度信息即可通过关键点检测网络识别出货架的视觉语义特征，扩展了货架定位的应用范围，降低了货架定位的硬件成本。
附图说明
为了更清楚地说明本发明中的技术方案，下面将对本发明描述中所需要使用的附图作一简单地介绍，显而易见地，下面描述中的附图是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本发明提供的一种货架定位方法的流程图示意图；
图2至图5为本发明提供的一种货架定位方法的原理示意图；
图6示出了本发明实施例提供的另一种货架定位方法的流程图；
图7为本发明提供的一种货架对接方法的流程示意图；
图8为本发明提供的一种货架对接方法的原理示意图；
图9为本发明实施例提供的另一种货架对接方法的流程图；
图10为本发明提供的一种货架定位装置的结构示意图；
图11为本发明提供的一种货架对接装置的结构示意图；
图12为本发明提供的一种电子设备的结构示意图。
具体实施方式
为使本发明实施例的目的、技术方案和优点更加清楚，下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
在本发明实施例中使用的术语是仅仅出于描述特定实施例的目的，而非旨在限制本发明。在本发明实施例和所附权利要求书中所使用的单数形式的“一种”、“所述”和“该”也旨在包括多数形式，除非上下文清楚地表示其他含义，“多种”一般包含至少两种。
取决于语境，如在此所使用的词语“如果”、“若”可以被解释成为“在……时”或“当……时”或“响应于确定”或“响应于检测”。类似地，取决于语境，短语“如果确定”或“如果检测(陈述的条件或事件)”可以被解释成为“当确定时”或“响应于确定”或“当检测(陈述的条件或事件)时”或“响应于检测(陈述的条件或事件)”。
另外，下述各方法实施例中的步骤时序仅为一种举例，而非严格限定。
相关技术中，可移动设备是自动化仓储物流系统中的重要组成部分之一。在仓储物流场景中，可移动设备通过对接货架、搬运货物、卸下货物来实现仓储货物的自动化调度搬运。
以顶升货架为例，可移动设备对接顶升货架之前，需要对顶升货架进行定位。相关技术中，通常是采用激光雷达、固态雷达、RGBD相机等设备采集顶升货架的深度信息，进而根据深度信息对顶升货架可插取侧进行定位。 但是，上述设备所采集的深度信息往往只能反映顶升货架中安装的平行板的几何特征，使得针对顶升货架的定位结果存在一定误差，定位结果准确性较差。
为了解决上述问题，本发明实施例提供的技术方案的核心思想是：
首先，通过可移动设备中的图像采集模块获取所处环境中的图像数据，然后将该图像数据输入到关键点检测网络中，以通过关键点检测网络从图像数据中提取货架关键点在图像坐标系中的第一位置信息。进而，根据第一位置信息、以及图像坐标系与可移动设备的车体坐标系之间的转换关系，确定货架关键点相对可移动设备的相对位姿。
该方案中，通过关键点检测网络从可移动设备所处环境的图像数据中提取出货架视觉语义特征(如货架关键点的第一位置信息)，并基于视觉语义特征确定货架相对可移动设备的相对位姿，大大提升了对货架所处位置的检测精度，提高货架定位的准确性。并且，该方案无需采集深度信息即可通过关键点检测网络识别出货架的视觉语义特征，扩展了货架定位的应用范围，降低了货架定位的硬件成本。
在介绍了技术方案的核心思路之后，下面具体介绍本发明的各种非限制性实施例。
图1为本发明实施例提供的一种货架定位方法的流程图，如图1所示，该方法包括如下步骤：
101、通过可移动设备中的图像采集模块获取所处环境中的图像数据。
102、将图像数据输入到关键点检测网络中，以通过关键点检测网络从图像数据中提取货架关键点在图像坐标系中的第一位置信息。
103、根据第一位置信息、以及图像坐标系与可移动设备的车体坐标系之间的转换关系，确定货架关键点相对可移动设备的相对位姿。
在本发明实施例中，可移动设备可以是自主移动机器人(Automated Mobile Robot，AMR)、载货车辆等。具体来说，自主移动机器人是指在工作环境中能够高度自主地进行空间移动的设备。例如，仓储协作机器人、拣货机器人或 者搬运机器人等。
以自主移动机器人为例，自主移动机器人中设置有所处环境对应有语义地图，该语义地图指的是包含所处环境中多个对象的语义信息的环境地图。对象的语义信息是指能够借助自然语言去领会和解释对象是什么或所属种类等信息，例如可以包含但不限于对象的名称、形状、位置等。例如，在仓储环境中，语义地图包括仓库中各个货架的位置、类型、尺寸，以及仓库中各种障碍物(如栏杆、台阶、门槛等)的位置、类型、尺寸。
本发明实施例中，可移动设备的图像采集模块，可以是具有图像采集能力的传感器模组。例如，图像采集模块是用于拍摄彩色(Red Green Blue，RGB)图像的广角单目相机。可以理解的是，实际应用中，RGB图像包括图片或视频。
本发明实施例提供的方法可以由多个可移动设备的控制调度端设备实现，也可由多个可移动设备实现。实际应用中，多个可移动设备的控制调度端可以设置在云端服务中心，也可以是设置在其中一台可移动设备中，还可以设置在其他形式的计算设备中，本发明并不限定。
在实际应用中，本发明实施例提供的方法可以应用于多种场景中，例如可以应用于仓储场景、物流分拣场景、物料配送场景、港口货运场景等。下面以仓储场景为例介绍本发明实施例的具体实施方式，其他场景可以参照仓储场景的实施方式实施，在此不再赘述。
以仓储场景为例，仓储场景指的是在如仓库、货仓、栈房等空间中储存货品的场景。在仓储场景中，包含存放的货品、以及用于存放货品的多个货架。进一步地，货架中包括货架层板以及货架边框。
在仓储场景中，可移动设备的移动过程中，通过图像采集模块能够采集可移动设备所处环境中的图像数据。可移动设备所处环境指的是可移动设备附近仓储场景中的部分区域。可移动设备在所处仓储场景中能够采集图像数据的范围与图像采集模块的视场有关，图像采集模块的视场角越大，其采集的图像数据的范围也就越大。在采集到所处仓储场景中的图像数据之后，可以通过关键点检测网络对所处仓储场景的图像数据进行关键点检测，以在图像数据中检测 出货架关键点。以图2示出的仓储场景为例，可移动设备通过图像采集模块对货架进行拍摄，将拍摄得到的图像数据经过关键点检测网络，进而通过关键点检测网络检测出图像数据中包含的货架关键点如图3所示。
具体地，通过关键点检测网络从图像数据中提取货架关键点在图像坐标系中的第一位置信息。如图3所示，通过关键点检测网络对货架的左上角顶点、左下角顶点、右下角顶点、右上角顶点所处位置进行标记，分别在图像数据中标出对应的关键点1、关键点2、关键点3、关键点4，得到图4所示的4个货架关键点的第一位置信息。进一步，在获取货架关键点的第一位置信息之后，根据货架关键点的第一位置信息、以及图像坐标系与可移动设备的车体坐标系之间的转换关系，确定货架关键点相对可移动设备的相对位姿。也就是，货架相对可移动设备的相对位姿。
在本发明实施例中，通过视觉特征提取技术检测出货架关键点所处位置，检测速度快，且通过关键点检测网络提取出的视觉特征不容易受其他干扰因素的影响，还能够避免深度信息带来的误检情况，为可移动设备的定位方案摆脱对深度信息的依赖提供了基础，有利于提高定位准确性和灵活性，大大拓展货架定位方法的应用范围。
实际应用中，凡是具有图像检测(或识别)功能的模型均可用于作为上述步骤中涉及的关键点检测网络，本发明实施例对此不做限定。优选地，可以采用各类神经网络模型作为关键点检测网络，例如卷积神经网络(Convolutional Neural Network，CNN)。具体地，在包含货架的图像数据中标注出货架关键点，接着，采用标注后的图像数据对卷积神经网络进行训练，即可得到用于货架关键点检测的卷积神经网络。
下面结合附图介绍图1示出的各个步骤的具体实现方式。
假设具体应用场景中设置有可移动设备。假设具体应用场景中包含用于存储货品的货架。实际应用中，货架可以是如图2示出的顶升货架。假设可移动设备中搭载有广角单目相机。
基于上述假设，可选地，101中通过可移动设备中的图像采集模块获取所处 环境中的图像数据的过程可以实现为：通过可移动设备的广角单目相机获取包含货架图像的RGB图像。
具体地，如图2所示，在一可选实施例中，可移动设备移动到货架前，以货架为拍摄目标，通过广角单目相机进行拍摄，得到包含货架的RGB图像。
除此之外，本发明实施例中还可通过其他方式获取可移动设备所处环境中的图像数据。例如，另一实施例中，在可移动设备移动过程中，通过广角单目相机扫描周围区域，得到周围区域对应的RGB视频。将RGB视频输入图像识别网络中，以通过图像识别网络从RGB视频中提取包含目标货架的图像帧，作为包含货架图像的RGB图像。
可以理解的是，无论采用何种图像采集方式，其目标均是获取包含货架图像的二维图像数据。从而为后续对货架关键点的检测提供基础。
本发明实施例中，假设图像采集模块为广角单目相机，由于广角拍摄功能可能会导致图像数据中存在畸变。
为避免畸变影响后续定位计算，本发明实施例中，可选地，在获取可移动设备所处环境中的图像数据之后，根据图像采集模块的焦距参数以及畸变参数对图像数据中的各个像素点坐标进行畸变矫正。
举例来说，假设图像数据中每一像素(u 0，v 0)。假设图像采集模块的焦距参数(f x，f v)以及畸变参数(k 1，k 2，k 3，p 1，p 2)。假设图像宽度为W和高度为H。基于上述假设，通过如下公式计算得到矫正后的像素坐标(u 1，v 1)，具体公式为：
Figure PCTCN2022117683-appb-000001
Figure PCTCN2022117683-appb-000002
Figure PCTCN2022117683-appb-000003
进而，102中，将经过畸变矫正后的图像数据输入到关键点检测网络中，以 通过关键点检测网络从图像数据中提取货架关键点在图像坐标系中的第一位置信息。通过检测出的货架关键点可以构建出货架的整体轮廓，从而，为后续获取货架关键点相对图像采集模块所处的可移动设备的相对位姿提供基础。
实际应用中，第一位置信息为货架关键点在二维图像坐标系中对应的像素点坐标。举例来说，假设某一货架的货架关键点记为关键点1、关键点2、关键点3、关键点4。基于此，在图4所示的货架图片中，检测出上述4个货架关键点在该货架图片中的像素坐标。具体地，依次从关键点检测网络中读取货架关键点的像素坐标，记为左上角顶点(x1，y1)、左下角顶点(x2，y2)、右下角顶点(x3，y3)、右上角顶点(x4，y4)。除了示例顺序以及数量之外，还可根据实际应用情况对关键点检测网络的输出进行设置。
实际应用中，可选地，本发明实施例的货架关键点可根据实际货架形态预先标定的。具体地，在训练关键点检测网络的过程中，预先在包含货架的图像数据中标定出货架关键点所处位置，并以标定后的图像数据作为关键点检测网络的训练样本。例如，将货架图片以及货架关键点在该货架图片中的二维坐标作为一组训练样本。货架图片中包含货架可以是完整的也可以是部分的。
为保证提升货架定位的准确性，可选地，预先设置货架关键点对应的个数阈值。进而，对比关键点检测网络提取到的货架关键点的个数是否与个数阈值一致。若不一致，则说明未提取到所有的货架关键点，此情况下，可以调整可移动设备所处位置，以使可移动设备移动到能够拍摄到所有货架关键点的位置。若一致，则说明提取到所有的货架关键点，此情况下，可以执行步骤103。
进而，103中，可选地，根据货架关键点在图像数据中的第一位置信息、以及图像坐标系与可移动设备的车体坐标系之间的转换关系，确定货架关键点相对可移动设备的相对位姿的过程，可以实现为：
根据预设货架参数、图像采集模块的相机参数以及第一位置信息，计算货架关键点在车体坐标系中的第二位置信息；基于图像坐标系与可移动设备的车体坐标系之间的转换关系、以及第二位置信息，计算货架关键点相对可移动设备的相对位姿。
其中，图像坐标系与可移动设备的车体坐标系之间的转换关系，是基于图像采集模块与可移动设备的相对位置关系得到的。可移动设备的车体坐标系是指物理世界中以车体为原点的三维坐标系。
假设货架参数包括货架尺寸。其中货架尺寸包括货架的长度、宽度、高度。假设相机参数包括焦距参数。基于此，上述步骤中，首先，根据预设货架参数、图像采集模块的相机参数以及第一位置信息，计算货架关键点在车体坐标系中的第二位置信息的过程包括：根据货架尺寸以及焦距参数，将货架关键点在图像坐标系中的二维坐标转换到车体坐标系中，得到货架关键点在车体坐标系中的三维坐标。
具体地，如图5所示，在一可选实施例中，假设O为图像坐标系的原点，假设O 1为车体坐标系的原点。基于此，参见图5示出的映射过程，可以将货架关键点的二维坐标P(u 1，v 1)从图像坐标系映射到车体坐标系中，转换为对应的三维坐标P(x c，y c，z c)。
进一步假设货架尺寸包括宽度和高度记为z c。假设焦距参数为(f x，f v)。基于上述假设，通过如下公式可以实现上述坐标系转换过程，得到货架关键点在车体坐标系中的三维坐标P(x c，y c，z c)，即：
Figure PCTCN2022117683-appb-000004
上述公式可以简化为：
Figure PCTCN2022117683-appb-000005
进而，为进一步提升货架定位的准确性，可选地，基于转换关系以及第二位置信息，计算货架关键点相对可移动设备的相对位姿之前，进一步还包括：根据货架的固有结构特征对第二位置信息进行修正。
值得说明的是，货架的固有结构特征是指基于货架结构的特定几何特征。例如，货架的固有结构特征包括但不限于：货架竖直边框两两平行，和/或货架水平边框两两平行。
基于上述固有结构特征，举例来说，假设基于图4所示二维坐标转换得到的4个货架关键点在车体坐标系中的三维坐标，记为左上角顶点(x1，y1，z1)、左下角顶点(x2，y2，z2)、右下角顶点(x3，y3，z3)、右上角顶点(x4，y4，z4)。
基于上述假设，通过判断处于其中一个水平面的两个顶点的横坐标差值与另一个水平面的两个顶点的横坐标差值是否相等来确定货架竖直边框是否两两平行。也即，若x4－x1＝x3－x2，则说明货架竖直边框两两平行。若x4－x1≠x3－x2，则说明货架竖直边框不平行。
另一示例中，仍基于上述假设，通过判断处于同一侧两个顶点的纵坐标差值与另一侧两个顶点的纵坐标差值是否相等来确定货架水平边框是否两两平行。也即，若z4－z3＝z2－z1，则说明货架水平边框两两平行。若z4－z3≠z2－z1，则说明货架水平边框不平行。
最终，基于经过修正后的第二位置信息、以及图像坐标系与可移动设备的车体坐标系之间的转换关系，计算货架关键点相对可移动设备的相对位姿。该相对位姿用于后续对货架进行定位和插取。
图1示出的货架定位方法中，通过关键点检测网络从可移动设备所处环境的图像数据中提取出货架视觉语义特征，并基于视觉语义特征确定货架相对可移动设备的相对位姿，大大提升了对货架所处位置的检测精度，提高货架定位的准确性。并且，本发明实施例无需采集深度信息即可通过关键点检测网络识别出货架的视觉语义特征，大大扩展货架定位的应用范围，降低货架定位的硬件成本。
进一步地，根据本申请的另一实施方式，还可同时利用上述图像采集模块与激光雷达相配合，从而将通过图像采集模块获取的图像数据得到的相对坐标与激光雷达所获取的点云数据进行融合。由此，可将激光雷达所获取的点云数据作为视觉图像数据的补充，从而对其进行额外的校正，以防止视觉数据的定位结果发生过大偏差，提高货架的定位精度。
图6示出了本发明实施例提供的另一种货架定位方法的流程图。如图6所示，除了步骤101、102和103之外，该方法还可包括步骤104、105和106。步骤101、102和103的详细执行过程和技术效果已在上文中详细描述，此处不再赘述。
在步骤104中，通过可移动设备中的激光雷达获取所处环境中的激光雷达点云数据。在该可移动设备中，除了装备有诸如广角单目相机的图像采集模块，还可装备有激光雷达，以用于采集可移动设备所处环境中的点云数据。
在步骤105中，对货架关键点相对可移动设备的坐标进行后处理。在上述步骤103中，通过坐标变换，已得到了货架关键点相对可移动设备的坐标。在此步骤中，可使用货架的先验知识进一步修正货架关键点相对于可移动设备的三维坐标。例如，货架的竖边垂直于地面，横边水平于地面。
仍如上所述，假设货架关键点在车体坐标系中的三维坐标，记为左上角顶点(x1，y1，z1)、左下角顶点(x2，y2，z2)、右下角顶点(x3，y3，z3)、右上角顶点(x4，y4，z4)。以左上角顶点为例，在步骤105中进一步修正后的坐标为：
x1 new＝x2 new＝(x1+x2)/2；
z1 new＝z2 new＝(z1+z2)/2；
y1 new＝y4 new＝(y1+y4)/2。
在步骤106中，将经过后处理的坐标与激光雷达点云数据进行坐标融合，以更新货架关键点相对可移动设备的相对位姿。由步骤105中修正后的左下角和右下角(或者左上角和右上角)的坐标可计算货架的宽度wid。以单线激光雷达所获取的激光点云数据为例，由于单线激光雷达的点云分布在一个平行于地面的平面上，因此可在以(x2，z2)为圆心、wid*0.1为半径的圆形区域内查找点云。如果存在多个激光点，则取距离可移动设备车体最近的3个点与(x2，z2)求平均，作为新的左下角坐标；如果圆形区域内激光点个数小于3，则以实际个数的激光点与(x2，z2)求平均，作为新的左下角坐标；如果圆形区域内不存在激光点，则当做误检丢弃。同理可更新右下角坐标。
由此，将激光雷达检测的激光点云数据作为图像采集模块所获取的图像数据的补充，可防止图像数据的定位结果出现过大偏差，从而提高货架的定位精度。
图7为本发明实施例提供的一种货架对接方法的流程图，如图7所示，该方法包括如下步骤：
601、通过可移动设备中的图像采集模块获取所处环境中的图像数据；
602、将图像数据输入到关键点检测网络中，以通过关键点检测网络从图像数据中提取货架关键点在图像坐标系中的第一位置信息；
603、根据第一位置信息、以及图像坐标系与可移动设备的车体坐标系之间的转换关系，确定货架关键点相对可移动设备的相对位姿；
604、根据相对位姿确定可移动设备与货架之间的对接路线，以使可移动设备基于对接路线执行对接流程。
上述步骤601至603的具体实现方式与图1所示方法中的步骤101至103类似，详细的执行过程和技术效果参见前述实施例中的描述，此处不再赘述。
在仓储场景中，可移动设备的移动过程中，通过图像采集模块能够采集可移动设备所处环境中的图像数据。由于图像采集模块的视场角越大，其采集的图像数据的范围也就越大，因此，本发明中可选地采用广角单目相机。
在采集到所处仓储场景中的图像数据之后，可以通过关键点检测网络对所处仓储场景的图像数据进行关键点检测，以在图像数据中检测出货架关键点。以图2示出的仓储场景为例，可移动设备通过图像采集模块对货架进行拍摄，将拍摄得到的图像数据经过关键点检测网络，进而通过关键点检测网络检测出图像数据中包含的货架关键点如图3所示。具体地，通过关键点检测网络从图像数据中提取货架关键点在图像坐标系中的第一位置信息。如图3所示，通过关键点检测网络对货架的左上角顶点、左下角顶点、右下角顶点、右上角顶点所处位置进行标记，分别在图像数据中标出对应的关键点1、关键点2、关键点3、关键点4，得到图4所示的4个货架关键点的第一位置信息。进一步，在获 取货架关键点的第一位置信息之后，根据货架关键点的第一位置信息、以及图像坐标系与可移动设备的车体坐标系之间的转换关系，确定货架关键点相对可移动设备的相对位姿。在确定货架关键点相对可移动设备的相对位姿之后，根据相对位姿确定可移动设备与货架之间的对接路线，以使可移动设备基于对接路线执行对接流程。
在本发明实施例中，通过关键点检测网络从可移动设备所处环境的图像数据中提取出货架视觉语义特征(即货架关键点的位置信息)，并基于视觉语义特征确定货架相对可移动设备的相对位姿。进而，在货架相对可移动设备的相对位姿的基础上，规划出可移动设备与货架之间的对接路线，以使可移动设备基于对接路线执行对接流程。从而，实现了可移动设备与货架的对接，大大提升了对货架所处位置的检测精度，提高货架定位以及货架对接的准确性。
下面结合附图对货架对接方法的一些可选步骤进行介绍。下述可选步骤也可用在图1所示的货架定位方法对应的实施例中。
在一可选实施例中，还将可移动设备导航至与待对接货架匹配的对接位置，该对接位置为所处环境中用于采集至少一个货架关键点的位置。进一步，还可根据历史关键点检测结果在货仓环境地图中标定出各个货架的对接位置，以便后续可以快速定位，并导航可移动设备到对接位置，提高货架定位及对接的效率。例如，图8所示的货仓环境地图中，根据历史关键点检测结果通过三角形标记标定出各个货架的可选对接位置。
基于上述接对位置，601中通过可移动设备中的图像采集模块获取所处环境中的图像数据的过程包括：在对接位置中通过图像采集模块拍摄所处环境中的图像数据。
通过上述步骤可以使可移动设备更快到达能够拍摄到货架关键点的位置，不仅能够辅助提高货架定位的准确性，还可缩短货架对接过程所耗费的时间，提高货架对接效率。
再一可选实施例中，可选地，还判断关键点检测网络提取到的货架关键点是否满足预设条件。若货架关键点不满足预设条件，则控制可移动设备围 绕货架移动。在可移动设备的移动过程中，重新执行步骤601至602，直到关键点检测网络提取到的货架关键点符合预设条件。
举例来说，假设预设条件是提取到的货架关键点数量是否达到设定个数阈值。基于此，判断关键点检测网络提取到的货架关键点数量是否达到设定个数阈值，若货架关键点数量未达到设定个数阈值，则说明可移动设备没有采集到所有货架关键点。简单来说，可移动设备采集货架图像的角度并未达到预设角度，可移动设备所处位置并未达到预设位置，此情况下，通过控制可移动设备围绕货架移动，使其移动到预设位置。在可移动设备的移动过程中，重新执行步骤601至602，直到关键点检测网络提取到的货架关键点数量达到设定个数阈值时，可以确定可移动设备达到预设位置。
除了示例的货架关键点数量之外，还可通过预设条件对图像数据的质量、采集角度等进行进一步筛选，以便提高货架定位结果以及对接结果的准确性。
通过上述步骤可以控制可移动设备达到预设位置(也即可以采集到完整货架关键点的位置)，从而，辅助提高货架定位及对接的准确性，缩短货架对接过程所耗费的时间，提高货架对接效率。
另一可选实施例中，在执行对接流程的过程中，可能会出现距离货架较远、对货架的拍摄角度较偏、可移动设备行进路线出现偏差等问题。因此，在执行对接流程的过程中，还可通过图像采集模块重新获取所处环境中的图像数据，并重新执行步骤602至603，以重新计算相对位姿，矫正对接流程，提高对接效率。
具体地，在一可选实施例中，重新执行确定相对位姿的步骤以矫正对接路线的过程包括：重新执行步骤602至603，以得到货架关键点相对可移动设备的相对位姿；根据重新计算的相对位姿，重新确定可移动设备与货架之间的对接路线，以使可移动设备基于重新确定的对接路线矫正对接流程。
通过上述步骤可以重新计算出相对位姿，从而在出现上述问题时重新确定对接路线，以矫正对接流程，从而进一步提高货架对接的准确性，缩短货架对接过程所耗费的时间，提高货架对接效率。
进一步地，根据本申请的另一实施方式，还可同时利用上述图像采集模块与激光雷达相配合，从而将通过图像采集模块获取的图像数据得到的相对坐标与激光雷达所获取的点云数据进行融合。
图9为本发明实施例提供的另一种货架对接方法的流程图，如图9所示，除了步骤601、602、603和604之外，在步骤603和604之间，该方法还可包括步骤605、606和607。步骤601、602、603和604的详细执行过程和技术效果已在上文中详细描述，此处不再赘述。
605、通过可移动设备中的激光雷达获取所处环境中的激光雷达点云数据；
606、对货架关键点相对可移动设备的坐标进行后处理；
607、将经过后处理的坐标与激光雷达点云数据进行坐标融合，以更新货架关键点相对可移动设备的相对位姿。
步骤605、606和607的详细执行过程和技术效果分别与步骤104、105和106类似，此处不再赘述。
以下将详细描述本发明的一个或多个实施例的装置。本领域技术人员可以理解，这些装置均可使用市售的硬件组件通过本方案所教导的步骤进行配置来构成。
图10为本发明实施例提供的一种货架定位装置的结构示意图，如图10所示，该装置包括：
获取模块81，用于通过可移动设备中的图像采集模块获取所处环境中的图像数据；
关键点检测模块82，用于将所述图像数据输入到关键点检测网络中，以通过所述关键点检测网络从所述图像数据中提取货架关键点在图像坐标系中的第一位置信息；
相对位姿确定模块83，用于根据所述第一位置信息、以及所述图像坐标 系与所述可移动设备的车体坐标系之间的转换关系，确定所述货架关键点相对所述可移动设备的相对位姿。
可选地，所述相对位姿确定模块83根据所述第一位置信息、以及所述图像坐标系与车体坐标系之间的转换关系，确定所述货架关键点相对所述可移动设备的相对位姿时，用于：
根据预设货架参数、所述图像采集模块的相机参数以及所述第一位置信息，计算所述货架关键点在所述车体坐标系中的第二位置信息；
基于所述转换关系以及所述第二位置信息，计算所述货架关键点相对所述可移动设备的相对位姿；
其中，所述转换关系是基于所述图像采集模块与所述可移动设备的相对位置关系得到的。
其中，可选地，所述货架参数包括货架尺寸，所述相机参数包括焦距参数。
所述相对位姿确定模块83根据预设货架参数、所述图像采集模块的相机参数以及所述第一位置信息，计算所述货架关键点在所述车体坐标系中的第二位置信息时，用于：
根据货架尺寸以及焦距参数，将所述货架关键点在所述图像坐标系中的二维坐标转换到所述车体坐标系中，得到所述货架关键点在所述车体坐标系中的三维坐标。
其中，可选地，所述装置还包括修正模块，用于：
基于所述转换关系以及所述第二位置信息，计算所述货架关键点相对所述可移动设备的相对位姿之前，根据货架的固有结构特征对所述第二位置信息进行修正；
其中，所述固有结构特征包括：货架竖直边框两两平行和/或货架水平边框两两平行。
可选地，所述装置还包括畸变矫正模块，用于：
将所述图像数据输入到关键点检测网络中之前，根据所述图像采集模块 的焦距参数以及畸变参数对所述图像数据中的各个像素点坐标进行畸变矫正。
可选地，所述货架关键点包括以下任意一个或多个：货架的左上角顶点、右上角顶点、左下角顶点、右下角顶点。
图10所示装置可以执行前述图1至图6所示实施例中提供的货架定位方法，详细的执行过程和技术效果参见前述实施例中的描述，在此不再赘述。
图11为本发明实施例提供的一种货架对接装置的结构示意图，如图11所示，该装置包括：
获取模块91，用于通过可移动设备中的图像采集模块获取所处环境中的图像数据；
关键点检测模块92，用于将所述图像数据输入到关键点检测网络中，以通过所述关键点检测网络从所述图像数据中提取货架关键点在图像坐标系中的第一位置信息；
相对位姿确定模块93，用于根据所述第一位置信息、以及所述图像坐标系与所述可移动设备的车体坐标系之间的转换关系，确定所述货架关键点相对所述可移动设备的相对位姿；
对接模块94，用于根据所述相对位姿确定所述可移动设备与货架之间的对接路线，以使所述可移动设备基于所述对接路线执行对接流程。
可选地，所述装置还包括导航模块，用于：
将所述可移动设备导航至与待对接货架匹配的对接位置，所述对接位置为所处环境中用于采集至少一个所述货架关键点的位置；
所述获取模块91通过可移动设备中的图像采集模块获取所处环境中的图像数据时，用于：
在所述对接位置中通过所述图像采集模块拍摄所处环境中的图像数据。
可选地，所述装置还包括判断模块，用于：
判断关键点检测网络提取到的货架关键点是否满足预设条件；
若所述货架关键点不满足所述预设条件，则控制所述可移动设备围绕货 架移动；
在所述可移动设备的移动过程中，通过所述图像采集模块重新获取所处环境中的图像数据，并执行通过所述关键点检测网络从所述图像数据中提取货架关键点在图像坐标系中的第一位置信息的步骤，直到所述关键点检测网络提取到的货架关键点符合所述预设条件。
可选地，所述装置还包括矫正模块，用于：
在执行对接流程的过程中，通过所述图像采集模块重新获取所处环境中的图像数据，并重新执行确定所述相对位姿的步骤，以矫正所述对接流程。
其中，可选地，所述矫正模块重新执行确定所述相对位姿的步骤，以矫正所述对接路线，用于：
重新执行确定所述相对位姿的步骤，以得到所述货架关键点相对所述可移动设备的参考相对位姿；
判断所述参考相对位姿与所述相对位姿的差值是否符合设定误差阈值；
若所述差值不符合设定误差阈值，则根据所述参考相对位姿重新确定所述可移动设备与货架之间的对接路线，以使所述可移动设备基于重新确定的对接路线矫正所述对接流程。
可选地，所述图像采集模块为广角单目相机。
图11所示装置可以执行前述图7至图9所示实施例中提供的货架对接方法，详细的执行过程和技术效果参见前述实施例中的描述，在此不再赘述。
在一个可能的设计中，上述图10或图11所示装置的结构可实现为一电子设备，如图12所示，该电子设备可以包括：处理器1001、存储器1002。其中，所述存储器1002上存储有可执行代码，当所述可执行代码被所述处理器1001执行时，使所述处理器1001至少可以实现如前述图1至图9所示实施例中提供的方法。
可选地，该电子设备中还可以包括通信接口1003，用于与其他设备进行通信。
另外，本发明实施例提供了一种非暂时性机器可读存储介质，所述非暂时 性机器可读存储介质上存储有可执行代码，当所述可执行代码被电子设备的处理器执行时，使所述处理器至少可以实现如前述图1至图9所示实施例中提供的方法。
以上所描述的装置实施例仅仅是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。本领域普通技术人员在不付出创造性的劳动的情况下，即可以理解并实施。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到各实施方式可借助加必需的通用硬件平台的方式来实现，当然也可以通过硬件和软件结合的方式来实现。基于这样的理解，上述技术方案本质上或者说对现有技术做出贡献的部分可以以计算机产品的形式体现出来，本发明可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本发明实施例提供的货架定位方法可以由某种程序/软件来执行，该程序/软件可以由网络侧提供，前述实施例中提及的电子设备可以将该程序/软件下载到本地的非易失性存储介质中，并在其需要执行前述货架定位方法时，通过CPU将该程序/软件读取到内存中，进而由CPU执行该程序/软件以实现前述实施例中所提供的货架定位方法，执行过程可以参见前述图1至图6中的示意。
最后应说明的是：以上实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的精神和范围。

Claims (17)

  1. 一种货架定位方法，其特征在于，所述方法包括：
    通过可移动设备中的图像采集模块获取所处环境中的图像数据；
    将所述图像数据输入到关键点检测网络中，以通过所述关键点检测网络从所述图像数据中提取货架关键点在图像坐标系中的第一位置信息；
    根据所述第一位置信息、以及所述图像坐标系与所述可移动设备的车体坐标系之间的转换关系，确定所述货架关键点相对所述可移动设备的相对位姿。
  2. 根据权利要求1所述的方法，其特征在于，所述根据所述第一位置信息、以及所述图像坐标系与车体坐标系之间的转换关系，确定所述货架关键点相对所述可移动设备的相对位姿，包括：
    根据预设货架参数、所述图像采集模块的相机参数以及所述第一位置信息，计算所述货架关键点在所述车体坐标系中的第二位置信息；
    基于所述转换关系以及所述第二位置信息，计算所述货架关键点相对所述可移动设备的相对位姿；
    其中，所述转换关系是基于所述图像采集模块与所述可移动设备的相对位置关系得到的。
  3. 根据权利要求2所述的方法，其特征在于，所述货架参数包括货架尺寸，所述相机参数包括焦距参数；
    所述根据预设货架参数、所述图像采集模块的相机参数以及所述第一位置信息，计算所述货架关键点在所述车体坐标系中的第二位置信息，包括：
    根据货架尺寸以及焦距参数，将所述货架关键点在所述图像坐标系中的二维坐标转换到所述车体坐标系中，得到所述货架关键点在所述车体坐标系中的三维坐标。
  4. 根据权利要求2所述的方法，其特征在于，基于所述转换关系以及所述第二位置信息，计算所述货架关键点相对所述可移动设备的相对位姿之 前，还包括：
    根据货架的固有结构特征对所述第二位置信息进行修正；
    其中，所述固有结构特征包括：货架竖直边框两两平行和/或货架水平边框两两平行。
  5. 根据权利要求1所述的方法，其特征在于，将所述图像数据输入到关键点检测网络中之前，还包括：
    根据所述图像采集模块的焦距参数以及畸变参数对所述图像数据中的各个像素点坐标进行畸变矫正。
  6. 根据权利要求1所述的方法，其特征在于，所述货架关键点包括以下任意一个或多个：货架的左上角顶点、右上角顶点、左下角顶点、右下角顶点。
  7. 根据权利要求1所述的方法，其特征在于，在根据所述第一位置信息、以及所述图像坐标系与所述可移动设备的车体坐标系之间的转换关系，确定所述货架关键点相对所述可移动设备的相对位姿之后，所述方法还包括：
    通过可移动设备中的激光雷达获取所处环境中的激光雷达点云数据；
    对所述货架关键点相对所述可移动设备的坐标进行后处理；
    将经过后处理的坐标与所述激光雷达点云数据进行坐标融合，以更新所述货架关键点相对所述可移动设备的相对位姿。
  8. 一种货架对接方法，其特征在于，包括：
    通过可移动设备中的图像采集模块获取所处环境中的图像数据；
    将所述图像数据输入到关键点检测网络中，以通过所述关键点检测网络从所述图像数据中提取货架关键点在图像坐标系中的第一位置信息；
    根据所述第一位置信息、以及所述图像坐标系与所述可移动设备的车体坐标系之间的转换关系，确定所述货架关键点相对所述可移动设备的相对位姿；
    根据所述相对位姿确定所述可移动设备与货架之间的对接路线，以使所述可移动设备基于所述对接路线执行对接流程。
  9. 根据权利要求8所述的方法，其特征在于，在根据所述第一位置信息、以及所述图像坐标系与所述可移动设备的车体坐标系之间的转换关系，确定所述货架关键点相对所述可移动设备的相对位姿之后，所述方法还包括：
    通过可移动设备中的激光雷达获取所处环境中的激光雷达点云数据；
    对所述货架关键点相对所述可移动设备的坐标进行后处理；
    将经过后处理的坐标与所述激光雷达点云数据进行坐标融合，以更新所述货架关键点相对所述可移动设备的相对位姿。
  10. 根据权利要求8或9所述的方法，其特征在于，还包括：
    将所述可移动设备导航至与待对接货架匹配的对接位置，所述对接位置为所处环境中用于采集至少一个所述货架关键点的位置；
    所述通过可移动设备中的图像采集模块获取所处环境中的图像数据，包括：
    在所述对接位置中通过所述图像采集模块拍摄所处环境中的图像数据。
  11. 根据权利要求8或9所述的方法，其特征在于，还包括：
    判断关键点检测网络提取到的货架关键点是否满足预设条件；
    若所述货架关键点不满足所述预设条件，则控制所述可移动设备围绕货架移动；
    在所述可移动设备的移动过程中，通过所述图像采集模块重新获取所处环境中的图像数据，并执行通过所述关键点检测网络从所述图像数据中提取货架关键点在图像坐标系中的第一位置信息的步骤，直到所述关键点检测网络提取到的货架关键点符合所述预设条件。
  12. 根据权利要求8或9所述的方法，其特征在于，还包括：
    在执行对接流程的过程中，通过所述图像采集模块重新获取所处环境中的图像数据，并重新执行确定所述相对位姿的步骤，以矫正所述对接流程。
  13. 根据权利要求8或9所述的方法，其特征在于，所述图像采集模块为广角单目相机。
  14. 一种货架定位装置，其特征在于，包括：
    获取模块，用于通过可移动设备中的图像采集模块获取所处环境中的图像数据；
    关键点检测模块，用于将所述图像数据输入到关键点检测网络中，以通过所述关键点检测网络从所述图像数据中提取货架关键点在图像坐标系中的第一位置信息；
    相对位姿确定模块，用于根据所述第一位置信息、以及所述图像坐标系与所述可移动设备的车体坐标系之间的转换关系，确定所述货架关键点相对所述可移动设备的相对位姿。
  15. 一种货架对接装置，其特征在于，包括：
    获取模块，用于通过可移动设备中的图像采集模块获取所处环境中的图像数据；
    关键点检测模块，用于将所述图像数据输入到关键点检测网络中，以通过所述关键点检测网络从所述图像数据中提取货架关键点在图像坐标系中的第一位置信息；
    相对位姿确定模块，用于根据所述第一位置信息、以及所述图像坐标系与所述可移动设备的车体坐标系之间的转换关系，确定所述货架关键点相对所述可移动设备的相对位姿；
    对接模块，用于根据所述相对位姿确定所述可移动设备与货架之间的对接路线，以使所述可移动设备基于所述对接路线执行对接流程。
  16. 一种电子设备，其特征在于，包括：存储器、处理器；其中，所述存储器上存储有可执行代码，当所述可执行代码被所述处理器执行时，使所述处理器执行如权利要求1-13中任一项所述的方法。
  17. 一种非暂时性机器可读存储介质，其特征在于，所述非暂时性机器可读存储介质上存储有可执行代码，当所述可执行代码被电子设备的处理器执行时，使所述处理器执行如权利要求1-13中任一项所述的方法。
PCT/CN2022/117683 2021-09-10 2022-09-08 货架定位方法、货架对接方法以及装置、设备、介质 WO2023036212A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202280045980.XA CN117897348A (zh) 2021-09-10 2022-09-08 货架定位方法、货架对接方法以及装置、设备、介质
JP2024513089A JP2024532356A (ja) 2021-09-10 2022-09-08 棚の位置決め方法、棚のドッキング方法及び装置、機器、媒体

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111064188.2A CN115797446A (zh) 2021-09-10 2021-09-10 货架定位方法、货架对接方法以及装置、设备、介质
CN202111064188.2 2021-09-10

Publications (1)

Publication Number Publication Date
WO2023036212A1 true WO2023036212A1 (zh) 2023-03-16

Family

ID=85417212

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/117683 WO2023036212A1 (zh) 2021-09-10 2022-09-08 货架定位方法、货架对接方法以及装置、设备、介质

Country Status (3)

Country Link
JP (1) JP2024532356A (zh)
CN (2) CN115797446A (zh)
WO (1) WO2023036212A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116852348A (zh) * 2023-06-01 2023-10-10 中国航空油料集团有限公司 井盖定位方法、装置和系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106556341A (zh) * 2016-10-08 2017-04-05 浙江国自机器人技术有限公司 一种基于特征信息图形的货架位姿偏差检测方法和系统
CN109795830A (zh) * 2019-03-04 2019-05-24 北京旷视科技有限公司 自动定位物流托盘的方法及装置
CN110388922A (zh) * 2018-04-17 2019-10-29 菜鸟智能物流控股有限公司 位置测量方法和位置测量装置
US20210042588A1 (en) * 2019-07-16 2021-02-11 Tata Consultancy Services Limited Method and system for region proposal based object recognition for estimating planogram compliance
CN112378333A (zh) * 2020-10-30 2021-02-19 支付宝(杭州)信息技术有限公司 仓储货物测量方法和装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106556341A (zh) * 2016-10-08 2017-04-05 浙江国自机器人技术有限公司 一种基于特征信息图形的货架位姿偏差检测方法和系统
CN110388922A (zh) * 2018-04-17 2019-10-29 菜鸟智能物流控股有限公司 位置测量方法和位置测量装置
CN109795830A (zh) * 2019-03-04 2019-05-24 北京旷视科技有限公司 自动定位物流托盘的方法及装置
US20210042588A1 (en) * 2019-07-16 2021-02-11 Tata Consultancy Services Limited Method and system for region proposal based object recognition for estimating planogram compliance
CN112378333A (zh) * 2020-10-30 2021-02-19 支付宝(杭州)信息技术有限公司 仓储货物测量方法和装置

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116852348A (zh) * 2023-06-01 2023-10-10 中国航空油料集团有限公司 井盖定位方法、装置和系统
CN116852348B (zh) * 2023-06-01 2023-12-15 中国航空油料集团有限公司 井盖定位方法、装置和系统

Also Published As

Publication number Publication date
CN115797446A (zh) 2023-03-14
JP2024532356A (ja) 2024-09-05
CN117897348A (zh) 2024-04-16

Similar Documents

Publication Publication Date Title
CN107160395B (zh) 地图构建方法及机器人控制系统
CN110163912B (zh) 二维码位姿标定方法、装置及系统
US10930015B2 (en) Method and system for calibrating multiple cameras
CN110176032B (zh) 一种三维重建方法及装置
WO2020034872A1 (zh) 目标获取方法、设备和计算机可读存储介质
CN106379684A (zh) 一种潜入式agv对接方法及系统以及潜入式agv
WO2021093420A1 (zh) 车辆导航方法、装置及计算机可读存储介质
CN111260289A (zh) 一种基于视觉导航的微型无人机仓库盘点系统及方法
CN113052907B (zh) 一种动态环境移动机器人的定位方法
WO2023036212A1 (zh) 货架定位方法、货架对接方法以及装置、设备、介质
CN112378333B (zh) 仓储货物测量方法和装置
CN110827353A (zh) 一种基于单目摄像头辅助的机器人定位方法
CN116309882A (zh) 一种面向无人叉车应用的托盘检测和定位方法及系统
CN114170521B (zh) 一种叉车托盘对接识别定位方法
JPWO2020090897A1 (ja) 位置検出装置、位置検出システム、遠隔制御装置、遠隔制御システム、位置検出方法、及びプログラム
CN111160280A (zh) 基于rgbd相机的目标物体识别与定位方法及移动机器人
CN112733773B (zh) 一种对象检测方法、装置、计算机设备和存储介质
CN115082395B (zh) 一种航空行李自动识别系统及方法
CN112639881A (zh) 距离测量方法、可移动平台、设备和存储介质
WO2020154911A1 (en) Sky determination in environment detection for mobile platforms, and associated systems and methods
CN111656404A (zh) 图像处理方法、系统及可移动平台
CN115439821A (zh) 基于多目视觉agv的定位与跟踪方法及系统
WO2021128314A1 (zh) 一种图像处理方法、设备、图像处理系统及存储介质
CN113994382A (zh) 深度图生成方法、电子设备、计算处理设备及存储介质
CN116486119A (zh) 基于无人搬运车的平板车厢检测方法、装置及无人搬运车

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22866675

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202280045980.X

Country of ref document: CN

WWE Wipo information: entry into national phase

Ref document number: 2024513089

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 24.06.24)