TWI804845B - Object positioning method and object positioning system - Google Patents

Abstract

An object positioning method and an object positioning system are provided. A sensing device collects point cloud data obtained from a scene including a target object. A processing device inputs surrounding area data centered on a key point and a preset feature descriptor to a neural network to calculate a scene feature descriptor of the scene. A processing device performs feature matching between the scene feature descriptor and the preset feature descriptor, and calculates position of the target object in an actual space.

Description

Object positioning method and object positioning system

The present invention relates to an electronic device, and in particular to an object positioning method and an object positioning system.

With the continuous development of factory automation, the transportation of goods has also become an important part of automation. In the handling of large goods, forklifts are the preferred automation objects. In order for the automatic handling forklift to be able to move goods smoothly, in addition to the function of autonomous navigation, it will be more flexible if there is a function of goods identification to automatically adjust the moving process of the forklift when the pallet is skewed, and generally it is necessary to let the forklift transport The goods will be on the pallet, so pallet identification has become an important technical link in the development of automatic handling forklifts.

Pallet identification technology is divided into two parts, including pallet identification and pallet positioning. Identification is to find the pallet in an uncertain environment, and positioning is to correlate the spatial position of the pallet with the forklift. Smooth cargo handling. Generally, cameras are used for plane imaging, which belongs to two-dimensional space information. Traditional machine vision recognition methods are also based on two-dimensional planes. However, pallet recognition requires positioning in addition to recognition. If a single camera is used for positioning, it is easy. There is a big margin of error. Dual cameras rely on the power of two cameras Parallax and geometric conditions are used to calculate the position of the object, but the amount of calculation is relatively large.

The "Prior Art" paragraph is only used to help understand the content of the present invention, so the content disclosed in the "Prior Art" paragraph may contain some conventional technologies that do not constitute the knowledge of those with ordinary skill in the art. The content disclosed in the "Prior Art" paragraph does not represent the content or the problem to be solved by one or more embodiments of the present invention, nor does it represent that it has been known or cognition.

The invention provides an object positioning method and an object positioning system, which can improve the accuracy of identification and positioning of target objects.

Other purposes and advantages of the present invention can be further understood from the technical features disclosed in the present invention.

To achieve one or part or all of the above objectives or other objectives, an embodiment of the present invention provides a method for locating an object, which includes the following steps. The point cloud data obtained from the scene including the target object is received by the sensing device. Through the processing device, key points are extracted from the point cloud data. By means of the processing device, the data of the surrounding area centered on the key point and the preset feature descriptor of the target object are input into the neural network to calculate a scene feature descriptor of the scene. By means of the processing device, feature matching is performed on the scene feature descriptor and the preset feature descriptor. The position of the target object in the real space is calculated by the processing device.

The invention also provides an object positioning system, including a sensing device, a storage devices and processing devices. The sensing device is used for collecting point cloud data obtained from a scene including a target object. The storage device is used for storing the default feature descriptor of the target object. The processing device is coupled to the storage device and the processing device, and is used to receive point cloud data, extract a key point from the point cloud data, and input the surrounding area data centered on the key point and the preset feature descriptor to the neural network, so as to Calculate the scene feature descriptor of the scene, perform feature matching between the scene feature descriptor and the preset feature descriptor, and calculate the position of the target object in the actual space.

Based on the above, the embodiment of the present invention extracts key points from the 3D point cloud data, and inputs the surrounding area data centered on the key points and the preset feature descriptor of the target object to the neural network to calculate the scene feature descriptor of the scene , perform feature matching on the scene feature descriptor and the preset feature descriptor, and calculate the position of the target object in the actual space. In this way, by inputting the preset feature descriptor and the surrounding area data including key points extracted from the 3D point cloud data into the neural network to calculate the scene feature descriptor of the scene, the feature extraction ability of the neural network can be used to effectively improve the target Accuracy and stability of object recognition and positioning.

In order to make the above-mentioned features and advantages of the present invention more comprehensible, the following specific embodiments are described in detail together with the accompanying drawings.

100: Object positioning system

102: Sensing device

104: Processing device

106: storage device

S202~S210, S302~S308: steps

FIG. 1 is a schematic diagram of an object positioning system according to an embodiment of the present invention.

FIG. 2 is a flowchart of an object positioning method according to an embodiment of the present invention.

FIG. 3 is a flowchart of an object locating method according to another embodiment of the present invention.

The aforementioned and other technical contents, features and effects of the present invention will be clearly presented in the following detailed description of a preferred embodiment with reference to the drawings. The directional terms mentioned in the following embodiments, such as: up, down, left, right, front or back, etc., are only directions referring to the attached drawings. Accordingly, the directional terms used are for the purpose of illustration and not for the purpose of limiting the invention.

FIG. 1 is a schematic diagram of an object positioning system according to an embodiment of the present invention, please refer to FIG. 1 . The object positioning system 100 may include a sensing device 102 , a processing device 104 and a storage device 106 , and the processing device 104 is coupled to the sensing device 102 and the storage device 106 . The sensing device 102 can collect point cloud data obtained from a scene including a target object. Spatial information uses point cloud data to indicate that the characteristics of the target object can be extracted using spatial geometry, and then processed to confirm whether it is the target object. In this embodiment, the sensing device 102 can be, for example, a TOF (Time-of-flight) camera, which can, for example, use infrared light or laser light as a light source, and calculate the distance to the object by calculating the time-of-flight reflected from the light source , so as to deduce the three-dimensional coordinates and generate the three-dimensional point cloud data of the scene including the target object, wherein the target object may be, for example, a pallet, which is not limited in this case. The storage device 106 can be, for example, any type of fixed or removable random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), flash memory (Flash memory), hard disk or other similar device or a combination of these devices that can store preset Assuming a feature descriptor, the preset feature descriptor is obtained after extracting key points and descriptors from the standard template. The standard template is a pre-made pallet template. In order to reduce the calculation time, the pallet template is stored after coordinate transformation and key point and descriptor extraction. In this embodiment, the key point can be selected to have a low amount of data in the point cloud data, a feature point with high stability and distinction that is not easily confused with other feature points, for example, a corner in the point cloud data can be detected as a key point. The key points can be extracted, for example, by means of uniform sampling (Uniform sampling), and this case is not limited thereto. In other embodiments, the key points can also be extracted by, for example, SIFT algorithm (Scale invariant feature transform), Harris algorithm (Harris corner detection) or NARF Algorithm (Normal aligned radial feature) to extract key points. The descriptor can be obtained by calculating its feature description based on the surrounding area data centered on the key point. In the example where the target object is a pallet, for example, a TOF camera can be used to obtain clear and complete point cloud data at a preset distance (for example, 1 meter) from the pallet, and the front end of the pallet can be obtained after filtering out noise section as a standard template.

The processing device 104 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor, a plurality of microprocessors, one or more microprocessors incorporating a digital signal processor core, a control Microcontroller, Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (Field Programmable Gate Array, FPGA), any other kind of integrated circuit, state machine or based on advanced A processor of an instruction set machine (Advanced RISC Machine, ARM). processing device 104 The point cloud data provided by the sensing device 102 can be received, key points can be extracted from the point cloud data, and the surrounding area data centered on the key points and the preset feature descriptor can be input into the trained neural network to calculate the The scene feature descriptor of the target object's scene. In some embodiments, in order to reduce the amount of calculation, the processing device 104 may delete part of the point cloud data first. For example, the processing device 104 can first segment the point cloud data according to the plane where the target object is located. The plane can be, for example, the floor plane or the shelf plane where the pallets are placed, and use the Euclidean clustering algorithm (Euclidean Cluster) to segment The point cloud data, that is, the point cloud data is divided into multiple point groups according to the spatial distance. The processing device 104 may choose to extract the key points only from the largest N point groups, thereby reducing the amount of computation, and preventing false detection and noise interference, where N is a positive integer.

Furthermore, due to the directionality of the spatial point cloud, when calculating the scene feature descriptor, the regional reference coordinates of the key points can be calculated first for coordinate conversion to ensure that the calculated scene feature descriptor is the same every time. In addition, in some embodiments, the surrounding area data can be smoothed with a Gaussian distribution to eliminate noise interference and avoid affecting the correctness of the calculation result of the scene feature descriptor.

In this embodiment, the neural network can be, for example, a Siamese neural network, and the Siamese neural network (Siamese Network) can be, for example, a structure including two convolutional neural networks (CNN). This case does not use This is the limit. In this embodiment, the training of the neural network can be carried out using the 3DMatch database, which is RGBD (RGB-Depth) indoor scene data, and each scene is composed of many scattered three-dimensional point clouds, and the adjacent point clouds (point cloud) has overlapping parts. in use When the twin neural network is trained, the key points and the points around the key points can be used as input. The training structure is to make the loss of the same key point as small as possible, and the loss of the key points farther away is better. In addition, Can use pallet data for training.

The processing device 104 can perform feature matching on the calculated scene feature descriptor and the preset feature descriptor, for example, can compare the preset feature descriptor with the scene feature descriptor of each point group, and judge whether each point group has a corresponding Matching scene feature descriptors. For example, it may be determined whether the similarity between the preset feature descriptor and the scene feature descriptor is higher than a threshold, and if it is higher than the threshold, the matching is successful. After completing the matching of the feature descriptors, the processing device 104 can perform coordinate transformation according to the matching result to calculate the position of the target object in the actual space. In some embodiments, after completing the matching of the feature descriptors, the processing device 104 can remove the incorrectly matched exception points, and then perform coordinate transformation, so as to avoid the influence of the incorrectly matched exception points from affecting the position of the target object in the actual space. correctness. In some embodiments, the processing device 104 controls the forklift to carry the pallet according to the calculated position of the target object in the actual space, or transmits the position of the target object in the actual space to the forklift, so that the forklift can transport the pallet accordingly. plate.

FIG. 2 is a flow chart of an object positioning method according to an embodiment of the present invention. Please refer to FIG. 1 and FIG. 2 at the same time. It can be known from the above embodiments that the object positioning method may at least include the following steps. First, the sensing device 102 receives the point cloud data obtained from the scene including the target object (step S202 ), which is not limited in this case. Then, by the processing device 104, key points are extracted from the point cloud data (step S204), and key points are extracted The way of points can be, for example, uniform sampling, which is not limited in this case. Then, by the processing device 104, the data of the surrounding area centered on the key point and the preset feature descriptor of the target object are input to the neural network to calculate the scene feature descriptor of the scene (step S206), wherein the neural network It may be, for example, a Siamese neural network, which may include two convolutional neural network architectures, which is not limited in this case. After that, feature matching is performed on the scene feature descriptor and the preset feature descriptor by the processing device 104 (step S208 ). Finally, the position of the target object in the real space is calculated by the processing device 104 (step S210). In some embodiments, the incorrectly matched exception points may be removed before calculating the position of the target object in the real space, so as to prevent the incorrectly matched exception points from affecting the accuracy of calculating the position of the target object in the real space.

FIG. 3 is a flowchart of an object locating method according to another embodiment of the present invention. The difference between this embodiment and the embodiment in FIG. 2 is that, after step S202, the processing device 104 in this embodiment can also divide the point cloud data into a plurality of point groups (step S302). The method can be, for example, segmenting according to the plane where the target object is located, and segmenting with the Euclidean clustering method, which is not limited in this case. Afterwards, key points are extracted from the largest N point groups (step S304 ), where N is a positive integer. This reduces the amount of computation and prevents false detection and noise interference. Then, the area reference coordinates of the key points are calculated by the processing device 104 for coordinate conversion (step S306 ), so as to ensure that the scene feature descriptors calculated each time are the same. Then, the processing device 104 smoothes the surrounding area data with Gaussian distribution (step S308 ) to further eliminate noise interference, and then enters step S206 . due to steps S206-S210 have been described in the embodiment of FIG. 2 , so details are not repeated here.

In summary, the embodiment of the present invention extracts key points from the 3D point cloud data, and inputs the surrounding area data centered on the key points and the preset feature descriptor of the target object to the neural network to calculate the scene features of the scene The descriptor is used to perform feature matching between the scene feature descriptor and the preset feature descriptor, and calculate the position of the target object in the actual space. In this way, by inputting the preset feature descriptor and the surrounding area data including key points extracted from the 3D point cloud data into the neural network to calculate the scene feature descriptor of the scene, the feature extraction ability of the neural network can be used to effectively improve the Accuracy and stability of object recognition and positioning. In some embodiments, the point cloud data can also be segmented according to the plane of the target object, and the Euclidean clustering algorithm can be used, and the amount of calculation can be reduced by choosing to extract key points only for larger point groups , and prevent false detection and noise interference.

But the above-mentioned ones are only preferred embodiments of the present invention, and the scope of implementation of the present invention cannot be limited with this, that is, all simple equivalent changes and modifications made according to the patent scope of the present invention and the contents of the description of the invention, All still belong to the scope covered by the patent of the present invention. In addition, any embodiment or scope of claims of the present invention does not need to achieve all the objectives or advantages or features disclosed in the present invention. In addition, the abstract and the title are only used to assist the search of patent documents, and are not used to limit the scope of rights of the present invention. Furthermore, the first, second, etc. mentioned in the specification are only used to indicate the names of components, and are not used to limit the upper limit or lower limit of the number of components.

S202~S210: steps

Claims (15)

A method for locating an object, comprising: using a sensing device to receive point cloud data obtained from a scene including a target object; using a processing device to extract a key point from the point cloud data; using the processing device, Input a surrounding area data centered on the key point and a preset feature descriptor of the target object into a neural network to calculate a scene feature descriptor of the scene; by the processing device, the scene The feature descriptor is matched with the preset feature descriptor; and the position of the target object in the actual space is calculated by the processing device. The method for locating an object according to claim 1, comprising: removing exception points of wrong matches. The object positioning method as described in Claim 1, comprising: segmenting the point cloud data according to the plane where the target object is located. The object positioning method as described in Claim 1, comprising: segmenting the point cloud data by a Euclidean clustering method. The object positioning method as described in Claim 1, comprising: calculating the area reference coordinates of the key point for coordinate transformation. The object positioning method according to claim 1, comprising: smoothing the surrounding area data with a Gaussian distribution. The object location method as described in claim item 1, wherein the neural network is Siamese neural network. The object positioning method as described in Claim 1, comprising: dividing the point cloud data into a plurality of point groups, and extracting the key points from the largest N point groups respectively, wherein N is a positive integer. An object positioning system, comprising: a sensing device, used to collect point cloud data obtained from a scene including a target object; a storage device, used to store a preset feature descriptor of the target object; and a processing device , coupled to the storage device and the processing device, for receiving the point cloud data, extracting a key point from the point cloud data, and inputting a surrounding area data centered on the key point and the default feature descriptor into A neural network is used to calculate a scene feature descriptor of the scene, perform feature matching between the scene feature descriptor and the preset feature descriptor, and calculate the position of the target object in the actual space. The object locating system as claimed in claim 9, wherein the processor also removes exception points of wrong matches. The object positioning system as claimed in item 9, wherein the processing device divides the point cloud data according to the plane where the target object is located. The object positioning system as claimed in claim 9, wherein the processing device divides the point cloud data by a Euclidean clustering method. In the object positioning system according to claim 9, the processing device further calculates the area reference coordinates of the key points for coordinate conversion. In the object positioning system as claimed in claim 9, the processing device further smoothes the surrounding area data with a Gaussian distribution. The object positioning system as claimed in item 9, wherein the neural network is a Siamese neural network.

Images