WO2020034872A1

WO2020034872A1 - Target acquisition method and device, and computer readable storage medium

Info

Publication number: WO2020034872A1
Application number: PCT/CN2019/099398
Authority: WO
Inventors: 吕仕杰
Original assignee: 深圳蓝胖子机器人有限公司
Priority date: 2018-08-17
Filing date: 2019-08-06
Publication date: 2020-02-20
Also published as: CN109086736A

Abstract

Disclosed are a target acquisition method and device, and a computer readable storage medium. The target acquisition method comprises: acquiring a first image captured by a first visual structure; inputting the first image to a pre-trained neural network for calculation, performing calculation and division on each object in two-dimensional information of the first image, and obtaining a pixel set corresponding to each object; acquiring border information of the target according to the pixel set of a target to be acquired, and acquiring depth information of the target according to three-dimensional information of the first image; and controlling, according to the border information and the depth information, a manipulator to move, and acquiring the target. The present invention has the effects of automatically sorting stacked objects and improving the success rate of sorting by a robot.

Description

Target acquisition method, device and computer-readable storage medium

Technical field

The invention relates to the field of robotic sorting, and in particular, to a method, a device, and a computer-readable storage medium for acquiring targets.

Background technique

With the development of logistics automation, robotic sorting is becoming more and more popular. More and more goods need to be sorted quickly. In practice, many goods are stacked together and need to be sorted.

However, in the existing sorting schemes, goods are stacked together and sorting is usually performed manually, so the efficiency is low.

Summary of the Invention

The main object of the present invention is to provide a target acquisition method, a device, and a computer-readable storage medium, which aim to automatically sort stacked objects and improve the success rate of robot sorting.

To achieve the above object, the present invention provides a target acquisition method for a robot to sort overlapping objects. The target acquisition method includes: obtaining a first image captured by a first visual structure; and inputting the first image to A pre-trained neural network performs calculations to calculate each object in the two-dimensional information of the first image, obtains the frame information of the target according to the pixel set of the target to be obtained, and according to the three-dimensional information of the first image Obtaining the depth information of the target; controlling the robot to move and obtain the target according to the frame information and the depth information.

Optionally, the target acquisition method further includes:

Obtain multiple training images;

Obtaining annotations of objects with a completeness of 70% in the training image according to the input instruction;

A neural network is trained according to the training image and corresponding labels.

Optionally, each of the objects in the two-dimensional information for calculating and segmenting the first image includes:

Segment each object in the picture through Fully Instance-aware Semantic Segmentation.

Optionally, the obtaining the frame information of the target according to the pixel set of the target includes:

The frame of the target is extracted according to the pixel set of the target and the RANSAC method.

Optionally, the target acquisition method further includes:

Obtaining a second image of the target captured by the second visual structure;

Obtaining the current pose of the target according to the three-dimensional information of the second image;

When the current posture does not match the preset posture, the robot is controlled to adjust the posture so that the target is in the preset posture.

The present invention also provides a target acquisition device. The target acquisition device includes a processor, a memory, and a target acquisition program stored on the memory and operable on the processor. The processor executes the following steps:

Obtaining a first image taken by a first visual structure located above;

Inputting the first image to a pre-trained neural network for calculation, calculating each object in the two-dimensional information for segmenting the first image, and obtaining a pixel point set corresponding to each object;

Obtaining the frame information of the target according to the pixel point set of the target to be obtained, and obtaining the depth information of the target according to the three-dimensional information of the first image;

Control the robot arm to move and acquire the target according to the frame information and depth information.

Optionally, when the target obtaining program is executed by the processor, the following steps are further implemented:

Obtain multiple training images;

Segment each object in the picture through Fully Instance-aware Semantic Segmentation. .

Optionally, when the target obtaining program is executed by the processor, the following steps are further performed:

Obtaining a second image of the target captured by a second visual structure located below;

When the current posture does not match the preset posture, controlling the manipulator to adjust the acquisition action so that the target is in the preset posture.

The present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a target acquisition program, and when the target acquisition program is executed by a processor, implements the steps of the target acquisition method described above. .

The target acquisition method provided by the present invention obtains a first image through a first visual structure, and identifies the target to be acquired through two-dimensional information in the first image. Then, through the combination of two-dimensional information and three-dimensional information, the amount of horizontal movement of the manipulator and the amount of vertical movement required are obtained. Finally, the manipulator moves to obtain the target. Therefore, in this embodiment, through the combination of two-dimensional and three-dimensional information, it is achieved to capture multiple objects, identify targets that need to be acquired from multiple objects, and obtain the positions of the targets for the robotic arm to acquire. The whole process is handled automatically without manual intervention, so it has the effect of automation. In addition, in this embodiment, two-dimensional and three-dimensional information are combined, two-dimensional information is used for target recognition, and target frame information is obtained, and then three-dimensional information is used to obtain depth information. This process has the effects of high efficiency and accuracy and sophisticated solutions.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly explain the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on the structure shown in these drawings without paying creative work.

FIG. 1 is a flowchart of a first embodiment of a method for obtaining a target according to the present invention; FIG.

FIG. 2 is a schematic diagram of an application example of the target acquisition method shown in FIG. 1; FIG.

FIG. 3 is a partial flowchart of a second embodiment of a method for acquiring a target according to the present invention; FIG.

FIG. 4 is a partial flowchart of a third embodiment of a method for obtaining a target according to the present invention; FIG.

The realization of the purpose, functional characteristics and advantages of the present invention will be further explained with reference to the embodiments and the drawings.

detailed description

It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

Example one

This embodiment proposes a target acquisition method for a robot to sort overlapping objects.

Please refer to FIG. 1 and FIG. 2 in combination. The target acquisition method includes:

In step S101, a first image captured by the first visual structure 100 is obtained.

In step S102, the first image is input to a pre-trained neural network for calculation, and each object in the two-dimensional information of the first image is calculated to obtain a pixel point set corresponding to each object.

Step S103: Obtain frame information of the target 300 according to the pixel set of the target 300 to be acquired, and obtain depth information of the target 300 according to the three-dimensional information of the first image.

Step S104: Control the manipulator to move and acquire the target 300 according to the frame information and the depth information.

In this embodiment, first a first image captured by the first visual structure 100 located above is passed. The first visual structure 100 can obtain RGB images and 3D images. For example, two separate cameras are installed to obtain RGB images and 3D images respectively; or a binocular camera can be used to obtain both RGB images and 3D images through calculation. Therefore, the first image obtained by the system includes both RGB information and 3D information. Normally, the robot will pick up the target from the top down, and then move it up with the target. Therefore, in this embodiment, the first visual structure 100 is disposed above and shoots downward.

In this embodiment, after obtaining the first image, the first image is input to a pre-trained neural network for calculation, and each object in the two-dimensional information of the first image is divided to obtain The set of pixels corresponding to each object. The system obtains two-dimensional information in the first image, for example: directly obtains a two-dimensional image; or removes depth information from the three-dimensional image to obtain a two-dimensional image. The two-dimensional image is then used as the input to the neural network. The pre-trained neural network can calculate and obtain the output value according to the calculation formula obtained in advance when the input value is obtained.

Among them, the neural network can perform convolution, classification, and dimension upgrading operations through the Fully Convolutional Instance-aware Semantic Segmentation scheme. By performing convolution processing on two-dimensional information, pixel classification can be implemented efficiently and accurately. After the classification, the dimension-reduced picture is subjected to a dimension-upgrading process, so as to obtain a classified image, and achieve the effect of segmenting each object in the two-dimensional information of the first image. The classified image is the same size as the first image, which can facilitate the operation of identifying the border according to the pixel set in the subsequent steps. This provides coordinates for the translation of the manipulator.

In this embodiment, after obtaining the pixel point set corresponding to each object, the frame information of the target 300 is obtained according to the pixel point set of the target 300 to be obtained, and the target is obtained according to the three-dimensional information of the first image. 300 depth information. Among them, since the pixel point set basically displays most of the information of the target 300, the frame of the target 300 can usually be calculated. For example, the frame of the target 300 can be extracted according to the pixel point set of the target 300 and the RANSAC method. Then obtain the length of the frame, the area covered in two-dimensional coordinates, and so on from the frame information. Further, the frame information can provide movement information of the manipulator in the forward, backward, leftward, and rightward directions, and is generally recorded as the amount of movement in the X-axis and Y-axis directions. Then, the depth at which the target 300 is located is obtained through the three-dimensional information in the first image. This depth information can provide the movement information of the manipulator in the up-down direction, which is usually recorded as the movement amount in the Z-axis direction.

In this embodiment, after obtaining the frame information and the depth information, the robot is controlled to move and acquire the target 300 according to the frame information and the depth information. Among them, during the acquisition, the suction can be performed by suction, or the robot can grasp it. The manipulator decelerates when it descends to a preset height, and determines whether it has touched the target 300 through a negative pressure sensor or a torque sensor. After determining that the target 300 is encountered, the negative pressure can be maintained to a preset value or the open claws can be closed to obtain the target 300.

The target acquisition method provided in this embodiment obtains a first image through the first visual structure 100, and identifies the target 300 to be acquired through the two-dimensional information in the first image. Then, through the combination of two-dimensional information and three-dimensional information, the amount of horizontal movement of the manipulator and the amount of vertical movement required are obtained. Finally, the target 300 is obtained through the movement of the robot. Therefore, in this embodiment, by combining two-dimensional and three-dimensional information, it is achieved to capture multiple objects, identify the target 300 to be acquired from the multiple objects, and obtain the position of the target 300 for the robotic arm to acquire. The whole process is handled automatically without manual intervention, so it has the effect of automation. And this embodiment uses two-dimensional and three-dimensional combination to recognize the target 300 through two-dimensional information, and obtains the frame information of the target 300, and then obtains the depth information through the three-dimensional information. This process has the effects of high efficiency, accuracy and sophisticated solutions.

Example two

This embodiment provides a method for obtaining a target. This embodiment is based on the foregoing embodiment, and additional steps are added. Please refer to Figure 3, as follows:

Step S201, obtaining a plurality of training images;

Step S202: Obtain a label of the 70% complete object in the training image according to the input instruction;

Step S203: Train a neural network according to the training image and corresponding labels.

The other steps of this embodiment are the same as those of the first embodiment. For details, refer to the first embodiment, and details are not described herein again.

In this embodiment, a plurality of training images are obtained first. Among them, there can be thousands of training images, and the more accurate the classification model can be trained by more training images. The desired output value can be obtained by inputting the input value into the classification model of the neural network.

In this embodiment, after obtaining a plurality of training images, according to an input instruction, a label of an object with a completeness of 70% in the training image is obtained. Among them, the input instruction is manually labeled, that is, the pixel points included in the object to be recognized are labeled in the training image. Among them, in this embodiment, objects with a completeness of 70% are set for labeling. The completeness means that in the training image, the object is only partially exposed, and by judging whether the exposed area reaches 70% of the object itself. If so, the integrity is 70%. By labeling only objects that are 70% complete, training can be made more targeted and the top-level objects available can be identified. When the system recognizes the object in the first image, it can only recognize the uppermost layer and can obtain the object.

In this embodiment, after obtaining the annotations of the objects with a completeness of 70% in the training image according to the input instruction, a neural network is trained according to the training image and the corresponding annotations. Among them, the neural network can continuously try formulas, change formulas, and various combinations of formulas through its own program, so as to continuously approach calculations on the training images to obtain labels like input instructions. When the approach degree reaches a preset value, the neural network saves the current acquisition algorithm, that is, the classification model.

Example three

This embodiment provides a method for obtaining a target. This embodiment is based on the above embodiment, and additional steps are added after obtaining the target 300. Please refer to FIG. 4 and FIG. 2 in detail, as follows:

Step S301: Obtain a second image of the target 300 captured by the second visual structure 200 located below.

Step S302: Obtain the current pose of the target 300 according to the three-dimensional information of the second image.

Step S303: when the current posture does not match the preset posture, control the manipulator to adjust the posture so that the target 300 is in the preset posture.

The other steps of this embodiment are the same as those of the second embodiment. For details, refer to the second embodiment, and details are not described herein again.

In this embodiment, a second image of the target 300 captured by the second visual structure 200 located below is first obtained. Among them, since only 3D information is required in the subsequent steps, the second visual structure 200 only needs to obtain 3D information. Since the manipulator moves upward after acquiring the target 300, the second vision structure 200 will shoot the target 300 from the bottom up. The first visual structure 100 is photographed from the top to the bottom. At this time, it will be blocked by the robot, so it is difficult to segment and obtain the attitude of the target 300. Therefore, by shooting the second visual structure 200 arranged from below to the bottom, the target 300 will not be blocked by the manipulator, so that the second image obtained by shooting can easily segment the target 300 and obtain the attitude of the target 300.

In this embodiment, after the second image of the target 300 is obtained, the current posture of the target 300 is obtained according to the three-dimensional information of the second image. The three-dimensional information of the second image includes a three-dimensional point cloud, and a plane in the three-dimensional point cloud can be extracted through the RANSAC scheme. The extracted plane then fits the shape and attitude of the target 300. A specific fitting scheme is, for example, projecting each of the extracted planes onto the plane in a respective front view direction to obtain two-dimensional plane data; and then fitting to obtain a rectangular region according to the two-dimensional plane data. Then, a rectangular region with the highest density and an area exceeding a preset threshold is used as a reference plane, and a rectangular region whose normal is perpendicular to the normal of the reference plane is used as the relevant plane; finally, a box is fitted according to the reference plane and the relevant plane Body area. Thereby, attitude information of the target 300 is obtained.

In this embodiment, after the current posture of the target 300 is obtained, it is determined whether the current posture matches a preset posture, and if not, the manipulator is controlled to adjust the posture so that the target 300 is in the preset posture. Among them, after the robot picks up the box, the box can be in various postures. In order to enable the box to be smoothly placed in a certain position, the attitude of the box needs to be adjusted. Therefore, when the current posture of the target 300 does not match the preset posture, the direction of the angle to be rotated is obtained through calculation, and then the corresponding adjustment is performed so that the target 300 is in the required preset posture.

Therefore, the target acquisition method provided in this embodiment obtains a second image, calculates the current posture of the target 300 from the three-dimensional information of the second image, and adjusts the target 300 to a preset posture. Thus, the effect of safely and stably placing the target 300 in a preset position can be achieved.

Embodiment 4

This embodiment provides a target acquisition device. The target acquisition device includes a processor, a memory, and a target acquisition program stored on the memory and executable on the processor. The target acquisition program is described by the The processor executes the following steps:

Obtaining a first image captured by the first visual structure 100 located above;

Obtaining frame information of the target 300 according to the pixel point set of the target 300 to be acquired, and obtaining depth information of the target 300 according to three-dimensional information of the first image;

Control the robot to move and acquire the target 300 according to the frame information and depth information.

The target acquisition device provided in this embodiment obtains a first image through the first visual structure 100 and identifies the target 300 to be acquired through the two-dimensional information in the first image. Then, through the combination of two-dimensional information and three-dimensional information, the amount of horizontal movement of the manipulator and the amount of vertical movement required are obtained. Finally, the target 300 is obtained through the movement of the robot. Therefore, in this embodiment, by combining two-dimensional and three-dimensional information, it is achieved to capture multiple objects, identify the target 300 to be acquired from the multiple objects, and obtain the position of the target 300 for the robotic arm to acquire. The whole process is handled automatically without manual intervention, so it has the effect of automation. And this embodiment uses two-dimensional and three-dimensional combination to recognize the target 300 through two-dimensional information, and obtains the frame information of the target 300, and then obtains the depth information through the three-dimensional information. This process has the effects of high efficiency, accuracy and sophisticated solutions.

The target acquisition device provided in this embodiment may also be adjusted by referring to the foregoing embodiment of the target acquisition method. For the technical characteristics of the adjustment and the beneficial effects brought by these technical characteristics, reference may be made to the foregoing embodiments, and details are not described herein again.

Example 5

This embodiment provides a computer-readable storage medium.

A target acquisition program is stored on the computer-readable storage medium, and when the target acquisition program is executed by a processor, the following steps are implemented:

The computer-readable storage medium provided in this embodiment obtains a first image through the first visual structure 100, and identifies the target 300 to be obtained through the two-dimensional information in the first image. Then, through the combination of two-dimensional information and three-dimensional information, the amount of horizontal movement of the manipulator and the amount of vertical movement required are obtained. Finally, the target 300 is obtained through the movement of the robot. Therefore, in this embodiment, by combining two-dimensional and three-dimensional information, it is achieved to capture multiple objects, identify the target 300 to be acquired from the multiple objects, and obtain the position of the target 300 for the robotic arm to acquire. The whole process is handled automatically without manual intervention, so it has the effect of automation. And this embodiment uses two-dimensional and three-dimensional combination to recognize the target 300 through two-dimensional information, and obtains the frame information of the target 300, and then obtains the depth information through the three-dimensional information. This process has the effects of high efficiency, accuracy and sophisticated solutions.

The computer-readable storage medium provided in this embodiment may also be adjusted with reference to the embodiments of the foregoing target acquisition method. For the technical characteristics of the adjustment and the beneficial effects brought by these technical characteristics, reference may be made to the foregoing embodiments, and details are not described herein again.

It should be noted that, in this article, the terms "including", "including" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, It also includes other elements not explicitly listed, or elements inherent to such a process, method, article, or device. Without more restrictions, an element limited by the sentence "including a ..." does not exclude that there are other identical elements in the process, method, article, or device that includes the element.

The sequence numbers of the foregoing embodiments of the present invention are only for description, and do not represent the superiority or inferiority of the embodiments.

Through the description of the above embodiments, those skilled in the art can clearly understand that the methods in the above embodiments can be implemented by means of software plus a necessary universal hardware platform, and of course, also by hardware, but in many cases the former is better. Implementation. Based on such an understanding, the technical solution of the present invention, in essence, or a part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, The optical disc) includes several instructions for causing a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in the embodiments of the present invention.

The embodiments of the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to the above specific implementations, and the above specific implementations are only schematic and not restrictive. Those of ordinary skill in the art at Under the enlightenment of the present invention, many forms can be made without departing from the spirit of the present invention and the scope protected by the claims, and these all fall within the protection of the present invention.

Claims

A target acquisition method for a robot to sort overlapping objects is characterized in that the target acquisition method includes:

Obtaining a first image shot through a first visual structure;

Inputting the first image to a pre-trained neural network for calculation, calculating each object in the two-dimensional information for segmenting the first image, and obtaining a pixel point set corresponding to each object;

Obtaining frame information of the target according to the pixel point set of the target to be obtained, and obtaining depth information of the target according to three-dimensional information of the first image;

Control the robot arm to move and acquire the target according to the frame information and depth information.
The target acquisition method according to claim 1, wherein the target acquisition method further comprises:

Obtain multiple training images;

Obtaining annotations of objects with a completeness of 70% in the training image according to the input instruction;

A neural network is trained according to the training image and corresponding labels.
The method for acquiring an object according to claim 1, wherein each of the objects in the two-dimensional information for calculating and segmenting the first image comprises:

Segment each object in the picture through Fully Instance-aware Semantic Segmentation.
The method for acquiring a target according to claim 1, wherein the obtaining frame information of the target according to the pixel set of the target comprises:

The frame of the target is extracted according to the pixel set of the target and the RANSAC method.
The target acquisition method according to any one of claims 1 to 4, wherein the target acquisition method further comprises:

Obtaining a second image of the target captured by the second visual structure;

Obtaining the current pose of the target according to the three-dimensional information of the second image;

When the current posture does not match the preset posture, the robot is controlled to adjust the posture so that the target is in the preset posture.
A target acquisition device, characterized in that the target acquisition device includes a processor, a memory, and a target acquisition program stored on the memory and operable on the processor, and the target acquisition program is processed by the processor. Implement the following steps when the processor executes:

Obtaining a first image taken by a first visual structure located above;

Inputting the first image to a pre-trained neural network for calculation, calculating each object in the two-dimensional information for segmenting the first image, and obtaining a pixel point set corresponding to each object;

Obtaining the frame information of the target according to the pixel point set of the target to be obtained, and obtaining the depth information of the target according to the three-dimensional information of the first image;

Control the robot arm to move and acquire the target according to the frame information and depth information.
The target acquisition device according to claim 6, wherein when the target acquisition program is executed by the processor, the following steps are further implemented:

Obtain multiple training images;

Obtaining annotations of objects with a completeness of 70% in the training image according to the input instruction;

A neural network is trained according to the training image and corresponding labels.
The target acquisition device according to claim 6, wherein each of the objects in the two-dimensional information for calculating and segmenting the first image comprises:

Segment each object in the picture through Fully Instance-aware Semantic Segmentation.
The method for acquiring a target according to any one of claims 6 to 8, wherein the target acquisition program further executes the following steps when executed by the processor:

Obtaining a second image of the target captured by a second visual structure located below;

Obtaining the current pose of the target according to the three-dimensional information of the second image;

When the current posture does not match the preset posture, controlling the manipulator to adjust the acquisition action so that the target is in the preset posture.
A computer-readable storage medium, wherein a target acquisition program is stored on the computer-readable storage medium, and when the target acquisition program is executed by a processor, the program according to any one of claims 1 to 5 is implemented. Steps of the target acquisition method.