CN112720464B - Target picking method based on robot system, electronic equipment and storage medium - Google Patents
Target picking method based on robot system, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112720464B CN112720464B CN202011451422.2A CN202011451422A CN112720464B CN 112720464 B CN112720464 B CN 112720464B CN 202011451422 A CN202011451422 A CN 202011451422A CN 112720464 B CN112720464 B CN 112720464B
- Authority
- CN
- China
- Prior art keywords
- robot
- image data
- data
- pose information
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1602—Programme controls characterised by the control system, structure, architecture
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
- B25J9/1697—Vision controlled systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
Abstract
The invention provides a target picking method based on a robot system, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring image data; determining all objects present in the image data; determining a pickup target from all objects present in the image data; and acquiring the position information of the picking target, and controlling a picking device of the robot system to pick the picking target. Therefore, the robot can automatically pick up the target and meet the requirements of users.
Description
Technical Field
The invention relates to the technical field of intelligent robots, in particular to a target picking method based on a robot system, electronic equipment and a storage medium.
Background
According to survey result data of the Chinese disabled people association, 1 hundred million of Chinese disabled people are predicted to be broken through in 2020, and only less than 10% of the current disabled people obtain rehabilitation services in different degrees. How to better provide service for the disabled in the process of fighting Chinese dreams has become a major topic in the period of comprehensively establishing the success stage of the well-being society and in front of the vast disabled workers.
The function of the current wheelchair is single, the main function is focused on the aspect of riding instead of walk, but the wheelchair is not suitable for users with defective upper limbs.
Disclosure of Invention
The invention provides a robot-based target picking method, electronic equipment and a storage medium. The robot can automatically pick up the target, and the user requirements are met.
In order to solve the above technical problems, a first technical solution provided by the present invention is: provided is a robot system-based object picking method, including: acquiring image data; determining all objects present in the image data; determining a pickup target from all objects present in the image data; and acquiring the position information of the picking target, and controlling a picking device of the robot system to pick the picking target.
Wherein the determining all objects present in the image data comprises: processing the image data by using an object recognition model to obtain a confidence coefficient parameter of an object in the image data; determining all objects present in the image data according to the confidence parameter.
Wherein the processing the image data by using the object recognition model to obtain the confidence coefficient parameter of the object in the image data comprises: acquiring a training sample set, and labeling the training sample set according to categories; and training the initial deep learning network by utilizing the training sample set to obtain an object detection model.
Wherein the obtaining of the training sample set and the labeling of the training sample set according to the category comprise: and marking the training sample set according to the category by adopting a one-hot label.
Wherein the obtaining of the training sample set and the labeling of the training sample set according to the category comprise: and performing data enhancement processing on the training sample set.
Wherein the robotic system is a wheelchair.
Wherein the method further comprises: determining real pose information of the robot according to the image data and the laser scanning data; judging whether the walking direction of the robot has obstacles or not by utilizing a preset map and combining the pose information; if so, the walking direction of the robot is planned again to realize obstacle avoidance.
Wherein the determining the real pose information of the robot according to the image data and the laser scanning data comprises: performing feature extraction on the image data; obtaining predicted pose information of the robot based on the extracted features and the preset map; determining the true pose information of the robot using the laser scan data and the predicted pose information.
Wherein said determining the true pose information of the robot using the laser scan data and the predicted pose information further comprises: calculating pose constraint parameters based on the image data by using a loop detection algorithm; determining the true pose information of the robot using the laser scan data, the pose constraint parameters, and the predicted pose information.
Wherein, the replanning of the walking direction of the robot further comprises: constructing a map to be updated according to the image data and the laser scanning data; and updating the preset map by using the map to be updated.
In order to solve the above technical problems, a second technical solution provided by the present invention is: provided is an electronic device including: a memory storing program instructions and a processor retrieving the program instructions from the memory to perform any of the robot system based object picking methods described above.
In order to solve the above technical problems, a third technical solution provided by the present invention is: there is provided a computer readable storage medium storing a program file executable to implement the robot system-based object picking method of any one of the above.
The method has the beneficial effects that the method is different from the prior art, and image data are obtained; determining all objects present in the image data; determining a pickup target from all objects present in the image data; and acquiring the position information of the picking target, and controlling a picking device of the robot system to pick the picking target. Therefore, the robot can automatically pick up the target and meet the requirements of users.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without inventive efforts, wherein:
FIG. 1 is a schematic flow chart of a first embodiment of a robot-based object picking method according to the present invention;
FIG. 2 is a schematic flow chart of a first embodiment of the robot-based object picking method according to the present invention;
FIG. 3 is a schematic structural diagram of an electronic device according to an embodiment of the invention;
FIG. 4 is a schematic structural diagram of an embodiment of a computer-readable storage medium according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 1, a schematic flow chart of a first embodiment of a robot-based object picking method according to the present invention includes:
step S11: image data is acquired.
Specifically, for realizing the technical scheme of this application, be provided with the camera on the robot system, the camera is used for acquireing image data.
In one embodiment, the camera is fixedly disposed on the robot system, and is not rotatable, and only can capture a preset area in which the robot is in the reach, such as a fixed area in front of the robot system. In another embodiment, the camera may rotate, which may take image data of all areas around the robotic system.
In the object pickup method of the present invention, if picking up fruit in a fruit tray is desired, a photo of the fruit tray needs to be taken by a camera as image data.
Step S12: all objects present in the image data are determined.
Specifically, since the number and type of fruits in the fruit tray are not constant, all objects present in the captured image data need to be determined in order to accurately pick up the target intended by the user.
Taking the image data as an example of a fruit tray, after obtaining the image data, the image data is processed to analyze the type of objects in the fruit tray in the image data, e.g., what fruit is in the fruit tray.
In a particular embodiment, the image data may be processed using an object recognition model to derive a confidence parameter for an object in the image data. For example, the object recognition model recognizes the image data, and recognizes that the confidence coefficient parameter for identifying apples and pears in the image data is 0.8 and 0.7, the confidence coefficient parameter for identifying mangos in the fruit tray is 0.7, and the confidence coefficient parameter for identifying peaches in the fruit tray is 0.2. All objects present in the image data are determined from the confidence parameter. Specifically, the confidence coefficient parameter obtained by the object recognition model may be compared with a preset confidence coefficient parameter, so as to determine the fruit existing in the fruit tray. For example, if the confidence parameter of apple, the confidence parameter of pear and the confidence parameter of mango are greater than the preset confidence parameter, it is indicated that apple, mango and pear are in the fruit tray.
In a particular embodiment, an object detection model for identifying all objects in the image data needs to be constructed. Specifically, when an object detection model is constructed, a training sample set needs to be obtained, and the training sample set is labeled according to categories; and training the initial deep learning network by utilizing the training sample set to obtain an object detection model.
Specifically, considering that an initial deep learning network needs a large amount of noisy data sets, in the present application, after a training sample set is obtained, data enhancement processing needs to be performed on the training sample set. For example, a plurality of sample images are acquired, the sample images are converted into gray scale images, and then gaussian filtering denoising is performed. And carrying out binarization on the sample image according to a gray threshold value, and finally finding out the outline of the object on the binarized image. Therefore, the contours of the object shot at different shooting angles can be converted into the standard contours of the object. For example, when the dish is photographed from the side, the contour of the photographed dish is an ellipse, and the contour of the dish can be converted into a circle by the above processing. In another embodiment, the sample image may also be affine transformed using an affine transformation matrix to transform objects in the sample image to a standard resolution image without translation and rotation.
In one embodiment, gaussian noise, poisson noise, etc. may also be added to the sample image; and/or, each sample image can be artificially rotated by a preset angle to generate another image, and the another image is added into the training sample set; and/or, each sample image can be manually translated randomly and added into a training sample set; and/or, a certain small scale change can be made to the sample image and added to the training set. The training sample set can be subjected to data enhancement processing by any method so that the training sample set can meet the requirements of an initial deep learning network.
The initial deep learning network used for training the object detection model is Tensorflow. Specifically, the initial deep learning network includes two convolutional layers, both of which use a 5x5 convolutional kernel, maximize pooling, and Relu activation functions. The convolution is to learn the characteristics of the correlation, such as the image spatial correlation characteristics, the maximum pooling enables the deep learning network to learn the scale characteristics and a certain degree of deformation, and the Relu activation function can enable the deep learning network to converge quickly. The initial deep learning network further comprises a full-connection layer, a regularization layer and an output layer, wherein a certain proportion of neurons are discarded randomly during each training of the regularization layer to prevent overfitting of the full-connection layer. The output layer outputs a confidence for each category.
In a specific embodiment, the training sample set is labeled according to categories by using one-hot labels. Wherein, the value of only one dimension is marked by 1, and the values of other dimensions are marked by 0. Specifically, the training sample set is labeled according to categories, for example, the sample set includes: the apple, the pear, the peach, the table, the chair, the pen and the like can be labeled according to categories of fruits, furniture, stationery and the like, and further, different types of fruits such as the apple, the pear and the peach can be labeled in the categories of the fruits. And training the initial deep learning network by utilizing the training sample set, so that the initial deep learning network learns the characteristics of different objects, and further an object detection model is obtained.
Step S13: the pickup target is determined from all objects present in the image data.
The user can determine the pickup target from all the determined objects. For example, if the user wants to eat an apple, the apple may be determined as a pickup target.
Step S14: and acquiring the position information of the picking target, and controlling a picking device of the robot system to pick the picking target.
After the picking target is determined, the position information of the picking target is obtained by using a visual positioning system, and a picking device of the robot is controlled to pick the picking target.
In one embodiment, the present application is exemplified by a robotic system. Specifically, a camera arranged on the wheelchair is used for shooting to obtain image data, and the image data is processed by using an object detection model to identify all objects existing in the image data. In one embodiment, the user can control the shooting angle of the camera to capture the desired image data. After all objects in the image data are identified, the objects can be displayed through a display device arranged on the wheelchair, or if the place where the user is in sight is the shot image data, the objects can not be displayed, namely the display device is not arranged on the wheelchair. After a user determines a target to be picked up, a picking instruction is input into the wheelchair, the wheelchair acquires position information of the picked-up target by using the visual positioning system, and a mechanical arm on the wheelchair is controlled to pick up the picked-up target according to the position information.
In one embodiment, a 6-degree-of-freedom lightweight robotic arm Kinova is used to implement pick-up and the like, and other degrees-of-freedom or types of robotic arms may be used. The Kinova mechanical arm is installed on the wheelchair through a connecting piece, sends an instruction through a user, receives the instruction, selects food specified by the user according to the natural language understanding result and the position information, clamps or grabs the food by using the tail end of the mechanical arm and feeds the food to the user.
After the mechanical arm receives the instruction, the path of the mechanical arm is planned based on the position of the control object and the task category, the control quantity is output, and the path and the control which accord with human habits are obtained through a learning-based method.
In another embodiment, the user may further perform a communication connection between the wheelchair and the mobile phone, such as a bluetooth connection, a wifi connection, and the like, and control the wheelchair by controlling the mobile phone to implement the above-mentioned pickup method. The method can be applied to other systems, for example, a kitchen, can automatically pick up vegetables when washing the vegetables, or can be applied to a drawing board, and can automatically pick up the vegetables when changing the drawing pen, and the method is particularly limited.
By the method, the disabled can be effectively assisted, the user requirements can be met, and the work of nursing staff is reduced.
Referring to fig. 2, a flowchart of a second embodiment of the target picking method based on a robot system of the present invention is shown, which includes:
step S21: image data is acquired.
In this embodiment, the step of acquiring the image data is the same as step S11 in the first embodiment shown in fig. 1, and is not repeated here.
Step S22: and determining the real pose information of the robot according to the image data and the laser scanning data.
Specifically, the robot system is provided with a laser sensor in addition to the camera. And determining the real pose information of the robot by using the laser scanning data and the image data acquired by the laser sensor.
The method for robot positioning and map drawing by singly utilizing visual data has advantages in positioning, but is easy to match and lose in scenes with few characteristics, and the map constructed by the visual data is usually only suitable for positioning but not suitable for path planning and navigation; the raster map constructed by the method of robot positioning and map drawing by using single laser data is inherently favorable for path planning, but the data of the laser radar has small information amount, is difficult to detect in a loop, is easy to fail when constructing a larger map, and is difficult to realize quick relocation. Therefore, the method combines the visual data and the laser data, and further realizes the positioning and mapping of the robot. Namely, the real pose information of the robot is determined according to the image data and the laser scanning data.
Specifically, in an embodiment, the feature extraction module may be used to perform feature extraction on the image data to obtain features in the data image. And obtaining the predicted pose information of the robot based on the extracted features and a preset map. Specifically, a preset map is stored in the robot system in advance. When the robot system is used online, a camera and/or a laser sensor on the robot system can be used for shooting or scanning the working space of the robot in advance, so that a preset map of the space where the robot system is located is constructed, and the preset map is stored.
After extracting the features from the data image, in conjunction with the preset map, it may be determined whether there is an obstacle in the robot walking route. Specifically, if the extracted features indicate that the position of an object is obtained by shooting a data image, the predicted pose information of the robot system at the moment can be determined by taking the object as a reference object in combination with a preset map, and after the predicted pose information of the robot system is determined, facilities around the robot can be obtained according to the preset map, so that a walking route is planned for the robot to avoid obstacles. As can be appreciated, the predicted pose information includes the predicted position and walking direction of the robot.
In the application, in order to accurately obtain the real pose information of the robot, the predicted pose information can be further constrained by using laser scanning data, and then the real pose information of the robot is determined. Specifically, in an embodiment, after the laser scanning data is acquired, estimated pose information of the laser data can be further obtained according to the laser scanning data by using a scan-to-scan or scan-to-map matching method. After the image data is subjected to feature extraction, prediction is carried out by utilizing the extracted features by adopting an EPnP algorithm to obtain predicted pose information, then local or global optimization is carried out by utilizing the predicted pose information and the predicted pose information of the laser data in a graph optimization mode, a map is further constructed, and the real pose information of the robot is obtained. In the process of building a robot map, due to the existence of measurement errors and noise, the optimal pose estimation between adjacent frames is not the optimal estimation in a local small range or even a global large range. In the process of map building in a large range by using a traditional method (such as mapping and the like), the map is generally found to be incapable of being closed when the robot returns to an initial position after moving for a period of time, which is a typical embodiment of the problem. Under the condition, the optimal estimation between the adjacent frames is optimized by combining the laser scanning data, so that the obtained real pose information is more accurate.
In another embodiment, in order to make the obtained real pose information of the robot more accurate, the determining the real pose information of the robot using the laser scan data and the predicted pose information further comprises: calculating pose constraint parameters based on the image data by using a loop detection algorithm; determining the true pose information of the robot using the laser scan data, the pose constraint parameters, and the predicted pose information.
In particular, for applications requiring a map to be constructed over a large area, loop detection is an essential core. Through loop detection and direct matching between the starting point and the loop point, the constraint between the starting point and the loop point after the robot runs for a long distance can be obtained, the constraint can be added into a pose optimization frame, and then the accumulated error is eliminated, so that the accuracy and the stability of the map building are greatly improved.
In the method for constructing the map based on the single laser data, the features of scene uniqueness are less in the data acquired by the laser radar, in other words, the laser data are similar in many places, such as rooms with similar structures, long corridors and the like. Therefore, single laser data is difficult to use for loop detection. The global loop detection of the map construction of the laser data usually depends on the odom estimation in the map construction process to detect the return to the similar position, and when the performance of the laser radar is poor and the map is large, the method has a common effect.
Unlike laser data, image data contains rich scene information, and the image feature set of most scenes is uniquely identifiable, which is advantageous in rapidly determining similar positions. Therefore, a loop detection method combining visual information and laser information is proposed, which mainly adopts a bag-of-words model to establish a feature dictionary and establishes a key frame for fast matching and scene identification.
Because the data, especially the image data, collected by the sensor in the map construction process are massive, and the data are highly isotropic in the similar position. Therefore, in the map building process, it is often necessary to choose a build key frame for fast data matching. Conventional visual map construction keyframes typically contain only image data or image feature data. In this application, we build a key frame that contains four pieces of information: the first part is the feature information of the image feature points, which has been processed by a bag-of-words model into feature index information, also called feature words, that can be used for fast matching search, and one image contains multiple feature words for fast matching. The second part is position information corresponding to each image feature point, which can be an image pixel position or a three-dimensional space position, and the information is used for verification and pose estimation after matching. The third part is laser scanning data information corresponding to the key frame, and the information can also be used for pose estimation after matching. And the fourth part is the estimation of the original pose of the robot in the mapping process corresponding to the key frame. In the map construction process, the selection of the key frame follows the following three principles: the difference from the last global repositioning is larger than 15 frames of data; the key frame position at least needs to detect not less than 50 image features; position and attitude difference from the last key frame: the movement distance is greater than 0.5m or the rotation angle is greater than 30 °.
The extraction process of the key frame comprises the following steps: for each data input in the map construction process, firstly judging whether the data input meets the condition of a key frame; when the conditions are met, extracting image features, searching indexes in the word bag model, and recording feature words; meanwhile, the positions of the feature points are recorded by methods such as matching and the like; recording the laser data after the preprocessing (when a map exists, the laser data can not be recorded); finally, the current machine position and attitude estimated by the map construction is recorded. It should be noted that the current robot position pose corresponding to the keyframe needs to be adjusted after each global or local optimization.
The loop detection algorithm mainly aims at comparing the current data with the data in the stored key frame, and specifically comprises the following steps: firstly, comparing predicted pose information obtained by calculation based on image data during map construction with predicted pose information calculated in a key frame, and judging whether the robot has passed near a certain point; if so, extracting the features of the current image, calculating the feature words, comparing the feature words with the feature words of the adjacent key frames, further matching and verifying the key frames with higher feature word matching degree by adopting the position information of the feature points, and calculating the relative pose difference between the current image data and the matched key frames. Matching is carried out based on the laser scanning data, and then final pose transformation is obtained and is used as a pose constraint parameter; determining the true pose information of the robot using the laser scan data, the pose constraint parameters, and the predicted pose information.
Step S23: and judging whether the robot has obstacles in the walking direction by using a preset map and the pose information.
Specifically, after the pose information, namely the specific position and the walking direction, of the robot is obtained in the above manner, whether the obstacle exists in the walking direction of the robot can be judged by using the preset map.
Step S24: if so, the walking direction of the robot is planned again to realize obstacle avoidance.
If the robot has the obstacle, the walking direction of the robot can be planned again, and then obstacle avoidance is achieved.
In a specific implementation, the preset map may be updated by using a map to be updated, which is constructed by using image data and laser scanning data. Specifically, other non-fixed objects such as toys, chairs, tables and the like can be added in the space at any time, so that after one map construction is completed, the newly constructed map is used for replacing the originally stored preset map, and reference can be conveniently carried out when whether obstacles exist or not is judged next time.
In the method shown in the embodiment, laser data and image data are simultaneously used as input of a system, and a scan-to-scan or scan-to-map matching method is adopted based on the laser data to obtain position and attitude pre-estimation; based on image data, the position and the posture of the robot are pre-estimated by extracting features, on one hand, the features are used for constructing an image feature dictionary and used for loop detection, and on the other hand, algorithms such as EPnP are combined. When the vision and laser are successfully positioned at the same time, the system simultaneously outputs two poses, and EKF fusion is carried out on the results of the two poses; and when the visual tracking is unsuccessful, splicing the point cloud data of the depth camera by adopting the positioning result of the laser to obtain a three-dimensional map. Meanwhile, feature detection and matching are continuously carried out on subsequent frames, map points in visual map construction are reinitialized, if the map points are successful, the fusion mode is continuously adopted, and otherwise, a three-dimensional map is built by using the positioning result of the laser all the time. Therefore, the reliability of the image and the positioning stability can be built, the obstacle avoidance function is accurately realized, and the influence of obstacles in front on the movement of the robot is prevented.
According to the method, key core technologies such as a multi-sensor fusion positioning navigation technology, a mechanical arm intelligent control technology and a computer vision recognition technology are added, the disabled can be effectively assisted, the self-sufficient capability of the disabled is improved, the work of nursing staff is reduced, and various requirements of human beings can be met better than the prior art.
Referring to fig. 3, a schematic structural diagram of an electronic device according to an embodiment of the present invention is shown, the electronic device includes a memory 202 and a processor 201, which are connected to each other.
The memory 202 is used to store program instructions to implement the robot system based object picking method of any of the above.
The processor 201 is used to execute program instructions stored by the memory 202.
The processor 201 may also be referred to as a Central Processing Unit (CPU). The processor 201 may be an integrated circuit chip having signal processing capabilities. The processor 201 may also be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The storage 202 may be a memory bank, a TF card, etc., and may store all information in the electronic device of the device, including the input raw data, the computer program, the intermediate operation results, and the final operation results. It stores and retrieves information based on the location specified by the controller. With the memory, the electronic device can only have the memory function to ensure the normal operation. The storage of electronic devices can be classified into a main storage (internal storage) and an auxiliary storage (external storage) according to the use, and also into an external storage and an internal storage. The external memory is usually a magnetic medium, an optical disk, or the like, and can store information for a long period of time. The memory refers to a storage component on the main board, which is used for storing data and programs currently being executed, but is only used for temporarily storing the programs and the data, and the data is lost when the power is turned off or the power is cut off.
In the several embodiments provided in the present application, it should be understood that the disclosed method and apparatus may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a module or a unit is merely a logical division, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a system server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the method of the embodiments of the present application.
Please refer to fig. 4, which is a schematic structural diagram of a computer-readable storage medium according to the present invention. The storage medium of the present application stores a program file 203 capable of implementing all the above-mentioned target picking methods based on the robot system, wherein the program file 203 may be stored in the storage medium in the form of a software product, and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device) or a processor (processor) to execute all or part of the steps of the methods according to the embodiments of the present application. The aforementioned storage device includes: various media capable of storing program codes, such as a usb disk, a mobile hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, or terminal devices, such as a computer, a server, a mobile phone, and a tablet.
The above description is only an embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (7)
1. An object picking method based on a robot system is characterized by comprising the following steps:
acquiring image data;
determining all objects present in the image data using an object recognition model;
determining a pickup target from all objects present in the image data;
acquiring the position information of the picking target, and controlling a picking device of the robot system to pick the picking target;
the step of determining all objects present in the image data using an object recognition model comprises:
processing the image data by using the object recognition model to obtain a confidence coefficient parameter of an object in the image data; the method specifically comprises the following steps: acquiring a training sample set, and labeling the training sample set according to categories; training an initial deep learning network by using the training sample set to obtain the object recognition model;
determining all objects present in the image data according to the confidence parameter;
the object picking method further comprises:
determining real pose information of the robot according to the image data and the laser scanning data;
judging whether the walking direction of the robot has obstacles or not by utilizing a preset map and combining the pose information;
if so, re-planning the walking direction of the robot to realize obstacle avoidance;
the determining the real pose information of the robot according to the image data and the laser scanning data comprises: performing feature extraction on the image data;
obtaining predicted pose information of the robot based on the extracted features and the preset map;
determining the real pose information of the robot using the laser scan data and the predicted pose information;
the determining the true pose information of the robot using the laser scan data and the predicted pose information further comprises:
calculating pose constraint parameters based on the image data by using a loop detection algorithm;
determining the true pose information of the robot using the laser scan data, the pose constraint parameters, and the predicted pose information.
2. The method of claim 1, wherein the obtaining a training sample set, labeling the training sample set by category comprises:
and marking the training sample set according to the category by adopting a one-hot label.
3. The method of claim 1, wherein the obtaining a training sample set, labeling the training sample set by category comprises:
and performing data enhancement processing on the training sample set.
4. The method of claim 1, wherein the robotic system is a wheelchair.
5. The method of claim 1, further comprising, after said replanning the direction of travel of the robot:
constructing a map to be updated according to the image data and the laser scanning data;
and updating the preset map by using the map to be updated.
6. An electronic device, comprising: a memory storing program instructions and a processor retrieving the program instructions from the memory to perform the robot system based object picking method according to any of claims 1-5.
7. A computer-readable storage medium, characterized in that a program file is stored, which is executable to implement the robot system based object picking method according to any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011451422.2A CN112720464B (en) | 2020-12-09 | 2020-12-09 | Target picking method based on robot system, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011451422.2A CN112720464B (en) | 2020-12-09 | 2020-12-09 | Target picking method based on robot system, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112720464A CN112720464A (en) | 2021-04-30 |
| CN112720464B true CN112720464B (en) | 2022-04-19 |
Family
ID=75599331
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011451422.2A Active CN112720464B (en) | 2020-12-09 | 2020-12-09 | Target picking method based on robot system, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112720464B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113761255B (en) * | 2021-08-19 | 2024-02-09 | 劢微机器人科技(深圳)有限公司 | Robot indoor positioning method, device, equipment and storage medium |
| CN114347040A (en) * | 2022-02-18 | 2022-04-15 | 创新奇智(合肥)科技有限公司 | Method and device for picking up target object, robot and storage medium |
| CN114851194A (en) * | 2022-04-26 | 2022-08-05 | 北京市商汤科技开发有限公司 | Object picking method and device of cleaning robot and related equipment and storage medium thereof |
| CN115406706B (en) * | 2022-11-03 | 2023-05-09 | 君华高科集团有限公司 | Full-automatic food sampling robot based on image recognition |
| CN115586748B (en) * | 2022-11-24 | 2023-03-10 | 苏州德机自动化科技有限公司 | Mobile intelligent flexible motion control system and method thereof |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104444351B (en) * | 2014-11-07 | 2016-11-02 | 京东方科技集团股份有限公司 | Mechanical arm and substrate pick device |
| CN105915859B (en) * | 2016-06-17 | 2019-04-05 | 武汉中仪物联技术股份有限公司 | Pipe detection system and detection method based on pipe robot |
| JP6937995B2 (en) * | 2018-04-05 | 2021-09-22 | オムロン株式会社 | Object recognition processing device and method, and object picking device and method |
| CN108748175A (en) * | 2018-06-06 | 2018-11-06 | 芜湖市赛博信息技术咨询服务有限公司 | The control system of charging multi-function robot |
| CN110660104A (en) * | 2019-09-29 | 2020-01-07 | 珠海格力电器股份有限公司 | Industrial robot visual identification positioning grabbing method, computer device and computer readable storage medium |
| CN110640732B (en) * | 2019-09-30 | 2022-10-11 | 佛山科学技术学院 | Self-adaptive grabbing method of flexible gripper |
-
2020
- 2020-12-09 CN CN202011451422.2A patent/CN112720464B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN112720464A (en) | 2021-04-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112720464B (en) | Target picking method based on robot system, electronic equipment and storage medium | |
| US10769480B2 (en) | Object detection method and system | |
| EP3547259B1 (en) | Image-based position detection method, image-based position detection apparatus, device and storage medium | |
| WO2020259481A1 (en) | Positioning method and apparatus, electronic device, and readable storage medium | |
| US9098740B2 (en) | Apparatus, method, and medium detecting object pose | |
| Mei et al. | Closing loops without places | |
| Gao et al. | Robust RGB-D simultaneous localization and mapping using planar point features | |
| CN111738261A (en) | Pose estimation and correction-based disordered target grabbing method for single-image robot | |
| CN114127805A (en) | Deep network training method | |
| Schauerte et al. | Saliency-based identification and recognition of pointed-at objects | |
| CN113378770A (en) | Gesture recognition method, device, equipment, storage medium and program product | |
| CN114821093A (en) | Method, apparatus and computer readable medium for extracting features from image | |
| CN110543817A (en) | Pedestrian re-identification method based on posture guidance feature learning | |
| Volkhardt et al. | People tracking on a mobile companion robot | |
| CN114565668A (en) | Instant positioning and mapping method and device | |
| CN111931571A (en) | Video character target tracking method based on online enhanced detection and electronic equipment | |
| CN111428743B (en) | Commodity identification method, commodity processing device and electronic equipment | |
| Badeka et al. | Harvest crate detection for grapes harvesting robot based on YOLOv3 model | |
| Suresh Kumar et al. | Selective fruit harvesting: Research, trends and developments towards fruit detection and localization–A review | |
| US11328170B2 (en) | Unknown object identification for robotic device | |
| Li et al. | Location and model reconstruction algorithm for overlapped and sheltered spherical fruits based on geometry | |
| CN110197100B (en) | Object positioning method and device | |
| WO2023050632A1 (en) | Positioning method and apparatus, and device, storage medium and computer program product | |
| CN115223245A (en) | Method, system, equipment and storage medium for detecting and clustering behavior of tourists in scenic spot | |
| Fang et al. | Understanding human-object interaction in RGB-D videos for human robot interaction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |