CN113674341A - Robot visual identification and positioning method, intelligent terminal and storage medium - Google Patents

Abstract

The invention discloses a robot visual recognition and positioning method, an intelligent terminal and a storage medium. , output the category information and image position information corresponding to each part to be disassembled through the image recognition model; according to the image position information and the predetermined conversion matrix from the camera to the robot end, determine the target position corresponding to each part to be disassembled information. The invention outputs the category information and image position information of each component to be disassembled through the image recognition model, and determines the target location information according to the image location information, so that the category information of the component to be disassembled and the location information of the component to be disassembled can be accurately identified and the location information of the component to be disassembled can be accurately located, and sharing is realized. The automatic classification and disassembly of bicycles and the recycling of shared bicycle parts solve the problem of waste of resources caused by manual violent disassembly.

Description

A robot visual recognition and positioning method, intelligent terminal and storage medium

technical field

The invention relates to the technical field of machine identification, in particular to a robot visual identification and positioning method, an intelligent terminal and a storage medium.

Background technique

Shared bicycles have the advantages of high degree of freedom, low price, low carbon and environmental protection, and are favored by young people with high frequency of use. While shared bicycles bring great convenience to people, tens of millions of shared bicycles face scrapping every year due to a large number of releases and the destruction of various human factors. Disposing the recycled shared bicycles as waste after violent disassembly will cause a huge waste of resources.

Therefore, the existing technology still needs to be improved and developed.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to provide a robot visual recognition and positioning method, an intelligent terminal and a storage medium in view of the above-mentioned defects of the prior art, aiming at solving the huge resource caused by the existing manual violent disassembly of shared bicycles. waste problem.

The technical scheme adopted by the present invention to solve the problem is as follows:

In a first aspect, an embodiment of the present invention provides a method for visual recognition and positioning of a robot, which is applied to an intelligent terminal connected to a camera and a robot, and the method includes:

acquiring an image to be identified; wherein the image to be identified includes a number of components to be disassembled;

Inputting the to-be-recognized image into an image recognition model, and outputting the category information and image position information corresponding to each component to be disassembled through the image recognition model;

According to the image position information and the predetermined transformation matrix from the camera to the robot end, the target position corresponding to each component to be disassembled is determined.

The robot visual recognition and positioning method, wherein, the generation method of the image recognition model comprises:

Input the training images in the training image set into a preset network model, and output the predicted attribute labels corresponding to each component in the training image through the preset network model; wherein, the training image set includes the training image and the training image The real attribute label corresponding to each component in the above, the real attribute label includes the real category information and the real image position information, and the predicted attribute label includes the predicted category information and the predicted image position information;

Update the model parameters of the preset network model according to the predicted attribute labels and the real attribute labels, and continue to perform the step of outputting the predicted attribute labels corresponding to each component in the training image through the preset network model , until the training situation of the preset network model satisfies the preset condition, so as to obtain the image recognition model.

The robot visual recognition and positioning method, wherein the model parameters of the preset network model are updated according to the predicted attribute label and the real attribute label, and the output through the preset network model is continued. The step of predicting the attribute label corresponding to each component in the training image until the training situation of the preset network model satisfies the preset condition includes:

Determine a loss value according to the predicted attribute label and the real attribute label, and compare the loss value with a preset threshold;

When the loss value is not less than the preset threshold, update the model parameters of the preset network model according to the preset parameter learning rate, and continue to output the training image through the preset network model. The step of predicting attribute labels corresponding to each component until the loss value is less than the preset threshold.

The robot visual recognition and positioning method, wherein the step of determining the target position corresponding to each component to be disassembled according to the image position information and the predetermined transformation matrix from the camera to the robot end includes:

According to the image position information, determine the center position coordinates corresponding to each component to be disassembled;

Coordinate transformation is performed on the coordinates of the center position according to a predetermined conversion matrix from the camera to the end of the robot, and the target positions corresponding to the components to be disassembled are determined.

The robot visual recognition and positioning method, wherein the step of determining the center position coordinates corresponding to the parts to be disassembled according to the image position information includes:

According to the image position information, determine the minimum circumscribed rectangle corresponding to each component to be disassembled;

Acquire the coordinates of the center point of the smallest circumscribed rectangle corresponding to each component to be disassembled, and determine the coordinates of the center point as the coordinates of the center position corresponding to each component to be disassembled.

The robot visual recognition and positioning method, wherein, the method for determining the transformation matrix from the camera to the robot end includes:

Obtaining a pre-designed checkerboard, and determining the coordinates of each corner point in the checkerboard under the robot base coordinate system and the coordinates under the camera coordinate system according to the checkerboard;

According to the coordinates of each corner point in the checkerboard in the robot base coordinate system and the coordinates in the camera coordinate system, the transformation matrix from the camera to the robot end is determined.

The described robot visual recognition and positioning method, wherein the step of determining the coordinates of each corner point in the checkerboard in the robot base coordinate system and the coordinates in the camera coordinate system according to the checkerboard includes:

The camera is calibrated according to the checkerboard, and the internal and external parameters and distortion coefficients of the camera are determined;

The image coordinates of each corner point in the checkerboard are obtained, and the coordinate transformation is performed on the image coordinates according to the internal and external parameters of the camera and the distortion coefficient, and the coordinates of each corner point in the checkerboard in the camera coordinate system are determined.

The described robot visual recognition and positioning method, wherein the step of determining, according to the checkerboard, the coordinates of each corner point in the checkerboard under the robot base coordinate system and the coordinates under the camera coordinate system further comprises:

According to the checkerboard, determine the transformation matrix from the checkerboard coordinate system to the robot end coordinate system and the coordinates of each corner point in the checkerboard under the checkerboard coordinate system;

The coordinates of each corner point in the checkerboard in the robot base coordinate system are determined according to the transformation matrix from the checkerboard coordinate system to the robot end coordinate system and the coordinates of each corner point in the checkerboard in the checkerboard coordinate system.

In a second aspect, an embodiment of the present invention further provides a robot visual recognition and positioning device, wherein the device includes:

an image acquisition module for acquiring an image to be recognized; wherein the image to be recognized includes a number of components to be disassembled;

an image recognition module, configured to input the to-be-recognized image into an image recognition model, and output the category information and image position information corresponding to each component to be disassembled through the image recognition model;

The target positioning module is configured to determine the target position corresponding to each component to be disassembled according to the image position information and the predetermined transformation matrix from the camera to the robot end.

In a third aspect, an embodiment of the present invention provides an intelligent terminal, including: a processor and a storage medium communicatively connected to the processor, where the storage medium is adapted to store a plurality of instructions; the processor is adapted to call the storage medium to execute the steps of implementing the above-mentioned robot visual recognition and positioning method.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium on which a plurality of instructions are stored, wherein the instructions are suitable for being loaded and executed by a processor, so as to implement the above-mentioned robot visual recognition and The steps of the positioning method.

Beneficial effects of the present invention: In the embodiment of the present invention, an image to be recognized is obtained first; wherein the image to be recognized includes a number of parts to be disassembled, and then the image to be recognized is input into an image recognition model, and each image is output through the image recognition model. The category information and image position information corresponding to the parts to be disassembled, and finally the target positions corresponding to the parts to be disassembled are determined according to the image position information and the predetermined conversion matrix from the camera to the robot end. Therefore, the image recognition model is used to output Category information and image position information, and determine the target position corresponding to each part to be disassembled according to the image position information, can accurately identify the category information of the part to be disassembled and accurately locate the position information of the part to be disassembled, and realize the automatic classification, disassembly and sharing of shared bicycles The recycling of bicycle parts solves the problem of waste of resources caused by manual violent disassembly.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments described in the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

1 is a schematic flowchart of a robot visual recognition and positioning method provided by an embodiment of the present invention;

2 is a schematic block diagram of a robot visual recognition and positioning device provided by an embodiment of the present invention;

FIG. 3 is a schematic block diagram of an internal structure of an intelligent terminal provided by an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer and clearer, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

It should be noted that if there are directional indications (such as up, down, left, right, front, back, etc.) involved in the embodiments of the present invention, the directional indications are only used to explain a certain posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly.

Compared with traditional bicycle rental, shared bicycles have the following advantages: First, shared bicycles do not need to be parked on special parking piles, and can park and go in most areas, with a higher degree of freedom of use; secondly, compared to traditional rental bicycles, shared bicycles The price of a few yuan or even tens of yuan in scenic spots is more advantageous; in addition, shared bicycles are in line with the current trend of low-carbon environmental protection and green travel. These advantages make shared bicycles expected to meet the needs of daily high-frequency use and become a new travel trend for young people.

However, while shared bicycles bring great convenience to people, tens of millions of shared bicycles face scrapping every year due to a large number of placements and the destruction of various human factors. The method is to treat the recycled shared bicycles as waste after manual violent disassembly, and some aluminum alloys, plastics, steel, etc. on the shared bicycles are available materials, which will cause a great waste of resources.

In order to solve the problems of the prior art, this embodiment provides a method for visual recognition and positioning of a robot, through which the category information of the parts to be disassembled can be accurately identified and the position information of the parts to be disassembled can be accurately located, so as to realize the automatic sharing of bicycles. Classified disassembly and recycling of shared bicycle parts to solve the problem of resource waste caused by manual violent disassembly. In a specific implementation, the image to be recognized is first acquired; wherein, the image to be recognized includes a number of parts to be disassembled, then the image to be recognized is input into an image recognition model, and the image recognition model is used to output the corresponding parts of each part to be disassembled. category information and image position information, and finally, according to the image position information and the predetermined conversion matrix from the camera to the robot end, determine the target position corresponding to each part to be disassembled, therefore, output each part to be disassembled through the image recognition model category information and image position information, and determine the target position according to the image position information, which can accurately identify the category information of the parts to be disassembled and accurately locate the position information of the parts to be disassembled, and realize the automatic classification and disassembly of shared bicycles and the recycling of shared bicycle parts. .

Exemplary method

This embodiment provides a method for visual recognition and positioning of a robot, and the method can be applied to an intelligent terminal connected with a camera and a robot. Specifically as shown in Figure 1, the method includes:

Step S100, acquiring an image to be recognized; wherein the image to be recognized includes a number of components to be disassembled.

Specifically, the to-be-recognized image is obtained by taking pictures of the shared bicycle to be disassembled by an industrial camera, and the to-be-disassembled shared bicycle includes several parts to be disassembled, such as nuts, screws, bolts, etc. The industrial camera has up to 2 million pixels, and The image to be identified can be collected at a speed of 15 frames per second. The industrial camera and the intelligent terminal are connected through a network cable, and then the UDP/IP protocol is used to create SOCKET communication to realize data transmission between the application layer and the application layer. When the object to be disassembled needs to be disassembled, such as the dismantling of the scrapped shared bicycle, the object to be disassembled is photographed by the industrial camera, and the image to be recognized is collected, and then the collected image to be recognized is transmitted by the industrial camera to the intelligent terminal for follow-up. In the step, the to-be-recognized image is input into an image recognition model to obtain category information and image position information corresponding to each to-be-disassembled component.

Step S200: Input the image to be recognized into an image recognition model, and output category information and image location information corresponding to each component to be disassembled through the image recognition model.

Specifically, the category information is the category to which each component to be disassembled belongs, for example, the component to be disassembled belongs to nuts, bolts or screws, etc., and the image position information is the two-dimensional image coordinates of each component to be disassembled on the to-be-recognized image , the category information and the position information are obtained by identifying and locating the image to be recognized by an image recognition model. Correspondingly, the step of acquiring the category information and image position information corresponding to each component to be disassembled may specifically be: The to-be-recognized image is input into an image recognition model, and the image recognition model outputs the category information and image location information corresponding to each to-be-disassembled part in the to-be-disassembled shared bicycle. The image recognition model includes a basic convolutional neural network and a connected regression classification network, such as some core backbone layers such as convolutional layers, activation layers, and pooling layers.

In a specific embodiment, the method for generating the image recognition model in step S200 includes:

Step S210: Input the training images in the training image set into a preset network model, and output the predicted attribute labels corresponding to each component in the training image through the preset network model; wherein, the training image set includes the training image and all the components in the training image set. The real attribute labels corresponding to each component in the training image, the real attribute labels include real category information and real image position information, and the predicted attribute labels include predicted category information and predicted image position information;

Step S220: Update the model parameters of the preset network model according to the predicted attribute label and the real attribute label, and continue to output the predicted attributes corresponding to each component in the training image through the preset network model The step of labeling, until the training situation of the preset network model satisfies the preset condition, so as to obtain the image recognition model.

Specifically, the training image set includes the training image and the real attribute labels corresponding to each component in the training image, and the several training images are obtained by photographing different types of components with an industrial camera. In order to improve the recognition of the image recognition model and positioning accuracy, when collecting the training image set, the training image set is increased by changing the relative positions of industrial cameras and components, and simulating different industrial background environments through different lighting, light and shade, distance, and image resolution. robustness. The real attribute label includes real category information and real image location information corresponding to each component in the training image.

In this embodiment, python is used to compile a deep learning algorithm in advance to build a network model. The network model has the same structure as the image recognition model, including a basic convolutional neural network and a connected regression classification network, such as convolutional layers, activation layers, Some core backbone layers such as pooling layers. After obtaining the training image set, the network model is trained in the deep learning algorithm compiled by python, and the training process includes the setting of attributes such as training rounds, training batches, etc., wherein the training process of the network model specifically includes: The training images in the training image set are input into a preset network model, and the predicted attribute labels corresponding to the components in the training images are output through the preset network model. Similar to the real attribute labels, the predicted attribute labels include predicted categories information and predicted image position information; then update the model parameters of the preset network model according to the predicted attribute label and the real attribute label, until the training situation of the preset network model satisfies the preset conditions, to obtain Image recognition model.

In a specific embodiment, in step S220, the model parameters of the preset network model are updated according to the predicted attribute label and the real attribute label, and the output of the preset network model is continued. The step of training the predicted attribute labels corresponding to each component in the image until the training situation of the preset network model satisfies the preset condition includes:

Step S221, determining a loss value according to the predicted attribute label and the real attribute label, and comparing the loss value with a preset threshold;

Step S222, when the loss value is not less than the preset threshold, update the model parameters of the preset network model according to the preset parameter learning rate, and continue to output the preset network model through the preset network model. The step of training the predicted attribute label corresponding to each component in the image until the loss value is less than the preset threshold.

Specifically, in this embodiment, a threshold for judging whether the training situation of the preset network model satisfies the preset condition is preset, and after the predicted attribute label is obtained, the loss value is determined according to the predicted attribute label and the real attribute label. Generally, the smaller the loss value, the better the performance of the network model. After the loss value is obtained, it is further judged whether the loss value is less than the preset threshold; if so, it indicates that the training of the preset network model meets the preset conditions; It means that the training situation of the preset network model does not meet the preset conditions, then update the model parameters of the preset network model according to the preset parameter learning rate, and continue to output the training images through the preset network model. In the step of predicting the attribute label corresponding to the component, during the training process of the network model, the background real-time monitoring can be performed through tensorboard to review the network training situation. When the loss value tends to be stable and less than the preset threshold, the network model training is completed.

Step S300 , according to the image position information and a predetermined conversion matrix from the camera to the robot end, determine the target position corresponding to each component to be disassembled.

Since the image position information corresponding to each part to be disassembled output by the image recognition model is the image position coordinate corresponding to each part to be disassembled, in order to facilitate the automatic disassembly of the part to be disassembled by the robot, it is also necessary to convert the image position information into the coordinate system of the robot end. target location information. In this embodiment, after obtaining the image position information corresponding to each part to be disassembled, coordinate transformation is further performed on the image position information through a predetermined transformation matrix from the camera to the robot end to obtain the target position information corresponding to each part to be disassembled.

In a specific embodiment, step S300 specifically includes:

Step S310, according to the image position information, determine the center position coordinates corresponding to the parts to be disassembled;

Step S320: Perform coordinate transformation on the coordinates of the center position according to a predetermined conversion matrix from the camera to the end of the robot, and determine the target positions corresponding to the components to be disassembled.

Specifically, after obtaining the image position information corresponding to the parts to be disassembled in this embodiment, the center position coordinates corresponding to the parts to be disassembled are determined according to the image position information, and then the center position coordinates corresponding to the parts to be disassembled are determined according to the predetermined transformation matrix from the camera to the end of the robot. Coordinate transformation is performed on the coordinates of the central position to determine the target position corresponding to each component to be disassembled.

In a specific implementation manner, step S310 specifically includes:

Step S311, according to the image position information, determine the minimum circumscribed rectangle corresponding to each component to be disassembled;

Step S312: Acquire the coordinates of the center point of the smallest circumscribed rectangle corresponding to each component to be disassembled, and determine the coordinates of the center point as the center position coordinates corresponding to each component to be disassembled.

In a specific embodiment, the coordinates of the center position corresponding to each part to be disassembled are the coordinates of the center point of the smallest circumscribed rectangle corresponding to each part to be disassembled. In this embodiment, after acquiring the image position information corresponding to each part to be disassembled, In the image position information corresponding to the parts, the largest and smallest horizontal and vertical coordinates are selected as the bounding box, the smallest circumscribed rectangle corresponding to each part to be disassembled is determined, and then the center point coordinates of the smallest circumscribed rectangle corresponding to each part to be disassembled are obtained, and the The center point coordinates are determined as the center position coordinates corresponding to each component to be disassembled.

In a specific implementation manner, the method for determining the transformation matrix from the camera to the robot end described in step S300 includes:

Step M310, obtaining a pre-designed checkerboard, and determining the coordinates of each corner point in the checkerboard under the robot base coordinate system and the coordinates under the camera coordinate system according to the checkerboard;

Step M320: Determine a conversion matrix from the camera to the end of the robot according to the coordinates of each corner point in the checkerboard in the robot base coordinate system and the coordinates in the camera coordinate system.

Specifically, camera calibration is a very important step in robot vision, which can help the robot to convert the recognized visual information, so as to complete the subsequent disassembly of the shared bicycle. Checkerboard, determine the coordinates of each corner point in the checkerboard in the robot base coordinate system and the coordinates in the camera coordinate system according to the checkerboard, and then determine the coordinates of each corner point in the robot base according to the checkerboard The coordinates in the system and the coordinates in the camera coordinate system determine the transformation matrix from the camera to the end of the robot. Among them, the calculation formula of the transformation matrix from the camera to the robot end is ^end T _camera =( ^base T _end ) ^-1base P( ^camera P) ^-1 , where ^end T _camera is the transformation matrix from the camera to the robot end, and ^base T _end is The transformation matrix from the robot end to the base coordinate system, ^base T _end can be obtained in real time according to the forward kinematics of the robot, ^camera P is the coordinates of each corner point in the checkerboard in the camera coordinate system, ^base P is the corner point in the checkerboard at The coordinates in the robot base coordinate system.

In a specific embodiment, in step M310, the step of determining the coordinates of each corner point in the checkerboard in the robot base coordinate system and the coordinates in the camera coordinate system according to the checkerboard includes:

Step M311, calibrating the camera according to the checkerboard, and determining internal and external parameters and distortion coefficients of the camera;

Step M312: Obtain the image coordinates of each corner point in the checkerboard, perform coordinate transformation on the image coordinates according to the internal and external parameters of the camera and the distortion coefficient, and determine the position of each corner point in the checkerboard in the camera coordinate system. coordinate.

Specifically, in this embodiment, when determining the coordinates of each corner point in the checkerboard in the camera coordinate system, the camera is first calibrated according to the checkerboard, the internal and external parameters and distortion coefficients of the camera are determined, and then The image coordinates of each corner point in the checkerboard are obtained, and the coordinate transformation is performed on the image coordinates according to the internal and external parameters of the camera and the distortion coefficient, and the coordinates of each corner point in the checkerboard in the camera coordinate system are determined. Among them, the calibration process of the camera is as follows: put the checkerboard in the camera obscura, take some pictures of the checkerboard in different directions, obtain a series of checkerboard pictures, obtain the coordinate values of each corner point on the checkerboard in the world coordinate system and the corresponding The coordinate values in the checkerboard image of , use the least squares method to obtain the homography matrix, and solve the internal and external parameters of the camera and the distortion coefficient to calibrate the camera.

In a specific embodiment, in step M310, the step of determining the coordinates of each corner point in the checkerboard in the robot base coordinate system and the coordinates in the camera coordinate system according to the checkerboard further includes:

Step M313, according to the checkerboard, determine the conversion matrix from the checkerboard coordinate system to the robot end coordinate system and the coordinates of each corner point in the checkerboard under the checkerboard coordinate system;

Step M314, according to the transformation matrix from the checkerboard coordinate system to the robot end coordinate system and the coordinates of each corner point in the checkerboard under the checkerboard coordinate system, determine that each corner point in the checkerboard is under the robot base coordinate system coordinate of.

Specifically, in this embodiment, when determining the coordinates of each corner point in the checkerboard in the robot base coordinate system, the upper left corner of the checkerboard may be set as the origin, and then measuring or determining each corner of the checkerboard according to the design size. The coordinates of the corner points in the checkerboard coordinate system and the translation coordinates from the checkerboard origin to the robot end coordinate origin, and then determine the transformation matrix from the checkerboard coordinate system to the robot end coordinate system according to the translation coordinates from the checkerboard origin to the robot end coordinate origin, Finally, according to the transformation matrix from the checkerboard coordinate system to the robot end coordinate system and the coordinates of each corner point in the checkerboard under the checkerboard coordinate system, determine the coordinates of each corner point in the checkerboard under the robot base coordinate system . The formula for calculating the coordinates of each corner point in the checkerboard in the robot base coordinate system is: ^base P= ^base T _end ^end T _board ^board P, where ^base T _end is the transformation matrix from the robot end to the base coordinate system , ^base T _end can be obtained in real time according to the forward kinematics of the robot, ^end T _board is the transformation matrix from the checkerboard coordinate system to the robot end coordinate system, ^board P is the coordinates of each corner point in the checkerboard in the checkerboard coordinate system.

Exemplary Equipment

As shown in FIG. 2 , an embodiment of the present invention provides a robot visual recognition and positioning device, which includes: an image acquisition module 210 , an image recognition module 220 , and a target positioning module 230 . Specifically, the image acquisition module 210 is configured to acquire an image to be recognized, wherein the image to be recognized includes a number of components to be disassembled. The image recognition module 220 is configured to input the to-be-recognized image into an image recognition model, and output category information and image position information corresponding to each component to be disassembled through the image recognition model. The target positioning module 230 is configured to determine the target position corresponding to each component to be disassembled according to the image position information and the predetermined transformation matrix from the camera to the robot end.

Based on the above embodiments, the present invention also provides an intelligent terminal, the principle block diagram of which may be shown in FIG. 3 . The intelligent terminal includes a processor, a memory, a network interface, a display screen, and a temperature sensor connected through a system bus. Wherein, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The nonvolatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The network interface of the intelligent terminal is used for communicating with external terminals through network connection. When the computer program is executed by the processor, a method for visual recognition and positioning of a robot is realized. The display screen of the smart terminal may be a liquid crystal display screen or an electronic ink display screen, and the temperature sensor of the smart terminal is pre-set inside the smart terminal to detect the operating temperature of the internal equipment.

Those skilled in the art can understand that the principle block diagram shown in FIG. 3 is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the intelligent terminal to which the solution of the present invention is applied. More or fewer components than shown in the figures may be included, or some components may be combined, or have a different arrangement of components.

In one embodiment, an intelligent terminal is provided that includes a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors One or more programs contain instructions to:

acquiring an image to be identified; wherein the image to be identified includes a number of components to be disassembled;

Inputting the to-be-recognized image into an image recognition model, and outputting the category information and image position information corresponding to each component to be disassembled through the image recognition model;

According to the image position information and the predetermined transformation matrix from the camera to the robot end, the target position corresponding to each component to be disassembled is determined.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database or other medium used in the various embodiments provided by the present invention may include non-volatile and/or volatile memory. Nonvolatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Road (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

In summary, the present invention discloses a robot visual recognition and positioning method, an intelligent terminal and a storage medium, including: acquiring an image to be recognized; wherein the image to be recognized includes a number of parts to be disassembled; The image is input to the image recognition model, and the category information and image position information corresponding to each part to be disassembled are output through the image recognition model; according to the image position information and the predetermined conversion matrix from the camera to the end of the robot, the each part to be disassembled is determined. The target position corresponding to the component. The invention outputs the category information and image position information of each component to be disassembled through the image recognition model, and determines the target location information according to the image location information, so that the category information of the component to be disassembled and the location information of the component to be disassembled can be accurately identified and the location information of the component to be disassembled can be accurately located, and sharing is realized. The automatic classification and disassembly of bicycles and the recycling of shared bicycle parts solve the problem of waste of resources caused by manual violent disassembly.

It should be understood that the application of the present invention is not limited to the above examples. For those of ordinary skill in the art, improvements or transformations can be made according to the above descriptions, and all these improvements and transformations should belong to the protection scope of the appended claims of the present invention.

Claims (10)

1. a robot visual recognition and positioning method, is characterized in that, is applied to the intelligent terminal that is connected with camera and robot, and described method comprises: acquiring an image to be identified; wherein the image to be identified includes a number of components to be disassembled; Inputting the to-be-recognized image into an image recognition model, and outputting the category information and image position information corresponding to each component to be disassembled through the image recognition model; According to the image position information and the predetermined transformation matrix from the camera to the robot end, the target position corresponding to each component to be disassembled is determined. 2. The robot visual recognition and positioning method according to claim 1, wherein the generation method of the image recognition model comprises: Input the training images in the training image set into a preset network model, and output the predicted attribute labels corresponding to each component in the training image through the preset network model; wherein, the training image set includes the training image and the training image The real attribute label corresponding to each component in the above, the real attribute label includes the real category information and the real image position information, and the predicted attribute label includes the predicted category information and the predicted image position information; Update the model parameters of the preset network model according to the predicted attribute labels and the real attribute labels, and continue to perform the step of outputting the predicted attribute labels corresponding to each component in the training image through the preset network model , until the training situation of the preset network model satisfies the preset condition, so as to obtain the image recognition model. 3. The robot visual recognition and positioning method according to claim 2, wherein the model parameters of the preset network model are updated according to the predicted attribute label and the real attribute label, and continue to execute The step of outputting the predicted attribute labels corresponding to each component in the training image through the preset network model until the training situation of the preset network model satisfies the preset conditions includes: Determine a loss value according to the predicted attribute label and the real attribute label, and compare the loss value with a preset threshold; When the loss value is not less than the preset threshold, update the model parameters of the preset network model according to the preset parameter learning rate, and continue to output the training image through the preset network model. The step of predicting attribute labels corresponding to each component until the loss value is less than the preset threshold. 4 . The robot visual recognition and positioning method according to claim 1 , wherein the target corresponding to each component to be disassembled is determined according to the image position information and a predetermined conversion matrix from the camera to the robot end. 5 . The steps for location include: According to the image position information, determine the center position coordinates corresponding to each component to be disassembled; Coordinate transformation is performed on the coordinates of the center position according to a predetermined conversion matrix from the camera to the end of the robot, and the target positions corresponding to the components to be disassembled are determined. 5. The robot visual recognition and positioning method according to claim 4, wherein the step of determining the center position coordinates corresponding to the parts to be disassembled according to the image position information comprises: According to the image position information, determine the minimum circumscribed rectangle corresponding to each component to be disassembled; Acquire the coordinates of the center point of the smallest circumscribed rectangle corresponding to each component to be disassembled, and determine the coordinates of the center point as the coordinates of the center position corresponding to each component to be disassembled. 6. The robot visual recognition and positioning method according to claim 1, wherein the method for determining the transformation matrix of the camera to the robot end comprises: Obtaining a pre-designed checkerboard, and determining the coordinates of each corner point in the checkerboard under the robot base coordinate system and the coordinates under the camera coordinate system according to the checkerboard; According to the coordinates of each corner point in the checkerboard in the robot base coordinate system and the coordinates in the camera coordinate system, the transformation matrix from the camera to the robot end is determined. 7. The robot visual recognition and positioning method according to claim 6, wherein the coordinates of each corner point in the checkerboard under the robot base coordinate system and in the camera coordinate system are determined according to the checkerboard. The steps under the coordinates include: The camera is calibrated according to the checkerboard, and the internal and external parameters and distortion coefficients of the camera are determined; Obtain the image coordinates of each corner point in the checkerboard, perform coordinate transformation on the image coordinates according to the internal and external parameters of the camera and the distortion coefficient, and determine the coordinates of each corner point in the checkerboard in the camera coordinate system. 8 . The robot visual recognition and positioning method according to claim 7 , wherein, according to the checkerboard, the coordinates of each corner point in the checkerboard under the robot base coordinate system and in the camera coordinate system are determined according to the checkerboard. 9 . The steps under the coordinates also include: According to the checkerboard, determine the transformation matrix from the checkerboard coordinate system to the robot end coordinate system and the coordinates of each corner point in the checkerboard under the checkerboard coordinate system; The coordinates of each corner point in the checkerboard in the robot base coordinate system are determined according to the transformation matrix from the checkerboard coordinate system to the robot end coordinate system and the coordinates of each corner point in the checkerboard in the checkerboard coordinate system. 9. An intelligent terminal, comprising: a processor and a storage medium communicatively connected to the processor, wherein the storage medium is adapted to store a plurality of instructions; the processor is adapted to call the instructions in the storage medium , so as to execute the steps of realizing the robot visual recognition and positioning method according to any one of the above claims 1-8. 10. A computer-readable storage medium having a plurality of instructions stored thereon, wherein the instructions are adapted to be loaded and executed by a processor to implement the robot according to any one of the preceding claims 1-8. The steps of the visual recognition and localization method.

Images