CN115123839B - AGV-based bin stacking method, device, equipment and storage medium - Google Patents
AGV-based bin stacking method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN115123839B CN115123839B CN202211068469.XA CN202211068469A CN115123839B CN 115123839 B CN115123839 B CN 115123839B CN 202211068469 A CN202211068469 A CN 202211068469A CN 115123839 B CN115123839 B CN 115123839B
- Authority
- CN
- China
- Prior art keywords
- bin
- dimensional model
- material box
- fork
- agv
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B65—CONVEYING; PACKING; STORING; HANDLING THIN OR FILAMENTARY MATERIAL
- B65G—TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS
- B65G57/00—Stacking of articles
- B65G57/02—Stacking of articles by adding to the top of the stack
- B65G57/16—Stacking of articles of particular shape
- B65G57/20—Stacking of articles of particular shape three-dimensional, e.g. cubiform, cylindrical
- B65G57/22—Stacking of articles of particular shape three-dimensional, e.g. cubiform, cylindrical in layers each of predetermined arrangement
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B66—HOISTING; LIFTING; HAULING
- B66F—HOISTING, LIFTING, HAULING OR PUSHING, NOT OTHERWISE PROVIDED FOR, e.g. DEVICES WHICH APPLY A LIFTING OR PUSHING FORCE DIRECTLY TO THE SURFACE OF A LOAD
- B66F9/00—Devices for lifting or lowering bulky or heavy goods for loading or unloading purposes
- B66F9/06—Devices for lifting or lowering bulky or heavy goods for loading or unloading purposes movable, with their loads, on wheels or the like, e.g. fork-lift trucks
- B66F9/063—Automatically guided
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B66—HOISTING; LIFTING; HAULING
- B66F—HOISTING, LIFTING, HAULING OR PUSHING, NOT OTHERWISE PROVIDED FOR, e.g. DEVICES WHICH APPLY A LIFTING OR PUSHING FORCE DIRECTLY TO THE SURFACE OF A LOAD
- B66F9/00—Devices for lifting or lowering bulky or heavy goods for loading or unloading purposes
- B66F9/06—Devices for lifting or lowering bulky or heavy goods for loading or unloading purposes movable, with their loads, on wheels or the like, e.g. fork-lift trucks
- B66F9/075—Constructional features or details
- B66F9/0755—Position control; Position detectors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B65—CONVEYING; PACKING; STORING; HANDLING THIN OR FILAMENTARY MATERIAL
- B65G—TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS
- B65G2201/00—Indexing codes relating to handling devices, e.g. conveyors, characterised by the type of product or load being conveyed or handled
- B65G2201/02—Articles
- B65G2201/0235—Containers
- B65G2201/0258—Trays, totes or bins
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Abstract
The application discloses a method, a device, equipment and a storage medium for stacking workbins based on AGV, which relate to the technical field of automatic stacking and comprise the following steps: when the AGV moves to a discharge preparation point, acquiring image data and depth data of a material box above a fork and a material box on a discharge layer through a visual recognition system, and constructing a first material box three-dimensional model of the material box above the fork and a second material box three-dimensional model of the material box on the discharge layer; judging whether the model meets the stacking requirement or not; if the position of the pallet fork is not met, determining offset data among the models, adjusting the position of the box body above the pallet fork by the vehicle control system according to the offset data until the stacking requirement is met, and stacking the material box on the material box of the unloading layer. The three-dimensional model that this application was judged to acquire image data and degree of depth data through the vision recognition system and is found whether satisfies the requirement of piling up, if not, then according to determining the position appearance of offset data adjustment fork top box, when satisfying, pile up the workbin on the layer workbin that unloads, improve workbin and pile up the precision.
Description
Technical Field
The invention relates to the technical field of automatic stacking, in particular to a method, a device, equipment and a storage medium for stacking a material box based on an AGV.
Background
At present, because the lid and the bottom fork of black workbin get the position and be the black, infrared laser can be absorbed at the black position, lead to TOF (Time of flight ) camera (utilize the equipment of TOF technical imaging promptly), two mesh structure light cameras, laser radar etc. transmission laser type's cameras can take place the data disappearance under normal power, can't satisfy the work that the workbin was piled up to the high accuracy, and the workbin is the components of a whole that can function independently combination formula, and the dark blue box in middle has the error for the black part, can't satisfy the demand that piles up the precision within 10 mm. That is to say, the method for automatically stacking black bins by AGVs (automatic Guided vehicles) in the prior art lacks flexibility and usability, and has the problem of poor stacking accuracy, so that in practical application, the black bins are stacked at a height of more than 6 meters, and the requirements of industrial application scenes on real-time performance, flexibility, working efficiency and safety and reliability cannot be met.
Disclosure of Invention
In view of the above, the present invention provides a AGV-based bin stacking method, apparatus, device and storage medium, which can solve the problem of lack of flexibility and usability when black bins are stacked at a height above a certain height in practical applications, and can improve the stacking accuracy of the bins. The specific scheme is as follows:
in a first aspect, the present application discloses an AGV-based bin stacking method, comprising:
when the AGV travels to a discharge preparation point, acquiring image data and depth data of a material box above a fork and a discharge layer material box through a visual recognition system, and constructing a first material box three-dimensional model of the material box above the fork and a second material box three-dimensional model of the discharge layer material box by using the image data and the depth data;
judging whether the first bin three-dimensional model and the second bin three-dimensional model meet the current stacking requirement;
if the position of the pallet fork is not met, determining offset data between the first bin three-dimensional model and the second bin three-dimensional model, and sending the offset data to a vehicle control system so that the vehicle control system can adjust the position of the box body above the pallet fork according to the offset data;
and after the pose of the box body above the fork is adjusted, the step of obtaining the image data and the depth data of the material box above the fork and the unloading layer material box is executed again until the built first material box three-dimensional model and the built second material box three-dimensional model meet the current stacking requirement, and the material box above the fork is stacked on the unloading layer material box.
Optionally, the visual recognition system includes an RGB camera, a TOF camera, a limit switch, an industrial personal computer, and a power supply; wherein, RGB camera level and forward are installed the both sides of AGV, TOF camera level and forward are installed the centre of AGV, limit switch installs the root of fork, the industrial computer is used for right image data with depth data fuses and handles in order to establish three-dimensional model, limit switch is used for restricting the position appearance of tray on the fork.
Optionally, the obtaining of the image data and the depth data of the material box above the fork and the material box of the unloading layer by the visual recognition system includes:
the image data of the material box above the fork and the unloading layer material box are obtained through the RGB camera in the visual recognition system, and the depth data of the material box above the fork and the unloading layer material box are obtained through the TOF camera in the visual recognition system.
Optionally, the constructing a first bin three-dimensional model of the bin above the fork and a second bin three-dimensional model of the discharge layer bin by using the image data and the depth data includes:
extracting detail features in the image data, and matching the detail features with three-dimensional features output by a pre-constructed target neural network to obtain a corresponding matching result;
judging whether the matching result meets a preset matching requirement or not;
and when the matching result meets the preset matching requirement, constructing a first bin three-dimensional model of the bin above the fork and a second bin three-dimensional model of the unloading layer bin based on the detail feature, the three-dimensional feature and the depth data.
Optionally, the matching the detail feature with a three-dimensional feature output by a pre-constructed target neural network to obtain a corresponding matching result includes:
and matching the detail features with three-dimensional features output by a target neural network model constructed in advance based on a Yolo5 algorithm to obtain corresponding matching results.
Optionally, the determining offset data between the first bin three-dimensional model and the second bin three-dimensional model includes:
and determining the front-back offset and the left-right offset between the first bin three-dimensional model and the second bin three-dimensional model.
Optionally, the sending the offset data to a vehicle control system so that the vehicle control system can adjust the pose of the box body above the fork according to the offset data includes:
and sending the offset data to a vehicle control system so that the vehicle control system can call a portal advancing function and a fork side shifting function according to the offset data to control the AGV to adjust the pose of the box body above the fork.
In a second aspect, the present application discloses an AGV-based bin stacking apparatus comprising:
the data acquisition module is used for acquiring image data and depth data of a material box above the fork and a material box on a discharging layer through a visual recognition system when the AGV advances to a discharging preparation point;
the three-dimensional model building module is used for building a first bin three-dimensional model of a bin above the fork and a second bin three-dimensional model of a discharge layer bin by using the image data and the depth data;
the stacking requirement judging module is used for judging whether the first bin three-dimensional model and the second bin three-dimensional model meet the current stacking requirement;
the offset determining module is used for determining offset data between the first bin three-dimensional model and the second bin three-dimensional model when the first bin three-dimensional model and the second bin three-dimensional model do not meet the current stacking requirement;
the position and orientation adjusting module is used for sending the offset data to a vehicle control system so that the vehicle control system can adjust the position and orientation of the box body above the fork according to the offset data;
and the bin stacking module is used for re-executing the step of acquiring the image data and the depth data of the bin above the fork and the unloading layer bin after the pose of the box above the fork is adjusted until the constructed three-dimensional model of the first bin and the constructed three-dimensional model of the second bin meet the current stacking requirement, and then stacking the bin above the fork on the unloading layer bin.
In a third aspect, the present application discloses an electronic device, comprising:
a memory for storing a computer program;
a processor for executing said computer program to implement the steps of the AGV-based bin stacking method disclosed in the foregoing.
In a fourth aspect, the present application discloses a computer readable storage medium for storing a computer program; wherein the computer program when executed by a processor implements the steps of the AGV-based bin stacking method disclosed above.
Therefore, the application provides an AGV-based bin stacking method, which comprises the following steps: when the AGV travels to a discharge preparation point, acquiring image data and depth data of a material box above a fork and a discharge layer material box through a visual recognition system, and constructing a first material box three-dimensional model of the material box above the fork and a second material box three-dimensional model of the discharge layer material box by using the image data and the depth data; judging whether the first bin three-dimensional model and the second bin three-dimensional model meet the current stacking requirement or not; if the position of the pallet fork is not met, determining offset data between the first bin three-dimensional model and the second bin three-dimensional model, and sending the offset data to a vehicle control system so that the vehicle control system can adjust the position of the box body above the pallet fork according to the offset data; and after the pose of the box body above the fork is adjusted, the step of obtaining the image data and the depth data of the material box above the fork and the unloading layer material box is executed again until the built first material box three-dimensional model and the built second material box three-dimensional model meet the current stacking requirement, and the material box above the fork is stacked on the unloading layer material box. Therefore, the image data and the depth data of the fork upper material box and the unloading layer material box are obtained through the visual recognition system, the first material box three-dimensional model of the fork upper material box and the second material box three-dimensional model of the unloading layer material box are built, whether the built three-dimensional model meets the stacking requirement is judged, if the stacking requirement is not met, the pose of the fork upper box body is adjusted according to the determined offset data, and when the built three-dimensional model meets the current stacking requirement, the fork upper material box is stacked on the unloading layer material box, the problem that the black material box lacks flexibility and usability when stacking operation is conducted above a certain height in practical application can be solved, and the stacking precision of the material box can be improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a flow chart of an AGV based bin stacking method disclosed herein;
FIG. 2 is a schematic view of a vision recognition system disclosed herein;
FIG. 3 is a flow chart of a method for determining a path of travel outside a vehicle body as disclosed herein;
FIG. 4 is a schematic illustration of an AGV based bin stacking apparatus according to the present disclosure;
fig. 5 is a block diagram of an electronic device disclosed in the present application.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
At present, the method for automatically stacking black material boxes by an AGV in the prior art lacks flexibility and usability, and has the problem of poor stacking precision, so that in practical application, the black material boxes are stacked at a height above a certain height, and the requirements of an industrial application scene on instantaneity, flexibility, working efficiency and safety and reliability cannot be met. Therefore, the application provides a workbin stacking scheme based on AGV, and the black workbin stacking operation in practical application at a certain height can be achieved, and the requirements of industrial application scenes on instantaneity, flexibility, working efficiency and safety and reliability are met.
The embodiment of the invention discloses an AGV-based bin stacking method, which comprises the following steps of:
step S11: when the AGV advances to the preparation point of unloading, obtain the image data and the depth data of fork top workbin and unloading layer workbin through visual identification system, and utilize image data and depth data to establish the first workbin three-dimensional model of fork top workbin with the second workbin three-dimensional model of unloading layer workbin.
It should be noted that the visual recognition system includes an RGB (Red-Green-Blue, red, green, blue) camera, a TOF camera, a limit switch, an industrial personal computer, and a power supply; wherein, RGB camera level and forward are installed AGV's both sides, TOF camera level and forward are installed AGV's centre, limit switch installs the root of fork, the industrial computer is used for right image data with depth data fuses and handles in order to found out three-dimensional model, limit switch is used for restricting the position appearance of tray on the fork. As shown in fig. 2, the RGB cameras on both sides provide image data of the bin, the TOF camera provides depth data of the bin, after receiving a corresponding instruction, the image data and the depth data are transmitted to the industrial personal computer through the network cable, the limit switch at the root of the pallet fork is used for limiting the pose of a pallet on the pallet fork, that is, when a goods taking task is performed, the position of the bin is changed by collision with the limit switch in the traveling process of the AGV, so that the traveling direction of the bin and the AGV is 90 ° right angle, and the RGB cameras and the TOF camera on both sides are fixed on the AGV fork frame body by adopting a lower-hanging slide rail type telescopic support, so that the photographing height can be increased along with the ascending of the AGV fork frame body, wherein the TOF camera can work in two modes, and firstly, in a normal mode, the infrared laser wavelength of the TOF camera is 940, so that stability within ± 2mm can be provided for identifying a conventional object; secondly, under the high energy mode, TOF camera's infrared laser wavelength is 850 to under the rated power, through reducing infrared laser wavelength, improve infrared laser's energy, make the light pulse by black workbin box surface absorption part back, still have partial light pulse reflection to receive by TOF camera's receiver, thereby obtain box degree of depth information, thereby avoid because black workbin can absorb infrared laser, lead to the camera of transmission laser type to appear the data disappearance under normal power, thereby can't satisfy the problem that the workbin work was piled up to the high accuracy. The TOF camera in the high-energy mode further reduces a resolving threshold of the depth map, namely when a pixel normal vector in the depth map is resolved, initial real part data and initial imaginary part data of each pixel are obtained, an edge confidence map is reestablished by reducing the resolving threshold, characteristic data are formed by combining the obtained real part data and the obtained imaginary part data with data in a calculation mode, and then more image basic data are reconstructed, so that the problem of data loss caused by the fact that the surface of a black object absorbs light pulses is solved. The industrial personal computer performs fusion processing on the RGB image and the depth image through a pre-designed deep learning framework, for example, the deep learning framework adopts a feature layer fusion method, firstly extracts texture features of a bin from the RGB image and the depth image, fuses the extracted texture features into a single feature vector, and then performs processing by using a pattern recognition method. Moreover, the deep learning framework may be a framework constructed based on a neural network, and the neural network may be a network constructed in advance based on a Yolo5 algorithm, and the neural network may be divided into 3 parts, namely, an encoder, a fusion layer, and a decoder. The encoder is a convolution recursion self-encoder and consists of a convolution neural network and a three-dimensional structure reconstruction network, after an RGB image and a depth image are input, the image is enhanced by a Laplacian operator after Gao Silv, the image contrast is improved, then the convolution neural network extracts contour features and structural features such as foot piers, grooves, concave-convex grooves, round holes, straight lines and the like in the RGB image and the depth image, finally the three-dimensional structure reconstruction network is used for decoding the features, the interrelation among all parts of the bin is reconstructed through cuboid and tree structures and bin known external reference information, the interrelation can include but is not limited to a connection relation, a rotational symmetry relation, a parallel symmetry relation and the like, the three-dimensional structure information of the bin is reconstructed, then in the fusion layer, the bin three-dimensional structure information in each image is comprehensively processed into a total input function, the input function mapping is defined as a mapping function of related features, and the bin known external reference information is reflected into the topological structure of the network bin through the interaction of the neural network and the environment, so that the three-dimensional structure and the digital knowledge base can be conveniently established. The high-level features in the RGB image are used for measuring semantic similarity, and the low-level features in the depth image are used for measuring fine-grained similarity. When the fine-grained similarity and similar semantics between the image and the nearest neighbor are queried, the low-layer features are refined to the sequencing result of the high-layer features, and the low-layer features of the depth image can be used for measuring and querying the fine-grained similarity between the image and the neighbor with the same semantics through the mapping function. The decoder is used for reconstructing the fused image by using the feature vector which is output by the fusion layer and contains the three-dimensional structure information of the bin. By learning and understanding the contour features and the structural features in the RGB image and the depth image, the distribution of weight is determined, the input three-dimensional structure information of the bin is explained, the input feature vector is converted into a high-level logic concept, namely, a decoder performs upsampling on the feature vector, then the upsampled image is subjected to convolution processing, the geometric shape of the bin is perfected, and finally the feature vector is reconstructed into a fused image.
For example, an RGB camera and a TOF camera acquire an RGB image and a depth image at the same time stamp, and then extract detailed features through a training encoder, and a decoder reconstructs an image according to the detailed features. The method comprises the steps of calculating characteristic point coordinates and bin external parameter information through data of a depth image, enhancing and fusing the detail characteristics at a fusion layer by using the characteristic point coordinates and the bin external parameter information, inputting the fused detail characteristics into a training set of a decoder for training, testing in a testing set after training is finished, and finally outputting a bin three-dimensional model trained through a Yolo5 neural network.
In this embodiment, in the process of fetching goods by the AGV, the AGV may travel from the current position to the goods fetching preparation point according to the calculated external goods fetching path, and then adjust the fork height to fetch goods, that is, when the AGV is at the current position and the vehicle control system receives corresponding goods fetching task information, the vehicle control system may send a corresponding goods fetching identification instruction to the visual identification system, so that the visual identification system identifies the bin at the current goods fetching preparation point according to the received goods fetching identification instruction to obtain corresponding bin image information, reconstruct a three-dimensional bin model based on the bin image information obtained during goods fetching, as shown in fig. 3, after the industrial personal computer in the visual identification system calls a corresponding camera interface to obtain image data and depth data corresponding to the bin collected by the current camera, extract detailed features in local image data output by RGB cameras on the left and right sides in the visual identification system, matching the characteristic point coordinates calculated by the data of the depth image and the information such as bin external parameters based on the detail characteristics with a bin three-dimensional model trained and finished by a Yolo5 neural network to finish the identification of the bin, reconstructing the three-dimensional model of the current bin, further inputting the actually measured bin external parameters, performing size verification on the reconstructed three-dimensional model of the current bin, and performing position calculation on the AGV based on the input camera external parameters, namely the conversion of a camera coordinate system to a world coordinate system, namely a rotation matrix and a translation matrix when the three-dimensional model passes the size verification to obtain a calculation result of relative position data of a car body containing the AGV and the bin to be taken, and after the data volume in the vector container meets the preset identification times N, performing mean processing on current data in the vector container to obtain more stable relative pose data, after the relative pose data is obtained, resolving the current vehicle body pose according to navigation coordinate data and corresponding supplement data of the AGV vehicle body, outputting the current vehicle body pose to an upper computer, calculating a goods taking external path for the AGV to travel from the current position to a goods taking preparation point according to the coordinate data corresponding to the current vehicle body pose by the upper computer, issuing the goods taking external path to a vehicle control system of the AGV by the upper computer so that the vehicle control system controls the AGV to travel to the goods taking operation place according to the goods taking external path to complete a goods taking operation task, and when the three-dimensional model does not pass size verification, clearing the relative pose data in the vector container, and identifying a material box of the current goods taking preparation point again to obtain corresponding material box image information again. Similarly, after the AGV finishes taking goods from the goods taking preparation point, when the AGV is controlled to move from the goods taking preparation point to the unloading preparation point, an external path from the goods taking preparation point to the unloading preparation point is calculated in the same mode as the above process, so that the AGV is controlled by the vehicle control system to move to the unloading preparation point according to the external path, when the AGV moves to the unloading preparation point, the vehicle control system in the AGV sends a recognition instruction to the visual recognition system again, so that the visual recognition system obtains the image data and the depth data of the goods fork upper bin and the unloading layer bin according to the received recognition instruction, specifically, the image data of the goods fork upper bin and the unloading layer bin are obtained through the RGB camera in the visual recognition system, the depth data of the goods fork upper bin and the unloading layer bin are obtained through the TOF camera in the visual recognition system, the image data and the depth data are uploaded to the visual recognition system, and the industrial personal computer constructs a three-dimensional model of the first unloading bin and a second unloading layer bin above the goods fork bin by utilizing the image data and the depth data. In the process of reconstructing the first bin three-dimensional model of the bin above the fork and the second bin three-dimensional model of the discharge layer bin, the method specifically includes: extracting detail features in the image data, and matching the detail features with three-dimensional features output by a pre-constructed target neural network to obtain corresponding matching results; judging whether the matching result meets a preset matching requirement or not; and when the matching result meets the preset matching requirement, constructing a first bin three-dimensional model of the bin above the fork and a second bin three-dimensional model of the unloading layer bin based on the detail feature, the three-dimensional feature and the depth data. The target neural network model may adopt a Yolo5 algorithm, that is, the detail features are matched with three-dimensional features output by the target neural network model constructed in advance based on the Yolo5 algorithm to obtain corresponding matching results. It can be understood that when the AGV advances to the unloading preparation point, the vehicle control system sends a second recognition instruction to the visual recognition system, so that a camera in the visual recognition system acquires corresponding image information, and then an industrial personal computer in the system extracts the detail features containing the contour features and the structural features in the image information, matches the detail features with the three-dimensional features corresponding to the detail features in the bin three-dimensional model completed through the Yolo5 neural network training, so as to obtain a corresponding matching result, and when the matching result meets the preset matching requirement, the first bin three-dimensional model of the bin above the fork and the second bin three-dimensional model of the unloading layer bin are reconstructed based on the bin three-dimensional model completed through the Yolo5 neural network training and the image information.
Step S12: and judging whether the first bin three-dimensional model and the second bin three-dimensional model meet the current stacking requirement.
In this embodiment, when reconstructing the first bin three-dimensional model of the bin above the fork and the second bin three-dimensional model of the discharge layer bin, the first bin three-dimensional model and the second bin three-dimensional model are calibrated to determine whether the first bin three-dimensional model and the second bin three-dimensional model meet the current stacking requirement.
Step S13: if the position of the pallet fork is not met, determining offset data between the first bin three-dimensional model and the second bin three-dimensional model, and sending the offset data to a vehicle control system so that the vehicle control system can adjust the position of the box body above the pallet fork according to the offset data.
In this embodiment, when the first bin three-dimensional model and the second bin three-dimensional model are calibrated, if the first bin three-dimensional model and the second bin three-dimensional model do not satisfy the current stacking requirement, offset data between the first bin three-dimensional model and the second bin three-dimensional model is determined through model comparison, where the offset may include, but is not limited to, a front-back offset, a left-right offset, an angle offset, and the like, and then the offset data is sent to a vehicle control system so that the vehicle control system can adjust the position of the box above the fork according to the offset data, for example, the front-back offset and the left-right offset between the first bin three-dimensional model and the second bin three-dimensional model are determined, and the offset data is sent to the vehicle control system so that the vehicle control system calls a gantry forward movement function and a fork side movement function according to the offset data to control the AGV to adjust the position of the box above the fork. If the first material box three-dimensional model and the second material box three-dimensional model meet the current stacking requirement, the AGV is controlled through the vehicle control system to adjust the height of the pallet fork, the material box to be unloaded is stacked on the material box of the unloading layer, the fork discharging operation is completed, the height of the pallet fork is adjusted to be the carrying height, and finally the stacking operation is completed.
Step S14: and after the pose of the box body above the pallet fork is adjusted, the step of obtaining the image data and the depth data of the material box above the pallet fork and the unloading layer material box is executed again until the constructed three-dimensional model of the first material box and the constructed three-dimensional model of the second material box meet the current stacking requirement, and the material box above the pallet fork is stacked on the unloading layer material box.
In this embodiment, when the first bin three-dimensional model and the second bin three-dimensional model do not meet the current stacking requirement and the pose of the box body above the fork is adjusted according to the determined offset, the step of obtaining the image data and the depth data of the bin above the fork and the unloading layer bin is executed again until the constructed first bin three-dimensional model and the constructed second bin three-dimensional model meet the current stacking requirement, and the bin above the fork is stacked on the unloading layer bin.
Therefore, in the embodiment of the application, the image data and the depth data of the material box above the fork and the material box on the unloading layer are obtained through the visual recognition system, the three-dimensional model of the first material box of the material box above the fork and the three-dimensional model of the second material box of the material box on the unloading layer are constructed, whether the constructed three-dimensional models meet the stacking requirement or not is further judged, if the three-dimensional models do not meet the stacking requirement, the position and the posture of the box body above the fork are adjusted according to the determined offset data, and the material box above the fork is further stacked on the material box on the unloading layer until the constructed three-dimensional models meet the current stacking requirement.
Correspondingly, the embodiment of the application also discloses an AGV-based bin stacking device, which is shown in FIG. 4 and comprises:
the data acquisition module 11 is used for acquiring image data and depth data of a material box above a fork and a material box on a discharging layer through a visual recognition system when the AGV advances to a discharging preparation point;
the three-dimensional model building module 12 is used for building a first bin three-dimensional model of the bin above the fork and a second bin three-dimensional model of the unloading layer bin by using the image data and the depth data;
the stacking requirement judging module 13 is configured to judge whether the first bin three-dimensional model and the second bin three-dimensional model meet a current stacking requirement;
an offset determination module 14, configured to determine offset data between the first bin three-dimensional model and the second bin three-dimensional model when the first bin three-dimensional model and the second bin three-dimensional model do not meet the current stacking requirement;
the pose adjusting module 15 is used for sending the offset data to a vehicle control system so that the vehicle control system can adjust the pose of the box body above the fork according to the offset data;
and the bin stacking module 16 is configured to, after the pose of the box body above the fork is adjusted, re-execute the step of obtaining image data and depth data of the bin above the fork and the unloading layer bin until the built first bin three-dimensional model and the built second bin three-dimensional model meet the current stacking requirement, and stack the bin above the fork on the unloading layer bin.
It is from top to bottom apparent, image data and the depth data of workbin above the fork and unloading layer workbin are obtained through the visual identification system in the embodiment of the application, establish the first workbin three-dimensional model of workbin above the fork and the second workbin three-dimensional model of unloading layer workbin, and then judge whether the three-dimensional model who establishes satisfies the requirement of piling up, if unsatisfying the requirement of piling up, then adjust the position appearance of fork top box according to determining offset data, when the three-dimensional model who establishes satisfies current requirement of piling up, and then pile up the workbin above the fork on unloading layer workbin, can solve the black workbin among the practical application and lack the problem of flexibility and ease for use when piling up the operation more than certain height, and can improve the precision of piling up of workbin.
In some specific embodiments, the data obtaining module 11 may specifically include:
the first data acquisition unit is used for acquiring image data of the material box above the fork and the unloading layer material box through the RGB camera in the visual recognition system;
and the second data acquisition unit is used for acquiring the depth data of the material box above the fork and the unloading layer material box through the TOF camera in the visual recognition system.
In some specific embodiments, the three-dimensional model building module 12 may specifically include:
a feature extraction unit configured to extract a detail feature in the image data;
the feature matching unit is used for matching the detail features with three-dimensional features output by a pre-constructed target neural network to obtain corresponding matching results;
the matching requirement judging unit is used for judging whether the matching result meets the preset matching requirement or not;
and the three-dimensional model building unit is used for building a first bin three-dimensional model of the bin above the fork and a second bin three-dimensional model of the unloading layer bin based on the detail features, the three-dimensional features and the depth data when the matching result meets the preset matching requirement.
In some specific embodiments, the structure matching unit may specifically include:
and the feature matching subunit is used for matching the detail features with three-dimensional features output by a target neural network model which is constructed in advance based on a Yolo5 algorithm to obtain a corresponding matching result.
In some specific embodiments, the offset determining module 14 may specifically include:
and the offset determining unit is used for determining the front-back offset and the left-right offset between the first bin three-dimensional model and the second bin three-dimensional model.
In some specific embodiments, the pose adjustment module 15 may specifically include:
and the pose adjusting unit is used for sending the offset data to a vehicle control system so that the vehicle control system can call a gantry forward moving function and a fork side moving function according to the offset data to control the AGV to adjust the pose of the box body above the fork.
Furthermore, the embodiment of the application also provides electronic equipment. FIG. 5 is a block diagram illustrating an electronic device 20 according to an exemplary embodiment, and the contents of the diagram should not be construed as limiting the scope of use of the present application in any way.
Fig. 5 is a schematic structural diagram of an electronic device 20 according to an embodiment of the present disclosure. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input output interface 25, and a communication bus 26. Wherein the memory 22 is used for storing a computer program which is loaded and executed by the processor 21 to implement the relevant steps of the AGV-based bin stacking method disclosed in any of the previous embodiments. In addition, the electronic device 20 in the present embodiment may be specifically an electronic computer.
In this embodiment, the power supply 23 is configured to provide a working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and a communication protocol followed by the communication interface is any communication protocol applicable to the technical solution of the present application, and is not specifically limited herein; the input/output interface 25 is configured to obtain external input data or output data to the outside, and a specific interface type thereof may be selected according to specific application requirements, which is not specifically limited herein.
In addition, the storage 22 is used as a carrier for resource storage, and may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage manner may be a transient storage or a permanent storage.
The operating system 221 is used for managing and controlling each hardware device on the electronic device 20 and the computer program 222, and may be Windows Server, netware, unix, linux, or the like. The computer programs 222 may further include computer programs that can be used to perform other specific tasks in addition to the computer programs that can be used to perform the AGV-based bin stacking method disclosed by any of the foregoing embodiments and executed by the electronic device 20.
Further, the present application discloses a computer readable storage medium, in which a computer program is stored, and when the computer program is loaded and executed by a processor, the steps of the AGV-based bin stacking method disclosed in any of the foregoing embodiments are implemented.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element.
While the present invention provides a AGV-based bin stacking method, apparatus, device and storage medium, the present invention has been described in detail, and the principle and implementation of the present invention are explained by applying specific examples, the description of the above examples is only used to help understanding the method and the core idea of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, the specific embodiments and the application range may be changed, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (8)
1. An AGV based bin stacking method, wherein the bins are black bins, comprising:
when the AGV travels to a discharge preparation point, acquiring image data and depth data of a material box above a fork and a discharge layer material box through a visual recognition system, and constructing a first material box three-dimensional model of the material box above the fork and a second material box three-dimensional model of the discharge layer material box by using the image data and the depth data;
judging whether the first bin three-dimensional model and the second bin three-dimensional model meet the current stacking requirement;
if the position of the pallet fork is not met, determining offset data between the first bin three-dimensional model and the second bin three-dimensional model, and sending the offset data to a vehicle control system so that the vehicle control system can adjust the position of the bin above the pallet fork according to the offset data;
after the pose of the material box above the fork is adjusted, the step of obtaining the image data and the depth data of the material box above the fork and the unloading layer material box is executed again until the built first material box three-dimensional model and the built second material box three-dimensional model meet the current stacking requirement, and the material box above the fork is stacked on the unloading layer material box;
the visual recognition system comprises an RGB camera, a TOF camera, a limit switch, an industrial personal computer and a power supply; the system comprises an AGV, an RGB camera, a TOF camera, an industrial personal computer and a pallet fork, wherein the RGB camera is horizontally and positively arranged on two sides of the AGV, the TOF camera is horizontally and positively arranged in the middle of the AGV, a limit switch is arranged at the root of the pallet fork, the industrial personal computer is used for fusing and processing the image data and the depth data to construct a three-dimensional model, and the limit switch is used for limiting the pose of a pallet on the pallet fork;
the image data and the depth data of the bin above the fork and the bin on the unloading layer are obtained through the visual recognition system, and the method comprises the following steps:
the image data of the material box above the fork and the unloading layer material box are obtained through the RGB camera in the visual recognition system, and the depth data of the material box above the fork and the unloading layer material box are obtained through the TOF camera in the visual recognition system.
2. The AGV-based bin stacking method of claim 1 wherein said using said image data and depth data to construct a first bin three-dimensional model of said bin above said forks and a second bin three-dimensional model of said unload layer bin comprises:
extracting detail features in the image data, and matching the detail features with three-dimensional features output by a pre-constructed target neural network to obtain corresponding matching results;
judging whether the matching result meets a preset matching requirement or not;
and when the matching result meets the preset matching requirement, constructing a first bin three-dimensional model of the bin above the fork and a second bin three-dimensional model of the unloading layer bin based on the detail feature, the three-dimensional feature and the depth data.
3. The AGV-based bin stacking method according to claim 2, wherein said matching said detail features with three-dimensional features of a pre-constructed target neural network output results in a corresponding matching result, comprising:
and matching the detail features with three-dimensional features output by a target neural network model constructed in advance based on a Yolo5 algorithm to obtain corresponding matching results.
4. The AGV-based bin stacking method of claim 1 wherein said determining offset data between said first bin three-dimensional model and said second bin three-dimensional model comprises:
and determining the front-back offset and the left-right offset between the first bin three-dimensional model and the second bin three-dimensional model.
5. The AGV based bin stacking method of any one of claims 1 to 4, wherein said sending said offset data into a vehicle control system for said vehicle control system to adjust the position of the bins above said forks based on said offset data comprises:
and sending the offset data to a vehicle control system so that the vehicle control system can call a gantry advancing function and a fork side shifting function according to the offset data to control the AGV to adjust the position of the material box above the fork.
6. The utility model provides a workbin stacking device based on AGV, its characterized in that, the workbin is black workbin, includes:
the data acquisition module is used for acquiring image data and depth data of a material box above the fork and a material box on a unloading layer through the visual recognition system when the AGV advances to the unloading preparation point;
the three-dimensional model building module is used for building a first bin three-dimensional model of a bin above the fork and a second bin three-dimensional model of a discharge layer bin by using the image data and the depth data;
the stacking requirement judging module is used for judging whether the first bin three-dimensional model and the second bin three-dimensional model meet the current stacking requirement;
the offset determining module is used for determining offset data between the first bin three-dimensional model and the second bin three-dimensional model when the first bin three-dimensional model and the second bin three-dimensional model do not meet the current stacking requirement;
the position and orientation adjusting module sends the offset data to a vehicle control system so that the vehicle control system can adjust the position and orientation of a material box above the fork according to the offset data;
the bin stacking module is used for re-executing the step of acquiring the image data and the depth data of the bin above the fork and the unloading layer bin after adjusting the pose of the bin above the fork until the built first bin three-dimensional model and the built second bin three-dimensional model meet the current stacking requirement, and then stacking the bin above the fork on the unloading layer bin;
the visual recognition system comprises an RGB camera, a TOF camera, a limit switch, an industrial personal computer and a power supply; the system comprises an AGV, an industrial personal computer, RGB cameras, a TOF camera, a position switch and a position control module, wherein the RGB cameras are horizontally and positively installed on two sides of the AGV, the TOF cameras are horizontally and positively installed in the middle of the AGV, the position control module is installed at the root of a pallet fork, the industrial personal computer is used for fusing and processing image data and depth data to construct a three-dimensional model, and the position control module is used for limiting the pose of a tray on the pallet fork;
the data acquisition module comprises:
the image data acquisition unit is used for acquiring image data of the material box above the fork and the material box on the unloading layer through the RGB camera in the visual recognition system;
and the depth data acquisition unit is used for acquiring the depth data of the material box above the fork and the unloading layer material box through the TOF camera in the visual recognition system.
7. An electronic device, comprising:
a memory for storing a computer program;
a processor for executing said computer program to implement the steps of the AGV-based bin stacking method according to any one of claims 1 to 5.
8. A computer-readable storage medium for storing a computer program; wherein the computer program when executed by a processor implements the steps of the AGV based bin stacking method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211068469.XA CN115123839B (en) | 2022-09-02 | 2022-09-02 | AGV-based bin stacking method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211068469.XA CN115123839B (en) | 2022-09-02 | 2022-09-02 | AGV-based bin stacking method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115123839A CN115123839A (en) | 2022-09-30 |
| CN115123839B true CN115123839B (en) | 2022-12-09 |
Family
ID=83387910
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211068469.XA Active CN115123839B (en) | 2022-09-02 | 2022-09-02 | AGV-based bin stacking method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115123839B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115848878B (en) * | 2023-02-28 | 2023-05-26 | 云南烟叶复烤有限责任公司 | AGV-based tobacco frame identification and stacking method and system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11142399B2 (en) * | 2016-10-06 | 2021-10-12 | Doerfer Corporation | Automated warehouse fulfillment system and method of operation |
| CN112429535B (en) * | 2020-11-09 | 2022-05-31 | 苏州罗伯特木牛流马物流技术有限公司 | Control system and method for multi-layer accurate stacking of ground piled cargos |
| CN112660686B (en) * | 2021-03-17 | 2021-06-01 | 杭州蓝芯科技有限公司 | Depth camera-based material cage stacking method and device, electronic equipment and system |
| CN114524209A (en) * | 2021-12-21 | 2022-05-24 | 杭叉集团股份有限公司 | AGV high-position stacking method and detection device based on double TOF cameras |
| CN114275712A (en) * | 2021-12-30 | 2022-04-05 | 中钞长城金融设备控股有限公司 | Stacking device and stacking method |
-
2022
- 2022-09-02 CN CN202211068469.XA patent/CN115123839B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN115123839A (en) | 2022-09-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7282080B2 (en) | 3D bounding boxes from 2D images and point cloud data | |
| CN106950985B (en) | Automatic delivery method and device | |
| CN110969655B (en) | Method, device, equipment, storage medium and vehicle for detecting parking space | |
| KR20210107119A (en) | Object detection method and apparatus, electronic device, and storage medium | |
| JP7263630B2 (en) | Performing 3D reconstruction with unmanned aerial vehicles | |
| KR102428050B1 (en) | Information supplement method, lane recognition method, intelligent driving method and related products | |
| WO2020215172A1 (en) | Obstacle detection method and device, mobile platform, and storage medium | |
| CN113657224B (en) | Method, device and equipment for determining object state in vehicle-road coordination | |
| CN110163930A (en) | Lane line generation method, device, equipment, system and readable storage medium storing program for executing | |
| CN115123839B (en) | AGV-based bin stacking method, device, equipment and storage medium | |
| CN108401461A (en) | Three-dimensional mapping method, device and system, cloud platform, electronic equipment and computer program product | |
| CN110163904A (en) | Object marking method, control method for movement, device, equipment and storage medium | |
| US11120280B2 (en) | Geometry-aware instance segmentation in stereo image capture processes | |
| Santamaria‐Navarro et al. | Terrain classification in complex three‐dimensional outdoor environments | |
| CN112258590B (en) | Laser-based depth camera external parameter calibration method, device and storage medium thereof | |
| WO2021016854A1 (en) | Calibration method and device, movable platform, and storage medium | |
| WO2020237516A1 (en) | Point cloud processing method, device, and computer readable storage medium | |
| CN113052109A (en) | 3D target detection system and 3D target detection method thereof | |
| CN109760064A (en) | A kind of method of adjustment and device of mobile robot self-position | |
| CN112097732A (en) | Binocular camera-based three-dimensional distance measurement method, system, equipment and readable storage medium | |
| CN112766135B (en) | Target detection method, device, electronic equipment and storage medium | |
| CN114764778A (en) | Target detection method, target detection model training method and related equipment | |
| CN113267761B (en) | Laser radar target detection and identification method, system and computer readable storage medium | |
| CN111247564A (en) | Method for constructing digital earth surface model, processing equipment and system | |
| Holz et al. | Continuous 3D sensing for navigation and SLAM in cluttered and dynamic environments |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |