CN116205561A - Material management method and system based on digital twinning - Google Patents

Material management method and system based on digital twinning Download PDF

Info

Publication number
CN116205561A
CN116205561A CN202211663514.6A CN202211663514A CN116205561A CN 116205561 A CN116205561 A CN 116205561A CN 202211663514 A CN202211663514 A CN 202211663514A CN 116205561 A CN116205561 A CN 116205561A
Authority
CN
China
Prior art keywords
layer
data
point cloud
physical
service layer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211663514.6A
Other languages
Chinese (zh)
Inventor
李淳芃
王伟博
夏为丙
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Luoyang Zhongke Artificial Intelligence Research Institute Co ltd
Original Assignee
Luoyang Zhongke Artificial Intelligence Research Institute Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Luoyang Zhongke Artificial Intelligence Research Institute Co ltd filed Critical Luoyang Zhongke Artificial Intelligence Research Institute Co ltd
Priority to CN202211663514.6A priority Critical patent/CN116205561A/en
Publication of CN116205561A publication Critical patent/CN116205561A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
    • G06Q10/087—Inventory or stock management, e.g. order filling, procurement or balancing against orders
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/12—Computing arrangements based on biological models using genetic models
    • G06N3/126—Evolutionary algorithms, e.g. genetic algorithms or genetic programming
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • Health & Medical Sciences (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Economics (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Multimedia (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Geometry (AREA)
  • Artificial Intelligence (AREA)
  • Medical Informatics (AREA)
  • Quality & Reliability (AREA)
  • Human Resources & Organizations (AREA)
  • Computer Graphics (AREA)
  • Databases & Information Systems (AREA)
  • Marketing (AREA)
  • Operations Research (AREA)
  • Strategic Management (AREA)
  • General Business, Economics & Management (AREA)
  • Development Economics (AREA)
  • Finance (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Accounting & Taxation (AREA)
  • Tourism & Hospitality (AREA)
  • Physiology (AREA)
  • Genetics & Genomics (AREA)
  • Biomedical Technology (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)

Abstract

The invention relates to a material management system and method based on digital twinning, wherein the management system comprises a physical layer, a service layer, a virtual layer and a connecting layer, wherein the physical layer collects physical data of materials conveyed on a conveyor belt and transmits the physical data to the service layer; the service layer processes and analyzes the real data acquired by the physical layer and provides informatization and intelligent service for the whole production process; the virtual layer is realized by a visual 3D engine and comprises a three-dimensional model in a warehouse and a system service module, and the data obtained by processing the service layer is used for carrying out visual mapping and providing corresponding services; the connection layer is responsible for data interaction between the physical layer and the service layer, and between the service layer and the virtual layer. The digital twin material management system can automatically generate a corresponding 3D model according to the data acquired by the physical space, and guide the production flow according to the related parameters of the model, thereby improving the production efficiency and the digital degree.

Description

Material management method and system based on digital twinning
Technical Field
The invention belongs to the field of digital twinning and intelligent manufacturing, and particularly relates to a material management method and system based on digital twinning.
Background
The digital twin is a concept and model which uses data as a tie to associate a physical space with a virtual space, real data information obtained by various sensors in the physical space is mapped to the virtual space, so that the description and modeling of the physical space are realized, and meanwhile, the data in the virtual space also intervenes in the production and manufacturing process of the physical space in a reverse way through analysis, prediction, decision making and other modes. Digital twinning is a technical system with universal applicability, is an important development direction of informatization and digital transformation in the current production and manufacturing, and is widely focused in recent years. At present, the implementation and application of the digital twin technology are still in an exploration stage, and no clear technical route and no structuring flow exist.
In recent years, with the rapid development of high and new technologies such as the internet of things and artificial intelligence, the degree of factory intellectualization and unmanned is higher and higher, and the material management system is taken as an application scene with stronger repeatability, so that the material management system is a main environment for realizing intelligent manufacturing at present.
At present, most of virtual-real mapping processes in digital twin systems need to be modeled in advance and mapped by data information, mapping flexibility for actual scenes is not high, and Chinese patent application CN112529511A discloses a storage warehouse management system based on machine vision, which takes machine vision as a core, identifies and positions information of goods, personnel and the like in images through an AI server, analyzes and obtains relevant information of the goods in real time, maps the relevant information to a visualization unit, and is convenient for banks to monitor single mortgage goods in a warehouse. Chinese patent application CN113822993a discloses a digital twin method and system based on 3D model matching, the method comprising: model and texture training, model matching, texture fusion and scene placement, matching the reconstructed model with the model stored in the database based on the IOU, forming a standard 3D model through texture fusion, and placing the model in the 3D scene, thereby improving the efficiency of digital twin.
The patent realizes the flow of digital information in the virtual and real space to a certain extent, but is limited in the aspect of physical model mapping, only a known corresponding model can be generated, and the method does not consider how to combine the information of the physical world and the virtual world more tightly, plays a role in guiding the material production process through the flow of the information, and realizes the purposes of virtual mapping and virtual control.
Disclosure of Invention
The invention aims to provide a material management method and system based on digital twinning, which acquire a material model in a physical space by means of binocular vision and reflect the material model to a virtual space, and simultaneously analyze data acquired by a sensor by utilizing an algorithm to reversely guide transportation and scheduling of materials in the physical space, so that a new idea is provided for the application and research of subsequent digital twinning in the field of intelligent manufacturing.
In order to achieve the above purpose, the technical scheme adopted by the invention is as follows: a material management system based on digital twinning comprises a physical layer, a service layer, a virtual layer and a connection layer; the physical layer is provided with a collection device for collecting physical data of the materials on the conveyor belt, and the collected physical data of the materials are transmitted to the service layer; the service layer comprises an edge service layer and a cloud service layer, wherein the edge service layer comprises a processing module for processing acquired physical data and a local database, and the cloud service layer comprises an analysis module for the physical data and a twin cloud database; the virtual layer comprises a three-dimensional model in a warehouse and a system service module, wherein the three-dimensional model in the warehouse comprises a material three-dimensional model and a goods shelf three-dimensional model, and the system service module comprises a visual angle switching module, a man-machine interaction module and a material state monitoring module; the connection layer is used for data interaction between the physical layer and the service layer and between the service layer and the virtual layer.
The acquisition device comprises a binocular camera and a laser transmitter for acquiring images, an encoder for acquiring the movement speed and a weight sensor for acquiring the mass of materials.
The binocular camera and the laser transmitter are used as a group of image devices and matched with each other, three groups of image devices are arranged in an image acquisition area of the conveyor belt, the three groups of image devices are arranged above the conveyor belt in a delta-shaped mode and irradiate materials among the three groups of image devices obliquely downwards, two groups of image devices distributed left and right are located at the rear end of the acquisition area, the other group of image devices are located at the front end of the acquisition area, and the arrangement positions of the two groups of image devices at the rear end are lower than those of the front end.
The physical data of the material comprises images, material quality data and movement speed data acquired by the binocular cameras under a plurality of angles when the material passes through the appointed position of the conveyor belt.
A material management method based on digital twinning adopts the management system, and comprises the following steps:
s1, acquiring material information of a physical layer, acquiring physical data of materials passing through a conveyor belt in real time by using an acquisition device, and transmitting the physical data to a service layer;
s2, analyzing an image algorithm of the service layer, performing feature extraction, stereo matching and point cloud splicing on the image acquired by the physical layer by the service layer to obtain three-dimensional point cloud data, and calculating the volume of the material by using the three-dimensional point cloud data;
s3, analyzing and predicting different material storage positions by using a genetic algorithm-based goods position optimization method in a service layer by taking the rest idle goods positions, the material quality and the material volume of the warehouse as input values, and solving the optimal goods positions of a plurality of materials to be put into warehouse; then, iteration is carried out through crossover and variation of a genetic algorithm to obtain an optimal storage position set of the group of materials, and the materials are guided to be stored in a warehouse;
s4, the virtual layer adopts a three.js3D engine to realize mapping of real data, actual instructions and planning results of the service layer, which are acquired by the physical layer, to the virtual warehouse; the state synchronization module of the virtual layer utilizes the information synchronization update in the local database and the cloud database to map the real-time change of the warehouse into the virtual space, when the materials of the physical warehouse are discharged, the corresponding material model in the virtual space also disappears, and the database information is updated.
In S2, three-dimensional point cloud data generation comprises two processes of multi-angle point cloud splicing and multi-frame point cloud splicing.
The multi-angle point cloud splicing is to generate corresponding point clouds from images acquired at three positions at the same moment, and rotate and translate point cloud data of single-line lasers acquired by two binocular cameras at the rear end to the coordinate system of the binocular cameras at the front end through a pre-obtained RT matrix which is mutually converted among the three binocular cameras.
And splicing the multi-frame point clouds to splice continuously acquired point cloud data according to the time flow, thereby obtaining a complete point cloud model without a bottom surface.
In S2, when the volume of the material is calculated, the surface point cloud of the conveyor belt without the material is acquired in advance, ping Miandian cloud is obtained according to a principal component analysis method, a rotation matrix and a translation matrix of the surface point cloud of the conveyor belt to an XOY plane are acquired, the coordinates of the point cloud of the material are acquired by the conveyor belt in real time, translation and rotation are carried out, point cloud data of which the bottom surface of the material is coplanar with the XOY plane are obtained, a single cube is obtained through rasterization of the point cloud data, and the volume of the point cloud of the material is obtained through three-dimensional integration.
The beneficial effects of the invention are as follows: according to the invention, the production materials are virtualized by means of the sensors in the physical scene, the acquired production information and the physical model are analyzed, the production process of the real scene is guided, and the real scene is mapped to the virtual scene correspondingly.
Compared with the prior art, the invention realizes a closed loop process combining digital twinning and material production and manufacturing, obtains the image of the material through the physical layer sensor, sends the image to the service layer to be processed to obtain the three-dimensional information of the material, predicts the storage position of the material according to the three-dimensional information and quality of the material, guides the next operation of the physical layer, and displays the corresponding material model in a digital twinning scene.
Drawings
FIG. 1 is a schematic diagram of a digital twin material management system according to the present invention;
FIG. 2 is a schematic diagram of a physical layer device according to the present invention;
FIG. 3 is a schematic diagram of a material twin body model generation and loading flow in the present invention;
FIG. 4 is a schematic diagram of a virtual warehouse rack;
fig. 5 is a schematic diagram of the cargo space after abstraction of the shelves in the warehouse.
Detailed Description
The invention is described in further detail below with reference to the drawings and examples, which are not intended to be limiting.
As shown in fig. 1 and 2, a material management system based on digital twinning comprises a physical layer, a service layer, a virtual layer and a connection layer; the method comprises the following steps: (1) The physical layer comprises a conveyor belt, a binocular camera, a laser, a weight sensor and an encoder, and the physical layer acquires physical data of materials conveyed on the conveyor belt in real time through various sensors and transmits the physical data to the service layer; (2) The service layer processes and analyzes the real data acquired by the physical layer and provides informatization and intelligent services for the whole production process, wherein the informatization and intelligent services comprise an edge service layer and a cloud service layer; the edge service layer comprises a visual reading module and a PLC reading module for obtaining physical data, a local database for storing the physical data and a processing module for processing the physical data, wherein the processing module comprises a three-dimensional point cloud generating module, an instruction generating module and a data checking module; the cloud service layer comprises an analysis module of physical data and a twin cloud database, wherein the analysis module comprises a three-dimensional point cloud analysis module, a twin model generation module and a material optimal goods position prediction module, and the cloud database adopts a relational database MySQL; the implementation of the service layer mainly comprises the acquisition and analysis of three-dimensional information of information, the generation of a virtual model and the analysis and guidance of warehouse behaviors by utilizing physical information; the connection between the edge service layer and the cloud service layer is constructed through a WebSocket real-time communication protocol, so that the transmission of local data to cloud data is realized, and the data consistency of a database in a cloud edge server is ensured; (3) The virtual layer uses a flash framework of python, and the front-end visualization function adopts an html5 embedded three.js3D engine to realize mapping of real data, actual instructions and planning results of the service layer, which are acquired by the physical layer, to the virtual warehouse; the material warehouse digital twin model of the virtual layer comprises a material three-dimensional model, a goods shelf three-dimensional model, a visual angle switching module, a man-machine interaction module and a material state monitoring module; (4) The connection layer is used for data interaction between the physical layer and the service layer, inside the service layer and between the service layer and the virtual layer.
The implementation of the above system is as follows.
(1) Physical layer implementation
The conveyor belt 5 adopts the PLC1200 as motion control equipment, the Snap7 open source library based on the S7 communication protocol is used for completing the control of a physical layer and the acquisition of sensor data in the communication process of the upper computer and the PLC, the data acquired by the sensor comprise the motion speed of the conveyor belt and the mass of materials passing through the tail end of the conveyor belt, and the mass data acquisition is completed by the weight sensor 4 positioned below the conveyor belt at the tail part of the device, as shown in fig. 2; the binocular camera and the single-line laser transmitter are used as matched image acquisition components to be arranged in three groups, a support 6 is arranged at the detection position of the conveyor belt 5, the binocular camera and the single-line laser transmitter acquire images of materials 7 entering the space of the support 6, wherein the first group of image acquisition components 1 are arranged at the top of the inlet side of the support 6 and are positioned right above the surface of the conveyor belt 5, and the second group of image acquisition components 2 and the third group of image acquisition components 3 are respectively arranged at the lower parts of the left upright post and the right upright post at the outlet side of the support 6; further, the angle between the single-line laser transmitters and the binocular camera at the inlet side of the support 6 is inclined downwards by 45 degrees, and the angle between the two single-line laser transmitters and the binocular camera at the outlet side of the support 6 is inclined downwards by 45 degrees; the method comprises the steps of obtaining internal and external parameters of three binocular cameras and an RT matrix of mutual conversion among the cameras by adopting a Zhang Zhengyou calibration method in advance, and sending the data to a database of a service layer for storage and management.
The physical data of the materials collected by the physical layer comprises: material images, material quality data, and movement speed data.
The conveyer belt is by PLC1200 control uniform velocity forward motion, drive a plurality of materials on the conveyer belt and advance, single line laser of single line laser emitter can sweep conveyer belt and material continually, record the length of conveyer belt forward motion by the encoder to gather corresponding image data by binocular camera, the weight sensor of conveyer belt afterbody discharge gate obtains the material quality, and the velocity of motion data is obtained by the encoder in real time, and corresponds with the image that gathers at that time.
(2) Service layer implementation
The service layer processes and analyzes the real data acquired by the physical layer and provides informatization and intelligent services for the whole production process, wherein the informatization and intelligent services comprise an edge service layer and a cloud service layer; the edge service layer comprises a visual reading module and a PLC reading module for obtaining physical data, a local database for storing the physical data and a processing module for processing the physical data, wherein the processing module comprises a three-dimensional point cloud generating module, an instruction generating module and a data checking module; the cloud service layer comprises an analysis module of physical data and a twin cloud database, wherein the analysis module comprises a three-dimensional point cloud analysis module, a twin model generation module and a material optimal goods position prediction module, the cloud database adopts a relational database MySQL, and data in the MySQL is analyzed by utilizing an algorithm.
In this embodiment, the material is transported on the conveyor belt to pass through the designated area and scanned by laser lines with multiple angles, and the processing module of the edge service layer performs feature extraction, stereo matching and point cloud stitching on the laser line images with the laser lines shot by the multi-angle binocular camera to obtain three-dimensional point cloud data of the material, and uses the point cloud data to complete calculation of the volume of the material, so as to obtain a complete material model.
The point cloud data generation method comprises two processes of multi-angle point cloud splicing and multi-frame point cloud splicing.
The multi-angle point cloud stitching is to generate corresponding point clouds from images acquired at three positions at the same moment, and rotate and translate point cloud data of single-line lasers acquired by two cameras at the rear end to the coordinate system of the front end camera through a pre-obtained RT matrix which is converted among the three cameras.
Specifically, the front-end camera obtains a weighted centroid connecting line corresponding to a laser line through a gray level gravity center method according to laser line images acquired by the left-right cameras at the rear end, and the formula is as follows:
Figure BDA0004013783060000051
Figure BDA0004013783060000052
f (x, y) is the gray value of the corresponding pixel point with coordinates (x, y), S is a small laser line area larger than the threshold value,
Figure BDA0004013783060000053
is the corresponding center coordinate.
Further obtaining pixel pairs connected with the centers of laser lines in left and right images through feature matching, calculating to obtain three-dimensional point cloud data relative to each camera according to the coordinates of the pixel pairs in the left and right images, the rotation matrix and the translation matrix obtained through camera calibration, and simultaneously using a rear-end left-side camera coordinate system o according to the pre-obtained RT matrix among three cameras L -x L y L z L And a right camera coordinate system o R -x R y R z R The point cloud data in the camera are converted into a coordinate system o corresponding to the front-end camera 1 -x 1 y 1 z 1 And (3) splicing the point cloud data of different angles into one coordinate system, wherein all the coordinate systems are right-hand coordinate systems, so that the transformation between the coordinate systems meets the rigid transformation. The formula for converting the point cloud data obtained by cameras on the left side and the right side of the rear end into the front end camera coordinate system is as follows:
Figure BDA0004013783060000061
Figure BDA0004013783060000062
wherein the matrix R is rotated L1 And R is R1 Is 3X 3Matrix, T, representing rotation relationship from the camera coordinate system at the left and right sides of the rear end to the front end camera coordinate system L1 And T R1 Respectively representing the translation relation from the camera coordinate systems at the two sides of the rear end to the front end.
And splicing the multi-frame point clouds to splice continuously acquired point cloud data according to the time flow, thereby obtaining a complete point cloud model without a bottom surface.
The multi-frame point cloud splicing process is as follows: under the condition that the laser is fixed, the point cloud data spliced at multiple angles are multi-section linear point clouds under different surfaces of the model, in order to restore the point cloud data of the whole moving material, the distance difference delta d between two frames is obtained according to the relative movement speed between the moving material and the camera, a plurality of adjacent frames are spliced into a three-dimensional point cloud image, and the point cloud data spliced by the multi-frame point cloud data are stored into a file with the type ply and stored in a local database.
And when the volume of the material is calculated, acquiring a three-dimensional point cloud set of the conveyor belt and the material under a reference coordinate system, wherein the material of the measured volume is placed on the plane of the conveyor belt, the origin of the reference coordinate system is positioned at the position of the optical center of the left lens of the front binocular camera, and the Z-axis direction of the reference coordinate system is vertical to the surface of the conveyor belt and the positive direction is downward. In order to avoid calibration errors and facilitate volume calculation, acquiring a material-free surface point cloud of a conveyor belt in advance, acquiring a Ping Miandian cloud according to a principal component analysis method, acquiring a rotation matrix and a translation matrix of the surface point cloud of the conveyor belt to an XOY plane, translating and rotating the material point cloud coordinate acquired by the conveyor belt in real time to acquire point cloud data of which the bottom surface of the material is coplanar with the XOY plane, preprocessing the point cloud data, denoising the point cloud data by taking the left and right limit position coordinates, the plane depth coordinates and the optical center coordinates of the conveyor belt as parameters of direct filtering, and smoothing the point cloud data by voxel filtering to ensure that the point cloud data is more accurate.
And (3) carrying out gridding treatment on the point cloud data by a three-dimensional point cloud generation module of the service layer, dividing the point cloud into a plurality of cubes with the bottom surface of 5mm multiplied by 5mm according to X, Y axis coordinates, replacing all data points by using an average value in the Z direction in each grid, calculating the volume of a single cube, and finally obtaining the volume of the point cloud through integration.
The twin model generation module of the cloud service layer obtains a twin model of a single material through point cloud processing. By splicing and rotating the pre-collected bottom surface point cloud and the scanned material point cloud, closed material point cloud data are obtained, the number of point clouds is further reduced by using voxel downsampling, the downsampled point clouds are converted into Mesh models by poisson reconstruction, files are stored in databases of local and cloud servers, the file types are obj, and a twinning model generation flow is shown in figure 3.
Further, the optimal material position prediction module of the cloud service layer analyzes and guides the next warehousing operation according to the acquired and processed physical information. And reading the coordinate values of the empty shelves from the database, taking all material transfer rate, volume information and quality information as known parameters, and continuously iterating by utilizing a genetic algorithm so as to obtain the optimal goods position for material warehouse entry.
Specifically, the cargo space in the warehouse is abstracted into a three-dimensional space shown in fig. 5, the cargo space comprises X, Y, Z directions, the number of corresponding cargo spaces is A, B, C, all cargo space numbers of the warehouse can be represented by integers in [0, (a×b×c) ], a single individual in the genetic algorithm corresponds to a storage scheme of a material, the initial population number is set to 300 before optimizing, all schemes are preprocessed through precoding, and the cargo space optimization rule of material in-out warehouse takes the stability of the goods shelf and the entrance efficiency as the fitness function considering the construction of the single individual, and the formula is as follows:
F＝w 1 ×f t +w 2 ×f m +w 3 ×f v
wherein f t 、f m 、f v Fitness values, w, corresponding to turnover rate, mass and volume fractions, respectively 1 、w 2 、w 3 The fitness value duty cycle is respectively given.
Figure BDA0004013783060000071
Wherein x, y and z respectively represent the transverse, longitudinal and high three-direction coordinates of the cargo space, L, W, H respectively represent the distance length of the cargo space in the corresponding direction, v x 、v y 、v z The speed of transporting goods in the three directions corresponds to the turnover rate of the goods, g is gravity acceleration, and T represents vector transposition;
f m ＝zgG T
wherein G is the corresponding mass of the material;
f v ＝zgV T
wherein V is the corresponding volume of the material.
And then continuously iterating, continuously updating the optimal solution through intersection and variation until the cost of the optimal solution is smaller than a threshold value or the iteration is finished, taking the result as an optimal storage goods space solution, further converting the optimal storage space solution into a TSP problem according to the space coordinate positions of a plurality of goods spaces, planning the optimal storage sequence through a particle swarm algorithm, and finally delivering the goods spaces and the storage sequence to an AGV to complete the warehousing process, and synchronously updating all three-dimensional information, model paths and corresponding analysis and prediction results into a local database and a cloud database.
(3) Virtual layer implementation
The virtual layer service uses a python flash framework, and the front-end visualization function adopts an html5 embedded three.js3D engine to map real data acquired by the physical layer, an actual instruction and a planning result of the service layer to the virtual warehouse. In the physical space, when a single material in a group of materials is placed in a designated goods space, an obj model file of the material is read from a database, a corresponding material file is obtained, the file type is mtl, the map_kd parameter in the mtl file is set as a designated material picture path, and a material picture can be obtained by using a local file or shooting by a binocular camera of a physical layer.
As shown in fig. 4, in the virtual three-dimensional space constructed by three.js, there is a pre-created shelf model, each goods location of a single shelf model has unique goods location coordinates, then according to the warehouse-in material signal in the physical space, the model file (obj) of the material is set to a designated material file (mtl), and according to the optimal goods location address stored in the database, the material model is set to be placed on the corresponding goods location in the virtual space by the position attribute of the material model. And adding event response to each model by taking the unique id corresponding to each model as the model identity identifier, acquiring the corresponding id identifier through click interaction operation, and simultaneously reading corresponding material data comprising material weight, volume warehouse-in time and picture information from a database of cloud service and displaying the material data on an interface.
The data checking module of the service layer utilizes the synchronous updating of the information in the local database and the cloud database to map the real-time change of the warehouse into the virtual space, when the materials of the physical warehouse are taken out of the warehouse, the corresponding material model in the virtual space also disappears, and the database information is updated.
(4) Connection layer implementation
The data interaction realized by the connection layer comprises the following steps: the edge service and the physical layer are connected in a wired communication mode such as a network port and USB, and data information acquired by the physical layer is transmitted to the edge service layer; when receiving the input-output-input signals, establishing connection between the edge service layer and the cloud service layer through a WebSocket real-time communication protocol, realizing transmission from local data to cloud data, and ensuring data consistency of a database in a cloud edge server; the cloud service layer and the virtual layer can directly conduct data interaction in the same system.
The above embodiments are only for illustrating the technical solution of the present invention and not for limiting it, and it should be understood by those skilled in the art that modifications and equivalents may be made to the specific embodiments of the present invention with reference to the above embodiments, and any modifications and equivalents not departing from the spirit and scope of the present invention are within the scope of the claims appended hereto.

Claims (9)

1. A digital twinning-based material management system, characterized in that: the system comprises a physical layer, a service layer, a virtual layer and a connection layer; the physical layer is provided with a collection device for collecting physical data of the materials on the conveyor belt, and the collected physical data of the materials are transmitted to the service layer; the service layer comprises an edge service layer and a cloud service layer, wherein the edge service layer comprises a processing module for processing acquired physical data and a local database, and the cloud service layer comprises an analysis module for the physical data and a twin cloud database; the virtual layer comprises a three-dimensional model in a warehouse and a system service module, wherein the three-dimensional model in the warehouse comprises a material three-dimensional model and a goods shelf three-dimensional model, and the system service module comprises a visual angle switching module, a man-machine interaction module and a material state monitoring module; the connection layer is used for data interaction between the physical layer and the service layer and between the service layer and the virtual layer.
2. A digital twinning-based material management system according to claim 1, wherein: the acquisition device comprises a binocular camera and a laser transmitter for acquiring images, an encoder for acquiring the movement speed and a weight sensor for acquiring the mass of materials.
3. A digital twinning-based material management system according to claim 2, wherein: the binocular camera and the laser transmitter are used as a group of image devices and matched with each other, three groups of image devices are arranged in an image acquisition area of the conveyor belt, the three groups of image devices are arranged above the conveyor belt in a delta-shaped mode and irradiate materials among the three groups of image devices obliquely downwards, two groups of image devices distributed left and right are located at the rear end of the acquisition area, the other group of image devices are located at the front end of the acquisition area, and the arrangement positions of the two groups of image devices at the rear end are lower than those of the front end.
4. A digital twinning-based material management system according to claim 3, wherein: the physical data of the material comprises images, material quality data and movement speed data acquired by the binocular cameras under a plurality of angles when the material passes through the appointed position of the conveyor belt.
5. A method of digital twinning-based material management, characterized in that it employs a management system according to any one of claims 1-4, comprising the steps of:
s1, acquiring material information of a physical layer, acquiring physical data of materials passing through a conveyor belt in real time by using an acquisition device, and transmitting the physical data to a service layer;
s2, analyzing an image algorithm of the service layer, performing feature extraction, stereo matching and point cloud splicing on the image acquired by the physical layer by the service layer to obtain three-dimensional point cloud data, and calculating the volume of the material by using the three-dimensional point cloud data;
s3, analyzing and predicting different material storage positions by using a genetic algorithm-based goods position optimization method in a service layer by taking the rest idle goods positions, the material quality and the material volume of the warehouse as input values, and solving the optimal goods positions of a plurality of materials to be put into warehouse; then, iteration is carried out through crossover and variation of a genetic algorithm to obtain an optimal storage position set of the group of materials, and the materials are guided to be stored in a warehouse;
s4, the virtual layer adopts a three.js3D engine to realize mapping of real data, actual instructions and planning results of the service layer, which are acquired by the physical layer, to the virtual warehouse; the state synchronization module of the virtual layer utilizes the information synchronization update in the local database and the cloud database to map the real-time change of the warehouse into the virtual space, when the materials of the physical warehouse are discharged, the corresponding material model in the virtual space also disappears, and the database information is updated.
6. The digital twinning-based material management method according to claim 5, wherein: in S2, three-dimensional point cloud data generation comprises two processes of multi-angle point cloud splicing and multi-frame point cloud splicing.
7. The digital twinning-based material management method of claim 6, wherein: the multi-angle point cloud splicing is to generate corresponding point clouds from images acquired at three positions at the same moment, and rotate and translate point cloud data of single-line lasers acquired by two binocular cameras at the rear end to the coordinate system of the binocular cameras at the front end through a pre-obtained RT matrix which is mutually converted among the three binocular cameras.
8. The digital twinning-based material management method of claim 6, wherein: and splicing the multi-frame point clouds to splice continuously acquired point cloud data according to the time flow, thereby obtaining a complete point cloud model without a bottom surface.
9. The digital twinning-based material management method according to claim 5, wherein: in S2, when the volume of the material is calculated, the surface point cloud of the conveyor belt without the material is acquired in advance, ping Miandian cloud is obtained according to a principal component analysis method, a rotation matrix and a translation matrix of the surface point cloud of the conveyor belt to an XOY plane are acquired, the coordinates of the point cloud of the material are acquired by the conveyor belt in real time, translation and rotation are carried out, point cloud data of which the bottom surface of the material is coplanar with the XOY plane are obtained, a single cube is obtained through rasterization of the point cloud data, and the volume of the point cloud of the material is obtained through three-dimensional integration.
CN202211663514.6A 2022-12-23 2022-12-23 Material management method and system based on digital twinning Pending CN116205561A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211663514.6A CN116205561A (en) 2022-12-23 2022-12-23 Material management method and system based on digital twinning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211663514.6A CN116205561A (en) 2022-12-23 2022-12-23 Material management method and system based on digital twinning

Publications (1)

Publication Number Publication Date
CN116205561A true CN116205561A (en) 2023-06-02

Family

ID=86508523

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211663514.6A Pending CN116205561A (en) 2022-12-23 2022-12-23 Material management method and system based on digital twinning

Country Status (1)

Country Link
CN (1) CN116205561A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116502785A (en) * 2023-06-30 2023-07-28 深圳市渐近线科技有限公司 Warehouse logistics intelligent management method, device, equipment and storage medium

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116502785A (en) * 2023-06-30 2023-07-28 深圳市渐近线科技有限公司 Warehouse logistics intelligent management method, device, equipment and storage medium
CN116502785B (en) * 2023-06-30 2024-01-16 深圳市渐近线科技有限公司 Warehouse logistics intelligent management method, device, equipment and storage medium

Similar Documents

Publication Publication Date Title
CN114004941B (en) Indoor scene three-dimensional reconstruction system and method based on nerve radiation field
CN110363058B (en) Three-dimensional object localization for obstacle avoidance using one-shot convolutional neural networks
CN112700499B (en) Deep learning-based visual positioning simulation method and system in irradiation environment
CN108509820B (en) Obstacle segmentation method and device, computer equipment and readable medium
CN109084746A (en) Monocular mode for the autonomous platform guidance system with aiding sensors
CN108961327A (en) A kind of monocular depth estimation method and its device, equipment and storage medium
CN113112504A (en) Plant point cloud data segmentation method and system
CN107483911A (en) A kind of signal processing method and system based on more mesh imaging sensors
Chen et al. An Apple Detection Method Based on Des‐YOLO v4 Algorithm for Harvesting Robots in Complex Environment
Li et al. Occluded apple fruit detection and localization with a frustum-based point-cloud-processing approach for robotic harvesting
CN107491071A (en) A kind of Intelligent multi-robot collaboration mapping system and its method
Brekke et al. Multimodal 3d object detection from simulated pretraining
CN114693785A (en) Target positioning method, system and related equipment
EP4336385A1 (en) Method and apparatus for updating target detection model
CN116205561A (en) Material management method and system based on digital twinning
CN111914615A (en) Fire-fighting area passability analysis system based on stereoscopic vision
CN108367436A (en) Determination is moved for the voluntary camera of object space and range in three dimensions
CN113034571A (en) Object three-dimensional size measuring method based on vision-inertia
CN116563466A (en) Deep learning-based three-dimensional Shan Mudian cloud completion method
CN114708475A (en) Point cloud multi-mode feature fusion network method for 3D scene understanding
Sleaman et al. Indoor mobile robot navigation using deep convolutional neural network
CN116824433A (en) Visual-inertial navigation-radar fusion self-positioning method based on self-supervision neural network
Palla et al. Fully Convolutional Denoising Autoencoder for 3D Scene Reconstruction from a single depth image
CN115359119A (en) Workpiece pose estimation method and device for disordered sorting scene
Zhang et al. UAVStereo: A Multiple Resolution Dataset for Stereo Matching in UAV Scenarios

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination