CN118334018A - Method for automatically identifying and positioning battery protection plate feeding device - Google Patents
Method for automatically identifying and positioning battery protection plate feeding device Download PDFInfo
- Publication number
- CN118334018A CN118334018A CN202410750331.0A CN202410750331A CN118334018A CN 118334018 A CN118334018 A CN 118334018A CN 202410750331 A CN202410750331 A CN 202410750331A CN 118334018 A CN118334018 A CN 118334018A
- Authority
- CN
- China
- Prior art keywords
- installation
- standard
- dimensional model
- battery protection
- algorithm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 69
- 238000009434 installation Methods 0.000 claims abstract description 199
- 230000033001 locomotion Effects 0.000 claims abstract description 61
- 238000001514 detection method Methods 0.000 claims abstract description 24
- 230000000007 visual effect Effects 0.000 claims abstract description 23
- 230000001681 protective effect Effects 0.000 claims abstract description 18
- 238000011900 installation process Methods 0.000 claims abstract description 17
- 238000012545 processing Methods 0.000 claims abstract description 13
- 238000012544 monitoring process Methods 0.000 claims abstract description 11
- 238000004422 calculation algorithm Methods 0.000 claims description 157
- 230000007547 defect Effects 0.000 claims description 39
- 239000013598 vector Substances 0.000 claims description 33
- 230000011218 segmentation Effects 0.000 claims description 26
- 238000004458 analytical method Methods 0.000 claims description 21
- 238000005457 optimization Methods 0.000 claims description 20
- 238000010586 diagram Methods 0.000 claims description 17
- 238000012549 training Methods 0.000 claims description 17
- 238000013507 mapping Methods 0.000 claims description 15
- 239000011159 matrix material Substances 0.000 claims description 15
- 238000004088 simulation Methods 0.000 claims description 14
- 238000013439 planning Methods 0.000 claims description 13
- 230000005477 standard model Effects 0.000 claims description 13
- 230000009466 transformation Effects 0.000 claims description 13
- 238000007781 pre-processing Methods 0.000 claims description 11
- 238000009826 distribution Methods 0.000 claims description 9
- 238000001914 filtration Methods 0.000 claims description 9
- 238000012360 testing method Methods 0.000 claims description 9
- 238000000605 extraction Methods 0.000 claims description 7
- 238000012795 verification Methods 0.000 claims description 7
- 238000003708 edge detection Methods 0.000 claims description 6
- 238000002372 labelling Methods 0.000 claims description 6
- 238000013528 artificial neural network Methods 0.000 claims description 5
- 238000013523 data management Methods 0.000 claims description 5
- 230000003993 interaction Effects 0.000 claims description 5
- 238000005259 measurement Methods 0.000 claims description 5
- 238000013441 quality evaluation Methods 0.000 claims description 5
- 239000003990 capacitor Substances 0.000 claims description 4
- 238000007635 classification algorithm Methods 0.000 claims description 4
- 238000009499 grossing Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000012217 deletion Methods 0.000 claims description 3
- 230000037430 deletion Effects 0.000 claims description 3
- 230000004927 fusion Effects 0.000 claims description 3
- 238000003384 imaging method Methods 0.000 claims description 3
- 239000000463 material Substances 0.000 claims description 3
- 230000009467 reduction Effects 0.000 claims description 3
- 239000002699 waste material Substances 0.000 claims description 3
- 230000001172 regenerating effect Effects 0.000 claims description 2
- 238000011179 visual inspection Methods 0.000 claims 1
- 230000000875 corresponding effect Effects 0.000 description 37
- 239000010410 layer Substances 0.000 description 26
- 230000008569 process Effects 0.000 description 24
- 230000006870 function Effects 0.000 description 16
- 238000013135 deep learning Methods 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 9
- 238000007726 management method Methods 0.000 description 7
- 230000009471 action Effects 0.000 description 6
- 238000013461 design Methods 0.000 description 6
- 238000004519 manufacturing process Methods 0.000 description 6
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 5
- 239000000523 sample Substances 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 230000001788 irregular Effects 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 238000010845 search algorithm Methods 0.000 description 3
- 101000911753 Homo sapiens Protein FAM107B Proteins 0.000 description 2
- 238000012897 Levenberg–Marquardt algorithm Methods 0.000 description 2
- 102100026983 Protein FAM107B Human genes 0.000 description 2
- 238000003070 Statistical process control Methods 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 2
- 230000002457 bidirectional effect Effects 0.000 description 2
- 238000009833 condensation Methods 0.000 description 2
- 230000005494 condensation Effects 0.000 description 2
- 230000002596 correlated effect Effects 0.000 description 2
- 238000006073 displacement reaction Methods 0.000 description 2
- 238000005265 energy consumption Methods 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 230000002068 genetic effect Effects 0.000 description 2
- 230000005484 gravity Effects 0.000 description 2
- 238000003709 image segmentation Methods 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 230000007704 transition Effects 0.000 description 2
- 238000013519 translation Methods 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 229920000122 acrylonitrile butadiene styrene Polymers 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000004888 barrier function Effects 0.000 description 1
- 238000005452 bending Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000001276 controlling effect Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000013016 damping Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 230000002950 deficient Effects 0.000 description 1
- 239000012636 effector Substances 0.000 description 1
- 238000005187 foaming Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012804 iterative process Methods 0.000 description 1
- 239000002346 layers by function Substances 0.000 description 1
- 238000011068 loading method Methods 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 230000035772 mutation Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000000704 physical effect Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000000513 principal component analysis Methods 0.000 description 1
- 238000012847 principal component analysis method Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 238000003466 welding Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02E—REDUCTION OF GREENHOUSE GAS [GHG] EMISSIONS, RELATED TO ENERGY GENERATION, TRANSMISSION OR DISTRIBUTION
- Y02E60/00—Enabling technologies; Technologies with a potential or indirect contribution to GHG emissions mitigation
- Y02E60/10—Energy storage using batteries
Landscapes
- Secondary Cells (AREA)
- Image Analysis (AREA)
Abstract
The application provides a method for automatically identifying and positioning a battery protection plate feeding device, which comprises the following steps: collecting standard images of battery protection boards of all preset shape categories, processing the standard images, and extracting standard shape data comprising geometric shapes and contour features; generating a standard three-dimensional model according to the standard shape data and the preset shape category associated with the standard image, and carrying out hierarchical structure identification on the standard three-dimensional model; acquiring a protective plate image of a battery protective plate to be installed, generating a corresponding protective plate three-dimensional model, and matching the corresponding shape type and the hierarchical structure; monitoring the battery protection board through a multi-angle visual detection system, and analyzing the surface quality and the element integrity of the battery protection board; if the collision risk exists in the installation process of the battery protection plate based on the mechanical arm movement track installation, generating a collision early warning signal.
Description
Technical Field
The invention relates to the technical field of information, in particular to a method for automatically identifying and positioning a battery protection plate feeding device.
Background
In the automatic production process of battery protection plates, the recognition of irregular shapes and the processing of multi-layer structures are two main technical difficulties, the two difficulties are mutually interwoven, complex contradictions and challenges are brought to the design and realization of equipment, different regular protection plates are caused by special requirements of batteries in various application scenes, the requirements require the protection plates to be custom designed in shape and structure, different from regular rectangles or circles, the irregular-shaped protection plates have irregular edges and contours, and the traditional template matching or geometric feature extraction method is difficult to work. The device needs to have a more flexible and intelligent visual algorithm, be able to adaptively identify and position the various shapes of the protective plates, and the irregular shape also presents challenges to the motion control and accuracy of the robotic arm, requiring finer and dynamic trajectory planning and servo control. The interior of the protective plate may contain a plurality of functional layers, such as a circuit layer, an insulating layer, an element layer and the like, and the position relationship and interconnection manner between the different layers increase the complexity of visual recognition, so that the device needs to have the capability of three-dimensionally reconstructing and analyzing the multi-layer structure, can accurately position and direction of each layer, and can accurately align and operate. The assembly and connection of the multi-layer structure also puts higher demands on the mechanical control of the equipment, more flexible and coordinated multi-axis motion control is needed, the production requirements, cost effectiveness and technical feasibility are comprehensively considered, and the flexible configuration and quick switching of the equipment are realized by adopting a modularized, parameterized and intelligent design method. These difficulties are interleaved together, providing serious challenges for visual identification, mechanical control and system integration of devices, requiring comprehensive utilization of technologies such as computer vision, robot control and artificial intelligence, and continuously innovating and optimizing design and implementation schemes of devices to meet increasing demands for automated production.
Disclosure of Invention
The invention provides a method for automatically identifying and positioning a battery protection plate feeding device, which mainly comprises the following steps:
Collecting standard images of battery protection boards of all preset shape categories, processing the standard images, and extracting standard shape data comprising geometric shapes and contour features;
generating a standard three-dimensional model according to the standard shape data and the preset shape category associated with the standard image, and carrying out hierarchical structure identification on the standard three-dimensional model;
acquiring a preset installation strategy corresponding to the standard three-dimensional model, wherein the preset installation strategy comprises integral embedded parameters and element wiring parameters, and associating the preset installation strategy with a preset shape class and a hierarchical structure;
Acquiring a protective plate image of a battery protective plate to be installed, generating a corresponding protective plate three-dimensional model, and matching the corresponding shape type and the hierarchical structure;
If the battery protection board to be installed is not matched with any preset shape category, associating the standard three-dimensional model with a corresponding preset installation strategy, training to obtain an installation strategy generation model, and generating a corresponding installation strategy according to the three-dimensional model of the battery protection board to be installed;
Before the battery protection board to be installed is installed through the mechanical arm, the corresponding mechanical arm movement track is configured for the installation strategy corresponding to the battery protection board to be installed by combining the installation environment information of the battery protection board to be installed, so that the mechanical arm can grasp the battery protection board to be installed according to the mechanical arm movement track, and the installation operation is completed;
Monitoring the battery protection board through a multi-angle visual detection system, and analyzing the surface quality and the element integrity of the battery protection board;
If the collision risk exists in the installation process of the battery protection plate based on the mechanical arm movement track installation, generating a collision early warning signal.
The technical scheme provided by the embodiment of the invention can have the following beneficial effects:
According to the method for automatically identifying and positioning the battery protection plate feeding device, the corresponding three-dimensional model and the corresponding installation strategy are generated, so that manual intervention is reduced, and the production efficiency is improved; the installation strategy is automatically generated through an installation strategy generation model obtained through training, so that the application range of the system is enlarged; through deep learning of standard shape data and installation strategies, optimization of installation actions, such as avoiding edge collision, and ensuring safety of an installation process and integrity of a protection plate; through the hierarchical structure identification and the automatic processing of the element wiring parameters, the system can accurately identify the element position, automatically generate a wiring path and improve the accuracy and efficiency of wiring; combining with the installation environment information, customizing the motion trail of the mechanical arm, ensuring the accurate execution of the installation operation, and reducing the installation error caused by human factors; the method can comprehensively monitor the installation process and the result, discover the problems of surface quality and element integrity in time, and ensure the product quality; the collision risk in the installation process can be recognized in advance, a warning is generated, the accident is effectively prevented, and the safety of equipment is protected.
Drawings
Fig. 1 is a flowchart of a method for automatically identifying and positioning a battery protection plate feeding device according to the present invention.
Fig. 2 is a schematic diagram of a method for automatically identifying and positioning a battery protection plate feeding device according to the present invention.
Fig. 3 is a schematic view of a method for automatically identifying and positioning a battery protection plate loading device according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in detail with reference to the accompanying drawings and specific embodiments.
As shown in fig. 1-3, a method for automatically identifying and positioning a battery protection board feeding device in this embodiment specifically may include:
Step S101, standard images of battery protection boards of all preset shape categories are collected, the standard images are processed, and standard shape data including geometric shapes and outline features are extracted.
Collecting standard images of the battery protection plates of all types according to the battery protection plates of the preset shape types; preprocessing the standard image through image processing, wherein the preprocessing comprises noise reduction and enhancement operations; and extracting standard shape data from the preprocessed standard image, and acquiring the geometric shape and contour characteristics of the battery protection board in the standard image through an edge detection and contour extraction algorithm.
Specifically, standard images of all types are collected according to the preset shape types of the battery protection plate, and preprocessing operations such as noise reduction and enhancement are performed on the standard images through an image processing technology, so that the image quality is improved, and preparation is made for subsequent feature extraction. And extracting shape characteristics of the preprocessed standard image, and acquiring geometric shape and contour characteristic data of the battery protection plate in the image through algorithms such as edge detection, contour extraction and the like. For example, the image is processed by a Canny edge detection algorithm, a high threshold is set to 150, and a low threshold is set to 50, so as to obtain a binarized edge image. Then, a contour extraction algorithm, such as a Suzuki85 algorithm, is applied to extract the contour of the battery protection plate in the image. By calculating the perimeter and area of the outline, the geometry of the protection plate is obtained, which is 80 pixels long, 60 pixels wide, 280 pixels perimeter and 4800 square pixels area. And further analyzing the shape characteristics of the outline, judging the shape type of the protection plate by calculating parameters such as the circularity, the rectangularity and the like of the outline, and considering the shape type as a circular protection plate when the circularity is more than 9 and as a rectangular protection plate when the rectangularity is more than 95. And carrying out batch processing and statistics on a batch of sample images to obtain the quantity distribution of the protection plates with various shapes and types, wherein the round protection plate accounts for 35% and the rectangular protection plate accounts for 65%. These geometric and shape feature data may be used for defect detection and classification recognition tasks.
Step S102, generating a standard three-dimensional model according to the standard shape data and the preset shape category associated with the standard image, and carrying out hierarchical structure identification on the standard three-dimensional model.
According to the standard shape data extracted from the standard image, combining preset shape categories, and generating a standard three-dimensional model of each preset shape category through a three-dimensional reconstruction algorithm; performing grid simplification and smoothing on the generated standard three-dimensional model; dividing the standard three-dimensional model into different parts and layers according to the geometric structure and topological relation of the standard three-dimensional model by utilizing a three-dimensional model segmentation algorithm to form a hierarchical structure tree; according to the hierarchical structure tree of the standard three-dimensional model, semantic labeling is carried out on each component by adopting a rule-based method or a learning-based method, semantic attributes and category identifiers are given to the component, and a mapping relation between semantic information and geometric information of the standard three-dimensional model is established; carrying out attribute analysis on the standard three-dimensional model after semantic annotation, extracting geometric attributes and topological attributes of each component, and constructing attribute feature vectors of the standard three-dimensional model; clustering and grouping the components of the standard three-dimensional model according to the attribute feature vector by adopting a clustering algorithm to obtain the similarity and the relevance among different components, and forming a new-level semantic structure; and comparing the clustering grouping result with the preset shape category, and matching the parts of the standard three-dimensional model with the standard category through similarity calculation and threshold judgment to realize classification and identification of the standard three-dimensional model.
Specifically, the standard three-dimensional model of each category is restored by a three-dimensional reconstruction algorithm based on multi-view geometry, such as SFM or MVS, using different perspective information of the standard image. And (3) carrying out grid simplification and smoothing on the generated three-dimensional model, reducing the number of faces and noise of the model, and improving the quality and rendering efficiency of the model. In order to simplify a complex three-dimensional model into a plurality of semantically distinct components, facilitating subsequent structural representation and attribute analysis, three-dimensional model segmentation algorithms such as region-growing-based segmentation or graph-cutting-based segmentation are utilized to divide the three-dimensional model into different components according to its geometric and topological relationships. And similar components are combined layer by adopting a bottom-up hierarchical clustering algorithm, such as condensation hierarchical clustering, so as to form a tree-shaped hierarchical structure. According to the hierarchical structure of the three-dimensional model, a rule-based method is adopted, and a predefined shape template library and attribute rules are utilized for matching and labeling; or adopting a learning-based method, performing end-to-end semantic segmentation and labeling on the model by using a three-dimensional convolutional neural network in deep learning, endowing each component with definite semantic attribute and category identification, and establishing a mapping relation between semantic information and geometric information of the three-dimensional model. And carrying out attribute analysis on the marked three-dimensional model, extracting geometric attributes such as size, volume, surface area and the like of each component, topological attributes such as adjacency relationship, father-son relationship and the like, and constructing an attribute feature vector of the three-dimensional model. Clustering the components of the three-dimensional model according to attribute feature vectors by adopting a clustering algorithm such as k-means or hierarchical clustering; the similarity and the relevance among different components are found out through the similarity among the metric attribute feature vectors such as Euclidean distance, mahalanobis distance and the like and the similarity among the topological structures of the metric model such as the kernel function and the like, so that a semantic structure with a higher level is formed. And comparing the clustering analysis result with a preset battery protection plate shape category, and matching the parts of the three-dimensional model with a standard category through similarity calculation and threshold judgment to realize automatic classification and identification of the three-dimensional model. When the three-dimensional reconstruction is carried out, an SFM algorithm can be adopted, SIFT feature points between standard images are extracted, a base matrix and an essential matrix between the images are estimated by using a RANSAC algorithm, further the relative pose transformation between the images is calculated, the three-dimensional coordinates of the feature points are restored by using a triangulation principle, and finally the three-dimensional model is obtained through Poisson surface reconstruction. the number of the reconstructed model faces is large, a EdgeCollapse algorithm can be used for grid simplification, the number of the model faces is reduced from 10 ten thousand to 5000, and meanwhile key geometric features are reserved. Then, the simplified model is smoothed by using a Laplacian smoothing algorithm, and is iterated for 10 times to eliminate grid noise. For semantically representing the model, a graph can be constructed for weight by using a graph-cut-based segmentation algorithm and by using a maximum flow minimum cut principle, and the model is segmented into 10 parts by using geometric attributes of edges such as normal vector included angles and semantic attributes such as whether two surfaces connected by the edges belong to the same part or not, so as to solve global optimal segmentation. and then, using a condensation hierarchical clustering algorithm, and combining the parts with similar semantics layer by taking the minimum distance between the parts as similarity measurement to form a 4-layer tree structure. For each component, 10 features of geometric attributes such as length, width, height, volume and the like are extracted, and 5 features of topological attributes such as adjacency relation with other components, depth in a hierarchical tree and the like are extracted to form a 15-dimensional attribute feature vector. And finally, clustering attribute feature vectors of the components by adopting a k-means algorithm, setting k=3, and classifying the components with the distance smaller than a threshold value such as 2.0 into the same category by calculating Euclidean distance between each component and a clustering center so as to realize automatic semantic labeling and classification recognition of the three-dimensional model.
Step S103, a preset installation strategy corresponding to the standard three-dimensional model is obtained, wherein the preset installation strategy comprises integral embedded parameters and element wiring parameters, and the preset installation strategy is associated with a preset shape class and a hierarchical structure.
Acquiring a preset installation strategy corresponding to the standard three-dimensional model from a pre-constructed knowledge base according to the shape category and the hierarchical structure of the standard three-dimensional model, wherein the preset installation strategy comprises integral placement parameters and element wiring parameters, the integral placement parameters comprise installation positions, installation directions and installation sequences, and the element wiring parameters comprise wiring modes, line sequences and line types of all elements; carrying out semantic analysis and structural representation on the preset installation strategy, extracting key installation links and elements, and forming a structural installation instruction sequence; mapping and associating the extracted installation instruction sequence with the hierarchical structure of the three-dimensional model, and generating an installation position and an installation parameter of each component according to the geometric attribute and the topological relation of each component of the model; adopting a rule-based reasoning method, and deducing an installation flow and a dependency relationship according to the logic relationship and the sequence of each link in the installation instruction sequence; simulation verification is carried out on the installation flow obtained by reasoning, feasibility and correctness of the installation process are verified through virtual assembly and interference check, and the part with problems is optimized and adjusted; integrating the verified preset installation strategy with the three-dimensional model to generate a digital virtual prototype containing installation information, storing the digital virtual prototype into a product data management system, associating the digital virtual prototype with product data outside the product data management system, and realizing digital management and sharing of the preset installation strategy; in the production and assembly process, the digital virtual prototype and a physical product are registered and displayed in a superimposed manner in real time through AR augmented reality, so that visual installation indication is provided for assembly staff.
Specifically, according to the shape type and the hierarchical structure of the standard three-dimensional model, a preset installation strategy corresponding to the standard three-dimensional model is obtained from a pre-built knowledge base, wherein the preset installation strategy comprises information such as the overall installation position, the installation direction, the installation sequence and the like of the battery protection plate, and information such as the wiring mode, the line sequence and the line type of each element, and the information is stored in a product data management system to serve as a data base of a subsequent step. Semantic analysis and structural representation are carried out on the obtained preset installation strategy, and a method based on natural language processing, such as semantic role labeling, dependency syntactic analysis and the like, is adopted to convert an installation strategy text into a structural data format, such as JSON or XML, and key installation links and elements are extracted to form a structural installation instruction sequence. And mapping and correlating the extracted installation instruction sequence with the hierarchical structure of the three-dimensional model, adopting a constraint-based assembly planning algorithm according to the geometric attribute and the topological relation of each part of the model, automatically deducing the installation position and the posture of each part, and calculating parameters required by installation, such as the position and the size of an installation hole, the model number and the length of a screw, and the like. And (3) adopting a knowledge-based reasoning method, and utilizing a pre-constructed installation rule base to infer complete installation steps and sequences according to keywords and semantic relations in an installation instruction sequence, so as to obtain a complete installation flow. And (3) performing simulation verification on the installation flow obtained by reasoning, verifying the feasibility and the correctness of the installation process through the technologies of virtual assembly, interference check and the like, and optimizing and adjusting the part with the problem. Integrating the verified installation strategy with the three-dimensional model, generating a digital virtual prototype containing complete installation information, storing the digital virtual prototype into a product data management system, and associating the digital virtual prototype with other product data to realize digital management and sharing of the installation strategy. In the actual production and assembly process, a visual-based tracking algorithm such as ORB feature point matching, pnP solving and the like is adopted through an AR augmented reality technology, the pose of a physical product is tracked in real time, and a digital virtual prototype is displayed in a corresponding position in a superimposed manner; meanwhile, by adopting a semantic segmentation algorithm based on deep learning, different parts of a physical product are identified in real time, so that virtual and real accurate alignment is realized, visual installation guidance is provided for assembly staff, and assembly efficiency and accuracy are improved. When the preset installation strategy is obtained, an ontology-based knowledge representation method can be adopted, an installation strategy knowledge base is represented as an ontology model containing concepts, attributes and relations, and knowledge retrieval and reasoning are achieved through SPARQL query languages, for example, the attribute value of the installation position under the concept of a battery protection board is queried. During semantic analysis, a BiLSTM-CRF model can be adopted to carry out semantic role marking, semantic roles such as actions, objects, tools and the like in an installation strategy text are identified and stored in a JSON format, and the actions are stored in a { "action": "placing", "object": "battery protection board" } ", position": "battery compartment" }. During assembly planning, a method based on geometric reasoning can be adopted, the B-Rep representation of a three-dimensional model is utilized, geometric information such as normal vectors and center points of all the surfaces is extracted, and the assembly sequence and pose of all the components are determined through spatial relationship reasoning such as parallelism, verticality, tangency and the like, so that an assembly path is generated. During assembly simulation, a virtual assembly technology based on a physical engine, such as a bullets, physX and the like, can be adopted to perform physical simulation on the assembly process, detect whether collision interference exists between components, calculate stress and deformation conditions, and optimize assembly paths and parameters. When AR assembly is guided, an ORB-SLAM 2-based monocular vision SLAM algorithm can be adopted to track the pose of a camera in real time, a CAD model is displayed in a superimposed mode in an edge line block diagram mode, an RGB-D-based instance segmentation algorithm MaskR-CNN is adopted to identify and segment physical components in real time, accurate registration of a virtual model and real components is achieved, and assembly steps and notes are displayed in an AR label mode.
And carrying out hierarchical structure identification on the standard three-dimensional model, judging a hierarchical overlapping region, carrying out importance assessment on key elements in the hierarchical overlapping region, and adjusting a corresponding preset installation strategy based on the importance of the key elements.
Carrying out hierarchical structure identification on the standard three-dimensional model, adopting a hierarchical segmentation algorithm based on a graph to segment the standard three-dimensional model into substructures with different levels, and calculating topological relations and geometric relations among the substructures; judging a layer overlapping region by analyzing the space position overlapping information among the substructures, and searching all elements in the overlapping region by adopting a three-dimensional space query algorithm based on voxel grids; according to the function and connection relation of the elements in the product structure, calculating importance scores of each element by adopting a network-based importance evaluation method, and determining key elements according to score ordering; comparing the importance scores of the key elements with the priorities in the preset installation strategies, and if the importance scores are higher than the preset priorities, advancing the installation sequence of the corresponding key elements and adjusting the installation mode and the installation parameters of the corresponding key elements; searching an installation scheme in the adjusted installation strategy space by adopting a heuristic search algorithm, and generating an assembly operation instruction by taking the assemblability and the assembly efficiency of the key elements as optimization targets; associating the adjusted installation strategy with the standard three-dimensional model, adopting an information model standard for product life cycle management, integrating geometric information, process parameters and quality requirements of an assembly process with characteristics, dimensions and tolerances of the standard three-dimensional model, and realizing digital twin; uploading the correlated standard three-dimensional model and the installation strategy to an MES system, and realizing real-time monitoring and visual management of the assembly process by adopting Web-based three-dimensional visualization to provide assembly indication and quality tracing for field operators.
Specifically, hierarchical structure identification is carried out on a standard three-dimensional model, a topology segmentation algorithm based on a Reeb graph is adopted, critical points, such as extreme points and saddle points, of the model are firstly extracted, then the Reeb graph is constructed according to connectivity among the critical points, different hierarchical substructures are divided according to branch structures of the Reeb graph, and the topology relationship and the geometric relationship among the substructures are calculated. And (3) judging a hierarchy overlapping region by analyzing the space position overlapping condition among the substructures, and rapidly searching all elements in the overlapping region by adopting a three-dimensional space query algorithm based on voxel grids, such as octree or KD tree. According to the function and connection relation of the elements in the product structure, the elements are regarded as nodes in the network, the geometric contact or constraint relation between the elements is regarded as edges, and the undirected weighted network is constructed. The weight of the node is calculated according to physical properties such as the volume, the mass and the like of the element, and the weight of the edge is calculated according to geometric properties such as the contact area, the constraint force and the like. On the basis, a network-based importance evaluation algorithm such as PageRank or HITS is adopted to calculate the importance score of each element, and key elements are determined according to the score ranking. And comparing the importance scores of the key elements with priorities in a preset installation strategy, and if the importance scores are higher than the preset priorities, advancing the installation sequence of the elements, and adjusting the installation mode and the installation parameters, such as the installation moment, the installation speed and the like. And adopting heuristic search algorithms such as a genetic algorithm or an ant colony algorithm, taking the total assembly time as an optimization target, taking the installation priority, accessibility, stability and the like of key elements as constraint conditions, searching an optimal installation scheme in an adjusted installation strategy space, and generating a final assembly operation instruction. The adjusted installation strategy is associated with the three-dimensional model, and the STEP-AP239 and other information model standards specially used for product life cycle management are adopted to integrate geometric information, technological parameters, quality requirements and the like of the assembly process with the characteristics, the dimensions, the tolerance and the like of the three-dimensional model, so that digital twin of the assembly process is realized. Uploading the correlated three-dimensional model and the installation strategy to an MES system, deploying the RFID, the sensor and other Internet of things equipment on an assembly site, and collecting the position, the moment, the time and other parameters in the assembly process in real time. The three-dimensional visualization technology such as WebGL is adopted, the acquired data are combined with the three-dimensional model, the visual simulation animation of the assembly process is generated, the real-time monitoring and visual management of the assembly process are realized, and visual assembly guidance and quality tracing are provided for field operators. When the hierarchical structure is identified, firstly, a three-dimensional model is abstracted into a high-dimensional manifold by adopting a critical point-based Reeb graph construction algorithm through morse theory, a maximum value point, a minimum value point and a saddle point of the model surface are extracted as critical points, then, according to the position and connectivity of the critical points, nodes and edges of the Reeb graph are generated, wherein the nodes represent substructures, the edges represent topological relations among the substructures, and finally, the Reeb graph containing 5 layers is obtained. When judging the layering overlapping area, adopting an octree space division algorithm to recursively divide the three-dimensional model space into a plurality of voxels with the size of 50mm multiplied by 50mm, establishing octree indexes for each voxel, then traversing the voxels contained in each substructure in the Reeb graph, rapidly searching other substructures overlapped with the substructures through the octree indexes, calculating the overlapping volume ratio, and finally determining that the overlapping area contains 8 key elements such as a battery compartment, a radiator, a protection board and the like. When the importance of the elements is evaluated, firstly, the matching relation among the elements is extracted based on a CAD model of the product structure, the element association network is constructed by geometric constraints such as coaxiality, parallelism and the like and physical connections such as bolting, welding and the like, then, a HITS algorithm is adopted, hub values and authiness values of each element are calculated through matrix solving, and the Hub values and authiness values are used as importance scores, wherein the battery (0.35), the protection board (0.28) and the radiator (0.2) are the highest in score. When the installation strategy is adjusted, the installation sequence of the battery is advanced from the 5 th step to the 2 nd step, the installation mode is changed from horizontal to vertical, the installation moment is increased from 1.5 N.m to 2 N.m, the installation sequence of the protection plate is advanced from the 7 th step to the 4 th step, the installation time is shortened from 30s to 20s, the installation position of the radiator is changed from the left side to the right side, and the distance from the battery is increased from 10mm to 15mm. When the optimal installation scheme is generated, a multi-objective genetic algorithm is adopted, the total assembly time, the priority of key elements and the assembly precision are taken as optimization targets, the assembly direction, the accessibility and the support stability are taken as constraint conditions, real number coding is adopted, the crossover probability is 0.8, the mutation probability is 0.1, iteration is 500 generations, the population scale is 100, the assembly time is finally shortened to 38min from the original 50min, and the assembly precision is improved to 0.12mm from 0.2 mm. When the digital twin in the assembly process is realized, STEP-AP239 standard is adopted, technological parameters such as assembly track, assembly force, assembly time and the like are fused with PMI information of the three-dimensional model, a digital twin model containing 1500 features is generated, and the digital twin model is uploaded to an MES system. When visual monitoring is realized, 30 RFID tags and 10 six-dimensional force sensors are arranged on an assembly site, assembly parameters are acquired once every 5s, the assembly parameters are uploaded to an MES system in real time through a wireless network, a Unity3D engine is adopted, 20 camera pictures of the assembly site are combined, the acquired assembly parameters are mapped onto a digital twin model, a three-dimensional visual assembly simulation interface is generated, the refreshing frequency is 10Hz, a manager can monitor the assembly process 360 degrees without dead angles, and once the deviation is found to be more than 0.15mm or the moment is found to be more than 2.5 N.m, the system automatically alarms, and the manager is guided to correct.
Step S104, obtaining a protective plate image of the battery protective plate to be installed, generating a corresponding protective plate three-dimensional model, and matching the corresponding shape type and the hierarchical structure.
Collecting multi-view images of the battery protection plate to be installed; obtaining the internal and external parameters of the image through camera calibration; adopting an SFM algorithm, and obtaining a three-dimensional point cloud model of the protection plate through a characteristic point matching and triangulation principle; preprocessing the three-dimensional point cloud model, and removing outliers by adopting a statistical filtering algorithm; converting the three-dimensional point cloud model into a triangular mesh curved surface model by using a Poisson surface reconstruction algorithm to obtain a three-dimensional model of the protection board; extracting shape characteristics of the three-dimensional model of the protection board, and calculating the average curvature and Gaussian curvature of each point on the surface of the three-dimensional model of the protection board by adopting a principal curvature analysis method; determining main shape characteristics of the protection plate according to the curvature distribution histogram; matching the extracted shape features with a pre-established shape feature library, and measuring the similarity between different shapes by adopting Hausdorff distance based on side length; determining the shape category of the protection plate through minimum distance classification; according to the shape category, carrying out semantic segmentation on the three-dimensional model of the protection board, dividing the three-dimensional model of the protection board into different functional areas by using an area growth algorithm, and constructing a hierarchical structure of the functional areas based on an adjacent relation; matching the divided functional areas with corresponding areas in a standard model, and realizing local shape registration by adopting an ICP algorithm; calculating a coordinate transformation matrix between the functional areas to obtain the relative position and direction of the three-dimensional model of the protection board in the standard three-dimensional model; and according to the registration result, carrying out assembly pose optimization on the three-dimensional model of the protection plate, and finding out assembly pose parameters under the constraint of meeting assembly tolerance by solving the weighted least square problem of the position and the direction so as to realize the assembly of the protection plate and the whole machine.
Specifically, a multi-view image of a battery protection plate to be installed is acquired, internal and external parameters of the image are acquired through camera calibration, and then a three-dimensional point cloud model of the protection plate is recovered through feature point matching and triangulation principles by using an SFM algorithm. Preprocessing the obtained point cloud model, removing outliers by adopting a median filtering algorithm, setting the size of a filtering window to be 5 multiplied by 5, and solving the median of points in the neighborhood of each point to serve as new coordinates of the point. And converting the point cloud into a triangular mesh curved surface model by using a Poisson surface reconstruction algorithm to obtain a three-dimensional model of the protection plate. And extracting shape characteristics of the generated three-dimensional model of the protection plate, calculating the average curvature and Gaussian curvature of each point on the surface of the model by adopting a principal curvature analysis method, dividing the curvature value into a plurality of sections, and counting the points in each section to form a curvature distribution histogram. If the number of points of the plane area exceeds 80%, the protection plate is considered to be characterized by taking the plane as a main shape; if the number of points greater than 0.1 is more than 20% for the region of greater curvature, the protective plate is considered to have more rounded or chamfered features. Matching the extracted shape features with a pre-established shape feature library, measuring the similarity between different shapes by adopting Hausdorff distance based on point sets, wherein the Hausdorff distance is defined as H (A, B) =max (H (A, B), H (B, A)), wherein H (A, B) =max (min (d (a, B)) and d (a, B) represents Euclidean distance between point a and point B, and determining the shape category of the protection plate by minimum distance classification, such as rectangle, L shape, U shape and the like. Dividing the three-dimensional model of the protection plate into different functional areas according to shape characteristics and topological relations, dividing the model into a plane area and a curved surface area according to curvature distribution, dividing the plane area into functional areas such as mounting holes and positioning columns according to the adjacent relation and size between the areas, and dividing the curved surface area into functional areas such as connectors and transition areas. And carrying out semantic segmentation on the divided functional areas, dividing the model into different functional areas by using an area growth algorithm, and constructing a topological structure of the areas based on the adjacent relation. Matching the separated functional areas with corresponding areas in the standard model, realizing local shape registration by adopting an ICP algorithm, and calculating a coordinate transformation matrix between the areas to obtain the relative position and direction of the three-dimensional model of the protection board in the standard model. Performing assembly pose optimization on the three-dimensional model of the protection plate according to the registration result, and taking the minimization of the assembly error as an objective function, namely minimizing the distance and angle deviation between the three-dimensional model of the protection plate and the whole machine model; and taking the assembly tolerance as a constraint condition, namely that the required distance deviation is not more than 0.1mm, the angle deviation is not more than 1 degree, solving the least square optimization problem by using a Levenberg-Marquardt algorithm, obtaining the optimal assembly pose parameter, and realizing the accurate assembly of the protection plate and the whole machine. When the image of the protection board is obtained, 5 high-definition industrial cameras can be adopted to shoot from the front, the back, the left side, the right side and the top of the protection board respectively, the resolution of the cameras is set to 2000 ten thousand pixels, the internal reference matrix and the external reference matrix of each camera are calculated through a Zhang Zhengyou calibration method, and the calibration error is controlled within 0.05 pixel. And carrying out sparse reconstruction on the multi-view image by using an SFM algorithm, extracting 5000 SIFT feature points, estimating a base matrix among cameras by using a RANSAC algorithm, and optimizing three-dimensional coordinates of the feature points by using a beam adjustment method to obtain a sparse point cloud model containing 10 ten thousand points. And carrying out median filtering on the sparse point cloud for 5 times, wherein the filtering radius is 10mm, removing 5% of outliers, carrying out surface reconstruction on the point cloud by adopting ScreenedPoisson algorithm, and setting the depth of octtree to be8 to obtain a curved surface model consisting of 50 ten thousand triangular patches. And (3) extracting features of the curved surface model by using a principal curvature analysis method, calculating the average curvature and Gaussian curvature of each vertex, dividing the curvature value into three sections of 0-0.01, 0.01-0.05 and 0.05-0.1, and counting the proportion of the number of the vertices in each section to the total number of the vertices, wherein the vertex with the average curvature smaller than 0.01 is 90% and the vertex with the Gaussian curvature larger than 0.05 is 15%, so that the protection plate is judged to be mainly characterized by a plane and has a small quantity of rounded corner features. 10 typical protection board samples in a standard model library are selected, hausdorff distances are calculated between the 10 typical protection board samples and a three-dimensional model of the protection board to be installed respectively, a distance threshold is set to be 2mm, and the protection board to be installed is determined to belong to a rectangular category through minimum distance matching. Dividing the three-dimensional model of the protection plate into a plane area and a curved surface area according to curvature distribution, dividing the plane area into 4 mounting hole areas and 1 positioning column area by combining boundary characteristics and topological relations of the areas, dividing the curved surface area into 2 connector areas and 3 transition areas, and carrying out area combination by using an area growth algorithm to obtain a semantic division result of the functional area. Registering the separated functional area with the standard model by utilizing an ICP algorithm, setting the iteration times to be 100 times, setting the convergence threshold to be 0.01mm, and obtaining a transformation matrix [0.9987,0.0523, -0.0001, -0.0523,0.9986,0.0012, 0.0001-0.0012,1.0000 ] of the protection board to be installed relative to the standard model, wherein the registration error is 0.08mm. Finally, optimizing the assembly pose of the protection plate by adopting a Levenberg-Marquardt algorithm, taking the weighted sum of the registration error of the assembly hole and the registration error of the positioning column as an objective function, taking the minimum distance between the protection plate and the machine shell of the whole machine as a constraint condition, obtaining a transformation matrix [0.9991,0.0436, -0.0003, -0.0436,0.9990,0.0009 and 0.0003-0.0009,1.0000 of the optimal assembly pose by 20 iterations, reducing the registration error of the assembly hole from 0.12mm to 0.06mm, reducing the registration error of the positioning column from 0.15mm to 0.08mm, and meeting the requirement of the assembly tolerance.
Step S105, if the battery protection board to be installed is not matched with any preset shape category, associating the standard three-dimensional model with a corresponding preset installation strategy, training to obtain an installation strategy generation model, and generating a corresponding installation strategy according to the three-dimensional model of the battery protection board to be installed.
For the battery protection board to be installed, which is not matched with the preset shape category, carrying out geometric deformation on the three-dimensional model of the battery protection board to be installed and the standard three-dimensional model, adopting an FFD algorithm, and adjusting the shape of the three-dimensional model of the battery protection board to be installed by moving a control point grid so that the three-dimensional model of the battery protection board to be installed approximates to the most similar standard three-dimensional model; extracting geometric features and topological features of the three-dimensional model according to the adjusted three-dimensional model to form feature vectors; establishing a mapping relation between the feature vector of the standard three-dimensional model and the corresponding preset installation strategy, and training by adopting an MLP neural network to obtain an installation strategy generation model; inputting the feature vector of the battery protection board to be installed into the trained installation strategy generation model, obtaining the generated installation strategy through forward propagation and threshold judgment, and decoding the installation strategy into an installation link and parameters; carrying out feasibility analysis on the generated installation strategy, checking whether a problem exists in the installation process through virtual assembly simulation, and if the problem exists, adjusting the MLP neural network to generate the new installation strategy until the simulation verification passes; associating the new installation strategy with the three-dimensional model of the battery protection board to be installed, constructing a semantic model containing geometric information, topology information and assembly information by adopting a knowledge representation method based on an ontology, and storing the semantic model into a PLM system; in the actual assembly process, the three-dimensional model and the new installation strategy which are related are read from the PLM system, three-dimensional visual assembly guidance is provided through a human-computer interaction interface, assembly progress and quality parameters are tracked in real time, and if the assembly progress is unqualified or the quality parameters exceed a preset threshold, an alarm prompt is sent out, and corrective measures are given.
Specifically, for identifying a battery protection plate to be installed, which is not matched with a preset shape, geometrically deforming a three-dimensional model and a standard model of the battery protection plate, adopting an ICP algorithm, firstly roughly registering point clouds on the two models, and then iteratively executing the following steps of finding the nearest point on a target model for each point on a source model; estimating a transformation matrix to enable the point cloud on the source model to coincide with the corresponding nearest point on the target model as much as possible; applying the transformation matrix to the source model, updating its position; repeating the steps until the distance error between the two models is smaller than a given threshold value, so that the two models approach the most similar standard model. Extracting geometric features and topological features of the deformed protection board model according to the deformed protection board model, wherein the geometric features comprise length, width, height, surface area, volume, curvature, normal vector and the like, the topological features comprise Euler numbers, number of connected domains, number of holes, adjacent relation and the like, the length is taken as an example, the length is obtained by calculating the side length of a bounding box, the curvature is taken as an example, the curvature value of each vertex is calculated by a principal component analysis method, and the characteristic vector of the protection board model is formed. And establishing a mapping relation between the feature vectors of the standard model and the corresponding preset installation strategies, training by adopting a multi-layer perceptron neural network, wherein an input layer is the feature vectors of the standard model, a hidden layer is 3 layers, each layer comprises 100 nodes, an output layer directly outputs the vectorized representation of the installation strategies, a attention mechanism is added in the MLP network, and the weight of the hidden layer is adaptively adjusted according to the importance of the input feature vectors. The loss function is Mean Square Error (MSE), the optimization algorithm is Adam, the learning rate is 0.001, and the method is iterated 1000 times to obtain an installation strategy generation model. The feature vector of the battery protection board to be installed is input into a trained installation strategy generation model, the generated installation strategy vector is obtained through forward propagation, and then the vector is decoded into specific installation steps and parameters. And carrying out feasibility analysis on the generated installation strategy, taking the installation strategy as the input of simulation, simulating the stress condition of the protection plate and the whole machine in the assembly process through a physical engine, and checking whether the problems of interference, collision and the like exist. If the problem is found, modifying the loss function of the MLP network by adding a penalty term, so that the generated strategy is more in line with the physical constraint, and retraining the network until the simulation verification passes. Associating the generated installation strategy with a three-dimensional model of the battery protection board to be installed, defining concepts such as products, parts, features and installation processes and hierarchical and constraint relationships among the concepts by using WebOntologyLanguage by adopting a knowledge representation method based on an ontology, analyzing the three-dimensional model into geometric elements and topological elements, and mapping the geometric elements and the topological elements to corresponding concepts in the ontology; The installation strategy is decomposed into working procedures, working steps, technological parameters and the like, mapped to corresponding concepts in the ontology, and a complete semantic model is formed and stored in a product full life cycle management system. In the actual assembly process, an associated protection plate model and an installation strategy are read from a PLM system, three-dimensional visual assembly guidance is provided for operators through a human-computer interaction interface, a force sensor, a displacement sensor and the like are installed on an assembly station, force and displacement data in the assembly process are collected in real time, whether the assembly quality is qualified or not is judged through comparison with standard process parameters, once the quality parameters exceed a preset threshold value, the reasons of parameter deviation are analyzed through a statistical process control method, corresponding corrective measures are provided, such as adjustment of assembly force, optimization of fixture design and the like, and stable and controlled assembly quality is ensured. When the ICP algorithm is applied to geometrically match the three-dimensional model of the protection plate with the standard model, firstly, the two models are respectively subjected to downsampling to obtain uniformly distributed point clouds, and the density of the point clouds is set to 10 points per square millimeter. And then, establishing a local coordinate system by taking the gravity center of the standard model as an origin, and calculating the gravity center coordinate of the model to be matched as an initial position. In the iterative process, a kd-Tree algorithm is adopted to search the nearest points, and for each source point, k nearest points are searched in the target point cloud, wherein the value of k is 10. And then calculating an optimal rigid body transformation matrix by using singular value decomposition, applying the transformation matrix to the source point cloud, and repeating iteration until the mean square distance of the two models is smaller than 0.01mm or the iteration times reach 100 times. After matching, extracting geometric features of the protection board model, obtaining the length, width and height dimensions of the model through principal component analysis, obtaining the surface area of the model through triangular mesh surface integration, and obtaining the volume of the model through triangular mesh volume segmentation. And calculating the Euler number of the model by adopting a vertex sharing method, and obtaining the number of connected domains and the number of holes by using the connected domain marks. The extracted 12 features form feature vectors, the feature vectors are input into an MLP network for training, the network has 3 hidden layers, 100 nodes in each layer are used, and a ReLU activation function is used. The output layer directly outputs 10-dimensional installation strategy parameter vectors, including assembly sequence, positioning reference, installation direction, installation force, installation time and the like. By adopting a cross entropy loss function and an Adam optimizer, setting the initial learning rate to be 0.001, the momentum factor to be 0.9, the regularization coefficient to be 0.0001, training 1000 epochs, and enabling batchsize of each epoch to be 32. In the simulation verification stage, a PyBullet physical engine is used for building a geometric model and quality attributes of components such as a shell, a protection plate, a screw and the like, setting the friction coefficient to be 0.3, setting the elasticity coefficient to be 0.2 and setting the damping coefficient to be 0.1. According to the generated installation strategy, controlling the clamping jaw of the mechanical arm to assemble the protection plate in place at the speed of 0.05m/s, detecting the contact force between objects in the assembling process, judging that the assembling is failed if the contact force exceeds 10N, modifying a loss function through a penalty term, and retraining the strategy to generate a model. When the knowledge ontology is constructed, prot g software is used for defining the product, the part, the feature, the process and other types, and isPartOf, hasFeature, hasProcess and other relations are used for analyzing the protection board model into individuals such as geometric entities, topological entities and the like and mapping the individuals to the part types, and the installation strategy is analyzed into individuals such as assembly, positioning, connection and the like and mapped to the process types. And finally, storing the ontology into the PLM system in an OWL format, and associating the ontology with the product structure tree. During actual assembly, a HoloLens head-mounted display is used for presenting three-dimensional assembly guidance for operators, a Kistler six-dimensional force sensor is installed at an assembly station, assembly force data are collected at the frequency of 1kHz, cpk indexes of assembly quality are judged through comparison with standard force values in an installation strategy, when Cpk is lower than 1.33, a statistical process control chart is updated, drift trend of assembly parameters is analyzed, and long-term stability of the assembly quality is ensured through adjustment of motion track and clamping force compensation of a mechanical arm.
And when the installation strategy generation model is trained, the association relation between the standard shape data and the integral placement parameters is subjected to deep learning, so that the installation strategy generation model generates the placement action for avoiding edge collision for the battery protection plate according to the geometric shape and the contour characteristics.
Training the association relation between the standard shape data and the integral placement parameters by adopting a deep learning algorithm according to the geometric shape and the outline characteristics of the battery protection plate to obtain a trained installation strategy generation model; obtaining geometric shape data and contour feature data of a battery protection board to be installed, and obtaining optimal placement parameters for the battery protection board to be installed by inputting the geometric shape data and the contour feature data into the trained installation strategy generation model; according to the obtained optimal placement parameters, calculating a motion track and key pose points of the battery protection board to be installed in the installation process by adopting a motion planning algorithm, and determining a placement action sequence capable of avoiding edge collision; the pose and the motion state of the battery protection board are monitored in real time through machine vision, the pose and the motion state are compared with the calculated optimal placement parameters, and whether deviation and collision risk exist or not is judged; according to the real-time monitoring result, the motion of the mechanical arm is adjusted and compensated in real time by adopting servo control, so that the battery protection board to be installed is ensured to be placed in the target position, and collision and damage are avoided; and acquiring the mounting pose of the battery protection plate to be mounted, judging whether the mounting meets the requirements or not by comparing the mounting pose with a preset standard pose, and optimizing the trained mounting strategy generation model according to the mounting result.
Specifically, training the shape data and the corresponding placement parameters of 1000 groups of standard battery protection plates by a deep learning algorithm to obtain an installation strategy generation model with the accuracy reaching 95%. The measurement is carried out on a battery protection plate to be installed, the length of the battery protection plate is 55mm, the width of the battery protection plate is 38mm, the thickness of the battery protection plate is 2mm, and the optimal placement angle of the battery protection plate is 30 degrees and the optimal placement height of the battery protection plate is 10mm through inputting geometric shape data into a trained model. According to the obtained optimal imbedding parameters, calculating a mechanical arm motion track avoiding collision through a motion planning algorithm, wherein the mechanical arm motion track comprises 5 key pose points which are respectively a starting point, a lifting point, an avoiding point, a descending point and a target point. In the process of installing the battery protection board, the pose change of the battery protection board is monitored in real time at the speed of 30 frames per second through machine vision, and when the battery protection board is detected to deviate from a preset track by more than 2mm, the collision risk is judged and a warning is sent out. According to the pose deviation data monitored in real time, the mechanical arm is controlled by the servo motor to carry out track adjustment within 1 second, the deviation is within +/-5 mm, stable placement of the battery protection plate is ensured, and the maximum impact force is controlled below 3 newtons. And comparing the final mounting pose of the battery protection plate with a preset standard pose, wherein the position deviation is 8mm, the angle deviation is 1 DEG, and the final mounting pose is within an acceptable range, and judging that the final mounting pose is qualified and feeding the result data back to the model for parameter optimization.
And when the installation strategy generation model is trained, performing deep learning on the association relation between the hierarchical structure and the element wiring so that the installation strategy generation model identifies the position information of any element according to the protection plate image and generates a corresponding wiring path.
Constructing a training data set containing a hierarchical structure of battery protection boards and element wiring relations, marking protection board images in historical assembly cases, and marking space information and wiring information of each element, wherein the space information comprises position coordinates, orientation angles and size, and the wiring information comprises a connection mode, a connection sequence and a connection line type; preprocessing the protection board image by adopting an image segmentation algorithm, segmenting the image into different element areas by using a threshold segmentation method, an edge detection method and an area growth method, and extracting characteristic information of each element area, wherein the characteristic information comprises positions, sizes and shapes; sequencing the extracted element characteristic information according to an assembly sequence to form a characteristic sequence representing an assembly state, and pairing the characteristic sequence with a corresponding wiring path sequence to form a training sample; constructing a sequence-to-sequence learning model based on an encoder-decoder structure, encoding the characteristic sequence through a bidirectional LSTM network, and decoding the encoded characteristic vector through another LSTM network to generate a corresponding wiring path sequence; leading in an attention mechanism at each time step of the decoder, adaptively adjusting attention weight distribution output by the encoder according to the correlation between the current generated wiring path and the assembly state characteristics, and enabling the generated wiring path to be matched with the assembly state; when the installation strategy generation model is trained, a cross entropy loss function is adopted to measure the difference between the generated wiring path sequence and the real path, and regularization items are introduced to restrict the parameters of the installation strategy generation model; iterative updating is carried out on parameters of the installation strategy generation model by using an Adam optimization algorithm, and the trained installation strategy generation model is deployed into an assembly strategy planning system; extracting a characteristic sequence of the current assembly state according to the real-time acquired protection board image, inputting the characteristic sequence into the installation strategy generation model for prediction, and generating an optimal wiring path of the element; verifying the generated wiring path through virtual simulation, constructing a digital twin model of the protection board in a three-dimensional scene, detecting whether a problem exists or not according to the motion trail of the wiring path control element, and optimizing and adjusting the path with the problem.
Specifically, the image segmentation algorithm is adopted to preprocess the image of the protection board, and the characteristics of the color, texture, shape and the like of the element area are combined, and a proper segmentation algorithm is selected, such as segmentation based on a color threshold value, segmentation based on edge detection, segmentation based on area growth and the like, and a plurality of segmentation algorithms are combined to improve the segmentation precision. The extracted element characteristic information is arranged according to the assembly sequence, the assembly sequence can be defined in advance according to functions and connection relations of elements in a circuit by adopting a rule-based method, and the optimal sequence can be searched by adopting a heuristic search algorithm according to indexes such as assembly efficiency, stability and the like by adopting an optimization-based method. A characteristic sequence representing the assembly state is formed, and the characteristic sequence is paired with a corresponding wiring path sequence to form a complete training sample. Constructing a sequence-to-sequence learning model based on an encoder-decoder structure, encoding the assembly state feature sequence through a multi-layer bidirectional LSTM network, setting the number of hidden units of each layer according to the scale and the complexity of training data, and learning and extracting context information and global dependency relation of an assembly process; and decoding the coded feature vector through another LSTM network to generate a corresponding wiring path sequence. Attention mechanism is introduced at each time step of the decoder, the attention weight distribution of the encoder output is adaptively adjusted according to the correlation between the current generated wiring path and the assembly state characteristics, specifically, dot products are made on the current decoder hiding state and the encoder output sequence to obtain attention score vectors, the attention score vectors are normalized by a softmax function to obtain attention weight distribution, and finally, weighted summation is carried out on the encoder output sequence by using the attention weight to obtain attention context vectors, so that the generated wiring path can be matched with the assembly state. During model training, a cross entropy loss function is adopted to measure the difference between the generated wiring path sequence and the real path, and a regularization term is introduced to constrain model parameters so as to prevent overfitting. And (3) carrying out iterative updating on model parameters by using an Adam optimization algorithm, dynamically adjusting the learning rate, and accelerating the convergence rate. The trained installation strategy generation model is deployed into an assembly strategy planning system, the characteristic sequence of the current assembly state is extracted according to the real-time acquired protection board image, and is input into the model for prediction, and the optimal wiring path of the element is automatically generated. Verifying the generated wiring path by a virtual simulation technology, constructing a digital twin model of the protection board in a three-dimensional scene, detecting whether interference collision, line crossing and other problems exist according to the motion track of a wiring path control element, adjusting the problematic path by adopting an optimization method based on physical constraint, parameterizing the wiring path into a series of control points, solving the optimal control point position by using algorithms such as sequence quadratic programming, gradient descent and the like according to the conditions such as element position, wire length, bending radius, obstacle avoidance constraint and the like, and generating the wiring path with no collision, no crossing and shortest length, thereby ensuring the feasibility and reliability of the wiring scheme. Firstly, randomly extracting 500 images of battery protection boards from a historical assembly case, manually marking the positions, orientations, sizes and the like of elements in the images by a professional assembly engineer to form a mask label at a pixel level, and marking the connection relation among the elements according to a circuit diagram to form a wiring sequence and a linear text label. And then preprocessing an original image by using OpenCV, firstly dividing a main board area by using RGB color threshold values, then extracting the edge contour of the element by using a Canny operator, and dividing the contour into independent element areas by using connected domain analysis to obtain the element areas with the size of more than 100x100 pixels. And defining 100 assembly sequence rules, such as a first-mount power module, a later-mount connector and the like, according to the circuit topology structure and the assembly process requirements, and arranging the element areas according to the rule sequence by using a rule matching algorithm to form an assembly sequence. And extracting the characteristics of the assembled sequence by using a transducer encoder to extract 256-dimensional hidden state sequences. Finally, a transducer decoder is used for decoding the hidden state sequence into a wiring path sequence, and the decoder adopts 8-head attention, each head is 64-dimensional, and outputs the coordinate sequence and the line type of the wiring path. The loss function is the mean square error of the path coordinates and the linear cross entropy, 100 epochs are trained by an Adam optimizer, and the learning rate is 0.0005. in the reasoning stage, the faster-rcnn is used for detecting the positions and the types of elements in the newly acquired protection board image, the detected element sequence is extracted and input into a trained model, and a wiring path is decoded. And mapping the wiring path coordinates onto a three-dimensional CAD model of the protection board, planning a motion track of the end effector by using a Moveit planner, constructing an assembly scene in Gazebo, simulating the planned track, and checking whether collision occurs with other elements. If collision occurs, defining a collision penalty term, and performing fine adjustment on the path by using a gradient-based trajectory optimization algorithm to obtain a collision-free path avoiding all elements. And finally outputting the verified reliable wiring scheme to guide the robot to complete the actual wiring assembly task.
Step S106, before the battery protection board to be installed is installed through the mechanical arm, the corresponding mechanical arm movement track is configured for the installation strategy corresponding to the battery protection board to be installed by combining the installation environment information of the battery protection board to be installed, so that the mechanical arm can grasp the battery protection board to be installed according to the mechanical arm movement track, and the installation operation is completed.
Acquiring three-dimensional model data of the battery protection plate to be installed, wherein the three-dimensional model data comprises geometric dimensions, shape characteristics and material properties of the battery protection plate to be installed; converting the three-dimensional model data into a target representation format which can be identified by the mechanical arm by adopting a three-dimensional reconstruction algorithm, wherein the target representation format comprises point clouds, grids or primitives; collecting installation environment information, wherein the installation environment information comprises fusion data of a visual sensor and an ultrasonic sensor of a radar sensor; constructing a three-dimensional scene map of the installation environment according to the installation environment information, wherein the three-dimensional scene map marks the installation position, the obstacle and the personnel information; inputting the target representation format and the three-dimensional scene map into a mechanical arm motion planning algorithm, wherein the mechanical arm motion planning algorithm comprises an RRT algorithm and a PRM algorithm, and generating a mechanical arm motion track from a starting position to a target position, and the mechanical arm motion track combines kinematics and dynamics constraint to optimize smoothness, time consumption and energy consumption of the mechanical arm motion track; performing collision detection and accessibility analysis on the motion trail of the mechanical arm, wherein the collision detection adopts a geometric algorithm or a physical algorithm, including an OBB algorithm and an FCL algorithm, and judging whether the motion trail collides with an obstacle of the installation environment; calculating the reachable configuration of the joint space of the mechanical arm through an inverse kinematics solver, and judging whether the motion track of the mechanical arm meets the joint limit of the mechanical arm or not; if collision or unreachable is detected, inserting an intermediate node at a key point of the motion track of the mechanical arm, regenerating a sub-track by adopting a local planning algorithm, and performing iterative optimization until a collision-free reachable track is obtained; converting the optimized motion track of the mechanical arm into a joint space control instruction of the mechanical arm, and mapping pose points of a Cartesian space into the rotation angle and the speed of a joint motor through a kinematic positive solution and interpolation algorithm; a track tracking control algorithm is adopted, wherein the track tracking control algorithm comprises a PID algorithm and a self-adaptive algorithm, and the mechanical arm is controlled to accurately execute the motion track; the tail end position of the motion track is provided with a grabbing strategy of the mechanical arm, wherein the grabbing strategy comprises grabbing force and grabbing height, and a tail end clamp of the mechanical arm is controlled to grab the battery protection plate to be installed in a preset gesture and with preset force; flexible gripping and assembly is achieved by a force control algorithm, which is an impedance control algorithm.
Specifically, three-dimensional model data of a rectangular battery protection plate with the length of 50mm, the width of 30mm and the thickness of 3mm are obtained, and the material property is ABS plastic. And converting the three-dimensional model into point cloud data containing 10000 points by using a poisson reconstruction algorithm. Mounting environment information of 5m×5m×3m was collected by 2 RGB cameras, 1 lidar, and 3 ultrasonic sensors. 1 installation position, 3 barriers and 2 persons are marked in the generated three-dimensional scene map. And generating a mechanical arm movement track from the initial position (0, 0) to the target position (5,8,2) by adopting an RRT algorithm, wherein the total track length is 3 meters, the optimized average speed is 5 meters/second, and the energy consumption is 15 joules. The trajectory is collision detected using the FCL algorithm, and a box is found to collide at (7,4,9). The rotation angles of the joints 2 and 4 at the positions are respectively 45 degrees and 60 degrees calculated by an inverse kinematics solver, and the joint limits are exceeded. An intermediate node (6,5,0) is inserted at the position (7,4,9), two sections of sub-tracks are regenerated by adopting a Quintic polynomial interpolation method, and a collision-free and reachable track is finally obtained through 5 times of iterative optimization. The optimized track is discretized into 100 pose points in Cartesian space, and then the pose points are converted into corner sequences of 6 joint motors through kinematic positive solution, wherein the angular resolution is 1 degree, and the time interval is 05 seconds. Track tracking control is carried out on the 6 joint motors by adopting a PID algorithm, wherein the values of three parameters are respectively that a proportional coefficient Kp=10, an integral coefficient Ki=5 and a differential coefficient Kd=1; the control error converges to within 1 mm. The grabbing force of the clamp at the tail end of the mechanical arm is 10 newtons, and the grabbing height is 2 cm. And adopting an impedance control algorithm based on force/position hybrid control, wherein the softening coefficient takes the value of Kt=500N/m in the translation direction and Kr=200 N.m/radian in the rotation direction. The accurate assembly of the battery protection plate and the mounting hole is realized, and the assembly error is less than 5 mm.
Step S107, monitoring the battery protection board through a multi-angle visual detection system, and analyzing the surface quality and the element integrity of the battery protection board.
Imaging the battery protection plate from different angles by adopting a multi-angle visual detection system to acquire multi-angle image data of the battery protection plate; splicing the multi-view image data into a 360-degree panorama by adopting an image splicing algorithm; performing image enhancement processing on the 360-degree panorama, adjusting the brightness and contrast of the 360-degree panorama by adopting a histogram equalization algorithm, and removing the noise of the 360-degree panorama by adopting a median filtering algorithm; positioning and extracting a battery protection board in the 360-degree panorama by adopting a target detection algorithm based on deep learning, wherein the target detection algorithm comprises a YOLO algorithm and an SSD algorithm, and a pixel level mask of the battery protection board is obtained; comparing the extracted battery protection board mask map with a pre-established defect template library, and calculating the similarity between the mask map and each defect template by adopting a similarity measurement algorithm, wherein the similarity measurement algorithm comprises a correlation coefficient matching algorithm and a Hamming distance algorithm; judging whether the surface of the battery protection board has defects or not according to a similarity threshold value; extracting characteristic parameters of the detected defect region by adopting a segmentation algorithm based on region growth; comparing the characteristic parameters with a preset waste judging standard, judging the severity of the defect, and obtaining a surface quality evaluation result of the battery protection plate; performing element identification on the battery protection board image by adopting an image classification algorithm based on a convolutional neural network, wherein the image classification algorithm comprises ResNet algorithm and Inception algorithm to obtain position coordinates of the element, and the element comprises a resistor, a capacitor and a chip; matching the identified element positions with a circuit schematic diagram, calculating the spatial distance and the topological relation between the elements by adopting a graphic algorithm, and constructing an element connection diagram; traversing the element connection graph, and checking whether element deletion, dislocation and short circuit exist; and judging the element integrity and reliability of the battery protection board by combining the conduction test result.
Specifically, a plurality of high-resolution industrial cameras are adopted to image a battery protection board from different angles, image data of a plurality of view angles such as the front face, the back face and the side face of the battery protection board are obtained, a SIFT algorithm based on feature point matching or a Lucas-Kanade algorithm based on image registration is adopted to splice the images of the plurality of view angles, firstly, scale-invariant feature transform (SIFT) key points of the images are extracted, then, the transformation relation among the images is determined through the key point matching, and then, geometric correction and fusion are carried out on the images according to the transformation relation, so that a spliced panoramic image is obtained. And carrying out image enhancement processing on the spliced panoramic image, adopting a histogram equalization algorithm to adjust the brightness and contrast of the image, adopting a median filtering algorithm to remove image noise, and improving the image quality. And (3) accurately positioning and extracting the battery protection board in the enhanced image by adopting a target detection algorithm based on deep learning, such as YOLO, SSD and the like, so as to obtain a pixel level mask of the protection board. Comparing the extracted protection board mask pattern with a pre-established defect template library, wherein the defect template library has the establishment flow of firstly collecting a large number of qualified product and defect product image samples, and manually marking to obtain the type, position, shape and other information of the defects; preprocessing the marked image, such as size normalization, contrast enhancement and the like; then adopting data enhancement technology, such as rotation, translation, scaling and the like, to expand the number of samples; And finally, clustering and grouping the defect samples by using a clustering algorithm such as K-means, GMM and the like to obtain representative templates of different types of defects, thereby forming a complete defect template library. And calculating the similarity between the mask map and each defect template by adopting a multi-scale pyramid matching or sliding window matching mode, and judging whether the surface of the protection board has defects such as scratches, stains, deformation and the like according to a similarity threshold value. And for the detected defect region, further extracting characteristic parameters such as the shape, the size and the position of the defect by adopting a region growth-based segmentation algorithm, comparing the characteristic parameters with a preset waste judgment standard, and automatically judging the severity of the defect to obtain a surface quality evaluation result of the protection plate. And (3) performing element identification on the qualified protection board image by adopting an image classification algorithm based on a convolutional neural network, such as ResNet, inception and the like, so as to obtain the position coordinates of key elements such as resistors, capacitors, chips and the like. Matching the identified element position information with a vectorization circuit schematic diagram extracted from a circuit design file or a schematic diagram, and mapping the element position into the schematic diagram by adopting a diagram matching algorithm, such as a maximum public sub-diagram isomorphism algorithm, a bipartite diagram matching algorithm and the like, and considering factors such as the direction, the size, the spacing and the like of elements to construct an element connection diagram. Before constructing the element connection diagram, the detected surface defects are spatially related to the identified element positions, whether each defect is overlapped with or close to the element is judged, the type, the size, the severity and the like of the defects overlapped with or close to the element are analyzed, whether the identification and the function of the element are affected is judged, and the element identification result is corrected or filtered according to the influence degree of the defects. After the corrected element connection diagram is obtained, traversing all nodes and edges in the diagram, and checking whether abnormal conditions such as element deletion, dislocation, short circuit and the like exist. Meanwhile, a Flyingprobe test technology is adopted to conduct conduction test between elements, two or more probes are simultaneously contacted with test points on a PCB, certain voltage or current is applied, conductivity and insulativity between the test points are detected, whether faults such as open circuit or short circuit exist in a circuit or not is judged, and the integrity and reliability of the whole circuit of the protection board are comprehensively judged by combining visual detection and conduction test results. In multi-view imaging of a battery protection plate, 5 industrial cameras with 2000 ten thousand pixels of resolution are used for shooting front images of the protection plate from five angles of 0 degree, 45 degrees, 90 degrees, 135 degrees and 180 degrees respectively, 10 images are shot at each view angle, then feature points of each image are extracted by using a SIFT algorithm, mismatching points are removed by using a RANSAC algorithm, feature points of different view angles are unified to the same coordinate system by using affine transformation, the feature points are clustered by using a DBSCAN clustering algorithm, convex hulls of each cluster are used as initial areas, adjacent areas are combined by using an area growth algorithm, and finally the spliced panoramic image is obtained. when the panoramic image is detected, a pre-trained YOLOv model is used for carrying out target detection, mAP (mean time between arrival) of the model on a COCO data set reaches 0.85, a defect region with the size larger than 5mm can be detected, the detected suspected defect region is subjected to feature matching with 500 templates in a standard defect template library, cosine similarity is used as a matching measure, a threshold value is set to be 0.8, a region with the matching degree higher than the threshold value is subjected to morphological filtering, 5 characteristic parameters such as length, width, area, circularity and the like of the defect are extracted, and an SVM classifier is used for carrying out five classification on the defect types such as scratches, stains, deformation, Breaking and foaming, and the classification accuracy reaches more than 95%. For a detected defect, if the area is greater than 20 square millimeters or the length is greater than 10 millimeters, then the defect is determined to be a serious defect and needs to be rejected, and if IoU of the defective area and the element area is greater than 0.3, then the defect is marked as a defect affecting the element. When the element identification is carried out, the EFFICIENTNET-B3 model is used for carrying out feature extraction on the protection board image, then the FPN structure is used for generating a multi-scale feature map, and then the FasterR-CNN detector is used for detecting 20 common SMT elements such as resistors, capacitors, ICs and the like, wherein the AP of the element detection reaches more than 98%. And matching the detected element position with a PCB element position file in a CSV format derived from AltiumDesigner, and solving the maximum matching of the two graphs by using a Hungary algorithm to obtain the corresponding position of the element in the PCB layout graph. And constructing a directed graph among the elements according to a network connection table in the PCB layout, solving the minimum connection length between any two elements by using a Dijkstra shortest path algorithm, comparing the minimum connection length with a standard value in a design specification, and judging that the wiring is abnormal if the connection length exceeds 1.2 times of the standard value. Finally, conducting test is carried out on pins of the element by using DF12-40 double-head electric detection of Kennel, 5V voltage is applied, resistance values among the pins are measured, if the resistance values are smaller than 0.1 ohm, the conduction is judged, if the resistance values are larger than 10 megaohms, the disconnection is judged, and if the resistance values are between the resistance values and the resistance values, the suspicious connection is judged, and manual rechecking is needed. Comparing the conduction test result with the element identification result, and if a non-conduction device or a device conducting in error appears, judging that the element is absent or misplaced, and eliminating the element is needed. Through the series of detection and analysis, the comprehensive quality evaluation result of the battery protection board is finally obtained, wherein the comprehensive quality evaluation result comprises the types and the quantity of surface defects, the missing and misplacement conditions of elements, the abnormal wiring condition and the like, and data support is provided for subsequent quality improvement and optimization.
Step S108, if the collision risk exists in the installation process of the battery protection plate based on the movement track installation of the mechanical arm, a collision early warning signal is generated.
Monitoring the movement track of the mechanical arm and acquiring the movement track of the mechanical arm in the installation process of the battery protection plate; acquiring real-time position information of an obstacle in an installation environment, detecting the obstacle in real time by adopting vision and a laser radar, and acquiring three-dimensional point cloud data of the obstacle; inputting the motion track of the mechanical arm and the three-dimensional point cloud data into a collision detection algorithm, wherein the collision detection algorithm comprises an algorithm based on a distance field and an algorithm based on a hierarchical bounding box; calculating the minimum distance between the mechanical arm and the obstacle, and judging whether the minimum distance is smaller than a safety threshold value or not; if the minimum distance is smaller than the safety threshold, the collision risk is considered to exist, and a collision early warning signal is generated; the collision early warning signal is sent to a control system, and the control system pauses the movement of the mechanical arm or plans a new obstacle avoidance track; and the collision early warning signal is presented to a human-computer interaction interface in the form of voice, light or words, so that operators are reminded of paying attention to safety, and countermeasures are recommended to be taken.
Specifically, real-time motion parameters of the mechanical arm, such as the angle of the joint 1 being 30 °, the speed of the joint 2 being 10 ° per second and the acceleration of the joint 3 being 5 ° per second, are acquired. And calculating the real-time pose and track according to the mechanical arm kinematics model, wherein the tail end position of the mechanical arm is (5 m,2m and 8 m) and the pose is (0 degree, 45 degrees and 0 degree) when t=1s. The real-time position of the obstacle in the environment is acquired, and the three-dimensional point cloud data of the obstacle A are displayed at (2 m,3m and 6 m) with the size of 4m multiplied by 5m multiplied by 1m. Inputting the track of the mechanical arm and the obstacle point cloud into a collision detection algorithm, and calculating the minimum distance between the mechanical arm and the obstacle A to be 1m based on the algorithm of the distance field. Judging whether the minimum distance is smaller than a safety threshold value, wherein if the safety threshold value is set to be 2m, 1m is smaller than the safety threshold value, and collision risk exists. And generating a collision early warning signal, wherein the early warning signal is used for warning that the mechanical arm possibly collides with the obstacle A, the minimum distance is 1m, and the motion is suggested to be paused and the track is planned again. And sending the early warning signal to a control system to enable the control system to pause the movement of the mechanical arm or re-plan the track, and simultaneously displaying a human-computer interaction interface in the forms of voice, light, characters and the like to remind operators of paying attention to safety.
It will be evident to those skilled in the art that the application is not limited to the details of the foregoing illustrative embodiments, and that the present application may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Claims (9)
1. A method for automatically identifying and positioning a battery protection plate feeding device, comprising the following steps:
Collecting standard images of battery protection boards of all preset shape categories, processing the standard images, and extracting standard shape data comprising geometric shapes and contour features;
generating a standard three-dimensional model according to the standard shape data and the preset shape category associated with the standard image, and carrying out hierarchical structure identification on the standard three-dimensional model;
acquiring a preset installation strategy corresponding to the standard three-dimensional model, wherein the preset installation strategy comprises integral embedded parameters and element wiring parameters, and associating the preset installation strategy with a preset shape class and a hierarchical structure;
Acquiring a protective plate image of a battery protective plate to be installed, generating a corresponding protective plate three-dimensional model, and matching the corresponding shape type and the hierarchical structure;
If the battery protection board to be installed is not matched with any preset shape category, associating the standard three-dimensional model with a corresponding preset installation strategy, training to obtain an installation strategy generation model, and generating a corresponding installation strategy according to the three-dimensional model of the battery protection board to be installed;
Before the battery protection board to be installed is installed through the mechanical arm, the corresponding mechanical arm movement track is configured for the installation strategy corresponding to the battery protection board to be installed by combining the installation environment information of the battery protection board to be installed, so that the mechanical arm can grasp the battery protection board to be installed according to the mechanical arm movement track, and the installation operation is completed;
Monitoring the battery protection board through a multi-angle visual detection system, and analyzing the surface quality and the element integrity of the battery protection board;
If the collision risk exists in the installation process of the battery protection plate based on the mechanical arm movement track installation, generating a collision early warning signal.
2. The method of claim 1, wherein the capturing standard images of the battery protection plates of each preset shape category, processing the standard images, extracting standard shape data, including geometric shape and contour features, comprises:
collecting standard images of the battery protection plates of all types according to the battery protection plates of the preset shape types;
Preprocessing the standard image through image processing, wherein the preprocessing comprises noise reduction and enhancement operations;
And extracting standard shape data from the preprocessed standard image, and acquiring the geometric shape and contour characteristics of the battery protection board in the standard image through an edge detection and contour extraction algorithm.
3. The method of claim 1, wherein the generating a standard three-dimensional model according to the standard shape data and the preset shape class associated with the standard image, and performing hierarchical structure identification on the standard three-dimensional model, comprises:
According to the standard shape data extracted from the standard image, combining preset shape categories, and generating a standard three-dimensional model of each preset shape category through a three-dimensional reconstruction algorithm;
Performing grid simplification and smoothing on the generated standard three-dimensional model;
Dividing the standard three-dimensional model into different parts and layers according to the geometric structure and topological relation of the standard three-dimensional model by utilizing a three-dimensional model segmentation algorithm to form a hierarchical structure tree;
According to the hierarchical structure tree of the standard three-dimensional model, semantic labeling is carried out on each component, semantic attributes and category identifiers are given to the components, and a mapping relation between semantic information and geometric information of the standard three-dimensional model is established;
Carrying out attribute analysis on the standard three-dimensional model after semantic annotation, extracting geometric attributes and topological attributes of each component, and constructing attribute feature vectors of the standard three-dimensional model;
Clustering and grouping the components of the standard three-dimensional model according to the attribute feature vector by adopting a clustering algorithm to obtain the similarity and the relevance among different components, and forming a new-level semantic structure;
And comparing the clustering grouping result with the preset shape category, and matching the parts of the standard three-dimensional model with the standard category through similarity calculation and threshold judgment to realize classification and identification of the standard three-dimensional model.
4. The method of claim 1, wherein the obtaining the preset installation policy corresponding to the standard three-dimensional model, including the overall placement parameters and the component connection parameters, associates the preset installation policy with the preset shape class and the hierarchy, includes:
Acquiring a preset installation strategy corresponding to the standard three-dimensional model from a pre-constructed knowledge base according to the shape category and the hierarchical structure of the standard three-dimensional model, wherein the preset installation strategy comprises integral placement parameters and element wiring parameters, the integral placement parameters comprise installation positions, installation directions and installation sequences, and the element wiring parameters comprise wiring modes, line sequences and line types of all elements;
carrying out semantic analysis and structural representation on the preset installation strategy to obtain an installation instruction sequence;
mapping and associating the extracted installation instruction sequence with the hierarchical structure of the three-dimensional model, and generating an installation position and an installation parameter of each component according to the geometric attribute and the topological relation of each component of the model;
deducing an installation flow and a dependency relationship according to the logic relationship and the sequence of each link in the installation instruction sequence;
simulation verification is carried out on the installation flow obtained by reasoning, feasibility and correctness of the installation process are verified, and the part with problems is optimized and adjusted;
Integrating the verified preset installation strategy with the three-dimensional model, generating a digital virtual prototype containing installation information, and storing the digital virtual prototype into a product data management system to be associated with product data outside the digital virtual prototype.
5. The method of claim 1, wherein the acquiring the protective plate image of the battery protective plate to be installed, generating the corresponding three-dimensional model of the protective plate, matching the corresponding shape class and the hierarchical structure, comprises:
collecting multi-view images of the battery protection plate to be installed;
Obtaining the internal and external parameters of the image through camera calibration;
Adopting an SFM algorithm, and obtaining a three-dimensional point cloud model of the protection plate through a characteristic point matching and triangulation principle;
preprocessing the three-dimensional point cloud model, and removing outliers by adopting a statistical filtering algorithm;
converting the three-dimensional point cloud model into a triangular mesh curved surface model by using a Poisson surface reconstruction algorithm to obtain a three-dimensional model of the protection board;
Extracting shape characteristics of the three-dimensional model of the protection board, and calculating the average curvature and Gaussian curvature of each point on the surface of the three-dimensional model of the protection board by adopting a principal curvature analysis method;
determining main shape characteristics of the protection plate according to the curvature distribution histogram;
matching the extracted shape features with a pre-established shape feature library, and measuring the similarity between different shapes by adopting Hausdorff distance based on side length;
determining the shape category of the protection plate through minimum distance classification;
According to the shape category, carrying out semantic segmentation on the three-dimensional model of the protection board, dividing the three-dimensional model of the protection board into different functional areas by using an area growth algorithm, and constructing a hierarchical structure of the functional areas based on an adjacent relation;
matching the divided functional areas with corresponding areas in a standard model, and realizing local shape registration by adopting an ICP algorithm;
Calculating a coordinate transformation matrix between the functional areas to obtain the relative position and direction of the three-dimensional model of the protection board in the standard three-dimensional model;
And according to the registration result, carrying out assembly pose optimization on the three-dimensional model of the protection plate, and finding out assembly pose parameters under the constraint of meeting assembly tolerance by solving the weighted least square problem of the position and the direction so as to realize the assembly of the protection plate and the whole machine.
6. The method according to claim 1, wherein if the battery protection board to be installed is identified not to match any preset shape category, associating the standard three-dimensional model with a corresponding preset installation strategy, training to obtain an installation strategy generation model, so as to generate a corresponding installation strategy according to the three-dimensional model of the battery protection board to be installed, including:
For the battery protection board to be installed, which is not matched with the preset shape category, carrying out geometric deformation on the three-dimensional model of the battery protection board to be installed and the standard three-dimensional model, adopting an FFD algorithm, and adjusting the shape of the three-dimensional model of the battery protection board to be installed by moving a control point grid so that the three-dimensional model of the battery protection board to be installed approximates to the most similar standard three-dimensional model;
extracting geometric features and topological features of the three-dimensional model according to the adjusted three-dimensional model to form feature vectors;
establishing a mapping relation between the feature vector of the standard three-dimensional model and the corresponding preset installation strategy, and training by adopting an MLP neural network to obtain an installation strategy generation model;
Inputting the feature vector of the battery protection board to be installed into the trained installation strategy generation model, obtaining the generated installation strategy through forward propagation and threshold judgment, and decoding the installation strategy into an installation link and parameters;
Carrying out feasibility analysis on the generated installation strategy, checking whether a problem exists in the installation process through virtual assembly simulation, and if the problem exists, adjusting the MLP neural network to generate a new installation strategy until simulation verification passes;
And associating the new installation strategy with the three-dimensional model of the battery protection board to be installed, constructing a semantic model containing geometric information, topology information and assembly information, and storing the semantic model into a PLM system.
7. The method of claim 1, wherein the configuring the corresponding movement track of the mechanical arm for the installation strategy corresponding to the battery protection board to be installed in combination with the installation environment information of the battery protection board to be installed before the battery protection board to be installed is installed by the mechanical arm, so that the mechanical arm grabs the battery protection board to be installed with the movement track of the mechanical arm, and completing the installation operation includes:
acquiring three-dimensional model data of the battery protection plate to be installed, wherein the three-dimensional model data comprises geometric dimensions, shape characteristics and material properties of the battery protection plate to be installed;
converting the three-dimensional model data into a target representation format which can be identified by the mechanical arm by adopting a three-dimensional reconstruction algorithm;
collecting installation environment information, wherein the installation environment information comprises fusion data of a visual sensor and a radar sensor ultrasonic sensor;
Constructing a three-dimensional scene map of the installation environment according to the installation environment information;
inputting the target representation format and the three-dimensional scene map into a mechanical arm motion planning algorithm to generate a mechanical arm motion track from a starting position to a target position;
performing collision detection and accessibility analysis on the motion trail of the mechanical arm, and judging whether the motion trail collides with an obstacle of the installation environment or not;
Calculating the reachable configuration of the joint space of the mechanical arm through an inverse kinematics solver, and judging whether the motion track of the mechanical arm meets the joint limit of the mechanical arm or not;
If collision or unreachable is detected, inserting an intermediate node at a key point of the motion track of the mechanical arm, regenerating a sub-track by adopting a local planning algorithm, and performing iterative optimization until a collision-free and reachable track is obtained;
Converting the optimized motion track of the mechanical arm into a joint space control instruction of the mechanical arm, and mapping pose points of a Cartesian space into the rotation angle and the speed of a joint motor through a kinematic positive solution and interpolation algorithm;
a track tracking control algorithm is adopted, wherein the track tracking control algorithm comprises a PID algorithm and a self-adaptive algorithm, and the mechanical arm is controlled to accurately execute the motion track;
The tail end position of the motion track is provided with a grabbing strategy of the mechanical arm, wherein the grabbing strategy comprises grabbing force and grabbing height, and a tail end clamp of the mechanical arm is controlled to grab the battery protection plate to be installed in a preset gesture and with preset force;
Flexible gripping and assembly is achieved by a force control algorithm, which is an impedance control algorithm.
8. The method of claim 1, wherein the monitoring of the battery protection plate by the multi-angle visual inspection system, analyzing the surface quality and the element integrity of the battery protection plate, comprises:
imaging the battery protection plate from different angles by adopting a multi-angle visual detection system to acquire multi-angle image data of the battery protection plate;
Splicing the multi-view image data into a 360-degree panorama by adopting an image splicing algorithm;
performing image enhancement processing on the 360-degree panorama;
Positioning and extracting a battery protection board in the 360-degree panorama by adopting a target detection algorithm to obtain a pixel level mask of the battery protection board;
comparing the extracted battery protection board mask map with a pre-established defect template library, and calculating the similarity between the mask map and each defect template by adopting a similarity measurement algorithm;
judging whether the surface of the battery protection board has defects or not according to a similarity threshold value;
Extracting characteristic parameters of the defects from the detected defect areas by adopting a segmentation algorithm;
Comparing the characteristic parameters with a preset waste judging standard, judging the severity of the defect, and obtaining a surface quality evaluation result of the battery protection plate;
performing element identification on the battery protection board image by adopting an image classification algorithm to obtain the position coordinates of the element, wherein the element comprises a resistor, a capacitor and a chip;
matching the identified element positions with a circuit schematic diagram, calculating the spatial distance and the topological relation between the elements by adopting a graphic algorithm, and constructing an element connection diagram;
Traversing the element connection graph, and checking whether element deletion, dislocation and short circuit exist;
And judging the element integrity and reliability of the battery protection board by combining the conduction test result.
9. The method of claim 1, wherein generating a collision early warning signal if it is detected that there is a collision risk in the installation process of the battery protection board based on the installation of the movement track of the mechanical arm, comprises:
monitoring the movement track of the mechanical arm and acquiring the movement track of the mechanical arm in the installation process of the battery protection plate;
acquiring real-time position information of an obstacle in an installation environment, detecting the obstacle in real time by adopting vision and a laser radar, and acquiring three-dimensional point cloud data of the obstacle;
inputting the motion track of the mechanical arm and the three-dimensional point cloud data into a collision detection algorithm, wherein the collision detection algorithm comprises an algorithm based on a distance field and an algorithm based on a hierarchical bounding box;
calculating the minimum distance between the mechanical arm and the obstacle, and judging whether the minimum distance is smaller than a safety threshold value or not;
if the minimum distance is smaller than the safety threshold, the collision risk is considered to exist, and a collision early warning signal is generated;
The collision early warning signal is sent to a control system, and the control system pauses the movement of the mechanical arm or plans a new obstacle avoidance track;
And the collision early warning signal is presented to a human-computer interaction interface in the form of voice, lamplight or words, so that operators are reminded of paying attention to safety, and countermeasures are recommended to be taken.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410750331.0A CN118334018B (en) | 2024-06-12 | 2024-06-12 | Method for automatically identifying and positioning battery protection plate feeding device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410750331.0A CN118334018B (en) | 2024-06-12 | 2024-06-12 | Method for automatically identifying and positioning battery protection plate feeding device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118334018A true CN118334018A (en) | 2024-07-12 |
| CN118334018B CN118334018B (en) | 2024-08-16 |
Family
ID=91780463
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410750331.0A Active CN118334018B (en) | 2024-06-12 | 2024-06-12 | Method for automatically identifying and positioning battery protection plate feeding device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118334018B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118455829A (en) * | 2024-07-15 | 2024-08-09 | 江苏时代新能源科技有限公司 | Battery welding quality detection method and device and electronic equipment |
| CN118543990A (en) * | 2024-07-23 | 2024-08-27 | 广东众森实业发展有限公司 | Automatic track planning method for blanking of battery protection plate |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103210502A (en) * | 2010-11-09 | 2013-07-17 | 富士胶片株式会社 | Solar cell protective sheet and method for manufacturing same, solar cell backing sheet, and solar cell module |
| CN107215658A (en) * | 2017-05-26 | 2017-09-29 | 惠州市三协精密有限公司 | A kind of battery core protection board feed mechanism |
| JP2018181047A (en) * | 2017-04-17 | 2018-11-15 | 凸版印刷株式会社 | Three-dimensional shape model generating device, three-dimensional shape model generating method and program |
| CN209777624U (en) * | 2019-04-02 | 2019-12-13 | 众联智能设备(深圳)有限公司 | Lithium battery protection circuit board feed mechanism |
| WO2023005392A1 (en) * | 2021-07-30 | 2023-02-02 | 东莞新能德科技有限公司 | Battery protection plate and manufacturing method therefor, battery, and electronic device |
| WO2023155069A1 (en) * | 2022-02-16 | 2023-08-24 | 佛山市南海区广工大数控装备协同创新研究院 | Deep-learning-based surface defect detection method for mobile phone battery |
| CN118155107A (en) * | 2024-04-23 | 2024-06-07 | 中国矿业大学 | Mechanical arm track planning and collision detection method based on augmented reality |
-
2024
- 2024-06-12 CN CN202410750331.0A patent/CN118334018B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103210502A (en) * | 2010-11-09 | 2013-07-17 | 富士胶片株式会社 | Solar cell protective sheet and method for manufacturing same, solar cell backing sheet, and solar cell module |
| JP2018181047A (en) * | 2017-04-17 | 2018-11-15 | 凸版印刷株式会社 | Three-dimensional shape model generating device, three-dimensional shape model generating method and program |
| CN107215658A (en) * | 2017-05-26 | 2017-09-29 | 惠州市三协精密有限公司 | A kind of battery core protection board feed mechanism |
| CN209777624U (en) * | 2019-04-02 | 2019-12-13 | 众联智能设备(深圳)有限公司 | Lithium battery protection circuit board feed mechanism |
| WO2023005392A1 (en) * | 2021-07-30 | 2023-02-02 | 东莞新能德科技有限公司 | Battery protection plate and manufacturing method therefor, battery, and electronic device |
| WO2023155069A1 (en) * | 2022-02-16 | 2023-08-24 | 佛山市南海区广工大数控装备协同创新研究院 | Deep-learning-based surface defect detection method for mobile phone battery |
| CN118155107A (en) * | 2024-04-23 | 2024-06-07 | 中国矿业大学 | Mechanical arm track planning and collision detection method based on augmented reality |
Non-Patent Citations (1)
| Title |
|---|
| 贾璐;于湧强: "双面贴装电池保护板生产中的工艺质量控制", 天津市电视技术研究会2013年年会论文集, 1 July 2013 (2013-07-01) * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118455829A (en) * | 2024-07-15 | 2024-08-09 | 江苏时代新能源科技有限公司 | Battery welding quality detection method and device and electronic equipment |
| CN118543990A (en) * | 2024-07-23 | 2024-08-27 | 广东众森实业发展有限公司 | Automatic track planning method for blanking of battery protection plate |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118334018B (en) | 2024-08-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112476434B (en) | Visual 3D pick-and-place method and system based on cooperative robot | |
| CN118334018B (en) | Method for automatically identifying and positioning battery protection plate feeding device | |
| Villalonga et al. | Tactile object pose estimation from the first touch with geometric contact rendering | |
| CN111563446B (en) | Human-machine interaction safety early warning and control method based on digital twin | |
| CN108764048B (en) | Face key point detection method and device | |
| EP3399466A1 (en) | Neural network training image generation system | |
| Wang et al. | Vision-assisted BIM reconstruction from 3D LiDAR point clouds for MEP scenes | |
| Roy et al. | Active recognition through next view planning: a survey | |
| CN103268495B (en) | Human body behavior modeling recognition methods based on priori knowledge cluster in computer system | |
| CN104933436B (en) | The multi-cam factory monitoring of view-based access control model with the scoring of dynamic integrality | |
| Aleotti et al. | Perception and grasping of object parts from active robot exploration | |
| CN115965960A (en) | Weld joint identification method based on deep learning and 3D point cloud | |
| Yu et al. | A deep-learning-based strategy for kidnapped robot problem in similar indoor environment | |
| CN115457130A (en) | Electric vehicle charging port detection and positioning method based on depth key point regression | |
| Billings et al. | SilhoNet-fisheye: Adaptation of a ROI based object pose estimation network to monocular fisheye images | |
| Pang et al. | A verification-oriented and part-focused assembly monitoring system based on multi-layered digital twin | |
| Wen et al. | CAE-RLSM: Consistent and efficient redundant line segment merging for online feature map building | |
| Xia et al. | [Retracted] Gesture Tracking and Recognition Algorithm for Dynamic Human Motion Using Multimodal Deep Learning | |
| CN114399515A (en) | Language description-based class-level target object 6D pose acquisition method and storage medium | |
| Ma et al. | Self-Supervised Drivable Area Segmentation Using LiDAR's Depth Information for Autonomous Driving | |
| CN116403275B (en) | Method and system for detecting personnel advancing posture in closed space based on multi-vision | |
| Wu et al. | A novel cable-grasping planner for manipulator based on the operation surface | |
| Adán et al. | 3D feature tracking using a dynamic structured light system | |
| Zhu et al. | Recurrent multi-view collaborative registration network for 3D reconstruction and optical measurement of blade profiles | |
| Aggarwal et al. | Object recognition and performance bounds |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |