CN116475081B - Industrial product sorting control method, device and system based on cloud edge cooperation - Google Patents

Industrial product sorting control method, device and system based on cloud edge cooperation Download PDF

Info

Publication number
CN116475081B
CN116475081B CN202310752435.0A CN202310752435A CN116475081B CN 116475081 B CN116475081 B CN 116475081B CN 202310752435 A CN202310752435 A CN 202310752435A CN 116475081 B CN116475081 B CN 116475081B
Authority
CN
China
Prior art keywords
sorting
sorted
target
industrial product
target industrial
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202310752435.0A
Other languages
Chinese (zh)
Other versions
CN116475081A (en
Inventor
王露露
蒋抱阳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Industrial Fulian Foshan Innovation Center Co ltd
Original Assignee
Industrial Fulian Foshan Innovation Center Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Industrial Fulian Foshan Innovation Center Co ltd filed Critical Industrial Fulian Foshan Innovation Center Co ltd
Priority to CN202310752435.0A priority Critical patent/CN116475081B/en
Publication of CN116475081A publication Critical patent/CN116475081A/en
Application granted granted Critical
Publication of CN116475081B publication Critical patent/CN116475081B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B07—SEPARATING SOLIDS FROM SOLIDS; SORTING
    • B07C—POSTAL SORTING; SORTING INDIVIDUAL ARTICLES, OR BULK MATERIAL FIT TO BE SORTED PIECE-MEAL, e.g. BY PICKING
    • B07C5/00—Sorting according to a characteristic or feature of the articles or material being sorted, e.g. by control effected by devices which detect or measure such characteristic or feature; Sorting by manually actuated devices, e.g. switches
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B07—SEPARATING SOLIDS FROM SOLIDS; SORTING
    • B07C—POSTAL SORTING; SORTING INDIVIDUAL ARTICLES, OR BULK MATERIAL FIT TO BE SORTED PIECE-MEAL, e.g. BY PICKING
    • B07C5/00—Sorting according to a characteristic or feature of the articles or material being sorted, e.g. by control effected by devices which detect or measure such characteristic or feature; Sorting by manually actuated devices, e.g. switches
    • B07C5/36—Sorting apparatus characterised by the means used for distribution
    • B07C5/361—Processing or control devices therefor, e.g. escort memory
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061—Partitioning or combining of resources
    • G06F9/5072—Grid computing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Quality & Reliability (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses an industrial product sorting control method, device and system based on cloud edge cooperation, and relates to the technical field of sorting control, wherein the method comprises the following steps: the sorting equipment end uploads industrial product data to be sorted to the edge end, and the edge end obtains a sorting detection model corresponding to the industrial product data at the cloud end; the edge end generates a sorting control instruction and sends the sorting control instruction to a sorting equipment end; the sorting equipment end performs image acquisition and target tracking processing on the target industrial products to be sorted; uploading the target industrial product image to be sorted to an edge end, and performing target image extraction processing on the edge end to obtain an extraction target image; inputting the extracted target image into a sorting detection model for sorting detection treatment, and transmitting a sorting detection result to a sorting equipment end; and the sorting equipment end performs sorting control treatment on defective products of the target industrial products to be sorted. In the embodiment of the invention, the sorting control of the target industrial products can be realized, and the sorting accuracy is improved.

Description

Industrial product sorting control method, device and system based on cloud edge cooperation
Technical Field
The invention relates to the technical field of sorting control, in particular to an industrial product sorting control method, device and system based on cloud edge cooperation.
Background
Currently, a quality inspection link in industrial production is a very important step. The traditional industrial quality inspection mode is usually carried out manually, has low efficiency and is easy to make mistakes, and a large amount of human resources are needed, so that the cost is high. To solve these problems, many enterprises try to adopt machine vision technology for quality inspection, but because of high equipment cost and complex algorithm, higher technical support is required. Meanwhile, the accuracy and consistency of quality inspection results are difficult to ensure by the traditional quality inspection method, and complex quality inspection scenes are difficult to deal with; the traditional cloud computing scheme is difficult to realize real-time quality inspection due to the problems of network delay, bandwidth and the like, and cannot meet the requirements of industrial production.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, and provides an industrial product sorting control method, device and system based on cloud-edge cooperation, which can realize sorting control of target industrial products, improve sorting accuracy and meet industrial requirements; meanwhile, the sorting equipment end does not need high equipment cost, so that the use cost of a user is reduced.
In order to solve the technical problems, the invention provides an industrial product sorting control method based on cloud edge cooperation, which is applied to a sorting equipment end, an edge end and a cloud end, wherein the sorting equipment end is connected with the edge end, and the edge end is connected with the cloud end; the method comprises the following steps:
the sorting equipment end uploads industrial product data to be sorted to the edge end, and the edge end obtains a sorting detection model corresponding to the industrial product data on the cloud end based on the industrial product data;
after the edge end obtains a sorting detection model corresponding to the industrial product data, a sorting control instruction for controlling the starting of the sorting equipment end is generated, and the sorting control instruction is issued to the sorting equipment end;
the sorting equipment starts a visual sensor to perform image acquisition and target tracking processing on the target industrial products to be sorted based on the sorting control instruction, and obtains images of the target industrial products to be sorted and tracking results of the target industrial products to be sorted;
uploading the target industrial product image to be sorted to the edge end, and carrying out target image extraction processing on the target industrial product to be sorted in the received target industrial product image to be sorted by the edge end to obtain an extraction target image;
Inputting the extracted target image into a sorting detection model corresponding to the industrial product data for sorting detection processing, obtaining a sorting detection result, and issuing the sorting detection result to the sorting equipment end;
and the sorting equipment end performs sorting control treatment on defective products of the target industrial products to be sorted based on the sorting detection result and the tracking result of the target industrial products to be sorted.
Optionally, the obtaining, by the edge, a sorting detection model corresponding to the industrial product data at the cloud based on the industrial product data includes:
the edge end uploads the industrial product data to the cloud end;
the cloud end matches the received industrial product data with a sorting detection model corresponding to the industrial product data in a sorting detection model database, and sends the sorting detection model corresponding to the industrial product data to the edge end;
and the sorting detection model in the sorting detection model database is correspondingly bound and stored with the industrial product data.
Optionally, the training process of the sorting detection model in the cloud includes:
a plurality of sorting detection models to be trained are obtained in a model warehouse in a mode of importing or uploading by a dock mirror image according to the requirements of sorting product types, wherein the sorting detection models to be trained at least comprise two sorting detection models to be trained, and each sorting detection model to be trained is different;
Selecting annotation data corresponding to the plurality of sorting detection models to be trained from an annotation data storage database of the cloud to form a training data set;
node parameter setting processing is carried out on a plurality of sorting detection models to be trained, training processing is carried out on the plurality of sorting detection models to be trained after setting by utilizing training data in the training data set, and a plurality of sorting detection models with converged training are obtained;
selecting a corresponding evaluation type on the cloud model evaluation platform, and setting different test data for the same model and the same test data for different models for evaluation during evaluation;
selecting corresponding test data from the cloud data platform to form a plurality of test data sets;
selecting corresponding evaluation indexes and evaluation scoring rules according to the properties and requirements of the sorting tasks, and performing test evaluation processing on a plurality of sorting detection models converged by training by utilizing a plurality of test data sets under the same computing environment to obtain a plurality of test evaluation results;
generating a comparison form table corresponding to a plurality of sorting detection models converged in training based on the plurality of test evaluation results, wherein the comparison form table comprises the score of each evaluation index and the comprehensive score;
And sorting from high to low according to each evaluation index score and the comprehensive score based on the comparison form table, and respectively selecting a training converged sorting detection model with the highest evaluation index score and the highest comprehensive score sorting to store in a sorting detection model database.
Optionally, the generating a sorting control instruction for controlling the starting of the sorting equipment includes:
and the edge end is triggered based on a sorting detection model corresponding to the industrial product data, and a sorting control instruction for controlling the starting of the sorting equipment end is generated.
Optionally, the sorting equipment starts the vision sensor to perform image acquisition and target tracking processing on the target industrial product to be sorted based on the sorting control instruction, and obtains an image of the target industrial product to be sorted and a tracking result of the target industrial product to be sorted, including:
the sorting equipment starts a visual sensor to acquire and process images of the target industrial products to be sorted according to preset interval time based on the sorting control instruction, and continuous multi-frame images of the target industrial products to be sorted are obtained;
performing foreground and background separation processing on the continuous multi-frame to-be-sorted target industrial product images to obtain foreground target images of the continuous multi-frame to-be-sorted target industrial product images;
Performing contour similarity matching processing on the contour model corresponding to the industrial product data and the foreground target image of the continuous multi-frame to-be-sorted target industrial product image to obtain a contour similarity matching result;
selecting and obtaining target industrial product images to be sorted from the continuous multi-frame target industrial product images to be sorted based on the contour similarity matching result;
and carrying out target tracking processing on the target industrial product to be sorted based on the continuous multi-frame target industrial product image to be sorted, and obtaining a target industrial product tracking result to be sorted.
Optionally, the performing target tracking processing on the target industrial product to be sorted based on the continuous multi-frame target industrial product image to be sorted to obtain a target industrial product tracking result to be sorted, including:
analyzing and processing the motion trail of the target industrial product to be sorted in the continuous multi-frame target industrial product image to be sorted by adopting a Kalman filtering algorithm to obtain a motion trail analysis result;
and carrying out target tracking processing on the target industrial products to be sorted based on the motion trail analysis result to obtain a target industrial product tracking result to be sorted.
Optionally, the edge end performs target image extraction processing on the target industrial product to be sorted in the received target industrial product image to be sorted, so as to obtain an extracted target image, which includes:
the edge end extracts the outline of the target industrial product to be sorted in the target industrial product image to be sorted based on an image outline extraction algorithm to obtain extracted outline information;
and carrying out target image extraction processing on the target industrial products to be sorted in the target industrial product images to be sorted based on the extraction contour information, and obtaining an extraction target image.
Optionally, the sorting equipment end performs sorting control processing of defective products on the target industrial products to be sorted based on the sorting detection result and the target industrial product tracking result to be sorted, including:
when the sorting detection result of the target industrial product to be sorted is a defective product, the sorting equipment end obtains the current position of the target industrial product to be sorted based on the tracking result of the target industrial product to be sorted;
and the sorting equipment end controls the sorting mechanical arm arranged on the sorting equipment end to sort defective products based on the current position of the target industrial product to be sorted.
In addition, the invention also provides an industrial product sorting control device based on cloud edge cooperation, which is applied to a sorting equipment end, an edge end and a cloud end, wherein the sorting equipment end is connected with the edge end, and the edge end is connected with the cloud end; the device comprises:
the obtaining module is as follows: the sorting equipment end is used for uploading industrial product data to be sorted to the edge end, and the edge end obtains a sorting detection model corresponding to the industrial product data on the cloud end based on the industrial product data;
the generation module is used for: after the edge end obtains a sorting detection model corresponding to the industrial product data, a sorting control instruction for controlling the starting of the sorting equipment end is generated, and the sorting control instruction is issued to the sorting equipment end;
image acquisition and target tracking module: the sorting equipment end starts a visual sensor to perform image acquisition and target tracking processing on the target industrial products to be sorted based on the sorting control instruction, and obtains images of the target industrial products to be sorted and tracking results of the target industrial products to be sorted;
the target extraction module: the method comprises the steps that the target industrial product image to be sorted is uploaded to the edge end, and the edge end performs target image extraction processing on the target industrial product to be sorted in the received target industrial product image to be sorted, so that an extraction target image is obtained;
Sorting detection module: the extraction target image is input into a sorting detection model corresponding to the industrial product data for sorting detection processing, a sorting detection result is obtained, and the sorting detection result is issued to the sorting equipment end;
sorting control module: and the sorting equipment end is used for sorting control treatment of defective products of the target industrial products to be sorted based on the sorting detection result and the tracking result of the target industrial products to be sorted.
In addition, the invention also provides an industrial product sorting control system based on cloud edge cooperation, which comprises a sorting equipment end, an edge end and a cloud end, wherein the sorting equipment end is connected with the edge end, and the edge end is connected with the cloud end; the system is configured to perform the industrial product sort control method of any one of the above.
In the embodiment of the invention, the sorting equipment end, the edge end and the cloud end are arranged, so that the hardware cost of the sorting equipment end is greatly reduced, the sorting detection model matched on the cloud end is utilized for sorting detection, the product defects can be automatically identified, and a large amount of data analysis and processing work can be completed in a short time, thereby greatly improving the detection efficiency and accuracy; meanwhile, the full-automatic quality inspection process can be realized, so that the quality inspection cost is greatly reduced; meanwhile, customized development and deployment can be carried out at the cloud according to different enterprise demands, and meanwhile, the sorting efficiency and accuracy can be continuously improved by continuously updating and upgrading so as to meet different customer demands.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings which are required in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of an industrial product sorting control method based on Yun Bian cooperation in an embodiment of the invention;
fig. 2 is a schematic structural diagram of an industrial product sorting control device based on Yun Bian cooperation in an embodiment of the invention;
fig. 3 is a schematic structural diagram of an industrial product sorting control system based on Yun Bian cooperation in an embodiment of the invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, fig. 1 is a flow chart of an industrial product sorting control method based on Yun Bian cooperation according to an embodiment of the invention.
As shown in fig. 1, the industrial product sorting control method based on cloud edge cooperation is applied to a sorting equipment end, an edge end and a cloud end, wherein the sorting equipment end is connected with the edge end, and the edge end is connected with the cloud end;
in the specific implementation process of the cloud-edge load deep learning model, a cloud-edge cooperative platform architecture is formed by a cloud end and an edge end, and related works such as training and optimizing of the cloud-edge load deep learning model are carried out, and meanwhile large-scale data storage and processing are carried out; the edge end can be used for processing data uploaded by the sorting equipment end, and the sorting equipment end can be used for performing operations such as data acquisition, sorting control and the like; in order to achieve the best performance, the model can be optimized and compressed according to the calculation capability, the storage capability, the network bandwidth and other factors of the equipment; under the cloud-edge cooperative architecture, training and optimization of the model can be completed in the cloud end, and reasoning of the model is performed on the edge end, so that the speed and accuracy of the model can be greatly improved.
The method comprises the following steps:
s11: the sorting equipment end uploads industrial product data to be sorted to the edge end, and the edge end obtains a sorting detection model corresponding to the industrial product data on the cloud end based on the industrial product data;
in the implementation process of the invention, the edge end obtains a sorting detection model corresponding to the industrial product data at the cloud end based on the industrial product data, and the sorting detection model comprises the following steps: the edge end uploads the industrial product data to the cloud end; the cloud end matches the received industrial product data with a sorting detection model corresponding to the industrial product data in a sorting detection model database, and sends the sorting detection model corresponding to the industrial product data to the edge end; and the sorting detection model in the sorting detection model database is correspondingly bound and stored with the industrial product data.
Further, the training process of the sorting detection model in the cloud comprises the following steps: a plurality of sorting detection models to be trained are obtained in a model warehouse in a mode of importing or uploading by a dock mirror image according to the requirements of sorting product types, wherein the sorting detection models to be trained at least comprise two sorting detection models to be trained, and each sorting detection model to be trained is different; selecting annotation data corresponding to the plurality of sorting detection models to be trained from an annotation data storage database of the cloud to form a training data set; node parameter setting processing is carried out on a plurality of sorting detection models to be trained, training processing is carried out on the plurality of sorting detection models to be trained after setting by utilizing training data in the training data set, and a plurality of sorting detection models with converged training are obtained; selecting a corresponding evaluation type on the cloud model evaluation platform, and setting different test data for the same model and the same test data for different models for evaluation during evaluation; selecting corresponding test data from the cloud data platform to form a plurality of test data sets; selecting corresponding evaluation indexes and evaluation scoring rules according to the properties and requirements of the sorting tasks, and performing test evaluation processing on a plurality of sorting detection models converged by training by utilizing a plurality of test data sets under the same computing environment to obtain a plurality of test evaluation results; generating a comparison form table corresponding to a plurality of sorting detection models converged in training based on the plurality of test evaluation results, wherein the comparison form table comprises the score of each evaluation index and the comprehensive score; and sorting from high to low according to each evaluation index score and the comprehensive score based on the comparison form table, and respectively selecting a training converged sorting detection model with the highest evaluation index score and the highest comprehensive score sorting to store in a sorting detection model database.
Specifically, when the sorting equipment end needs sorting operation, industrial product data needing sorting is firstly required to be obtained, the industrial product data is uploaded to the edge end, after the industrial product data is received by the edge end, the industrial product data is uploaded to the cloud end, then the far end performs matching processing in a sorting detection model database by utilizing the industrial product data, the sorting detection model corresponding to the industrial product data is matched, and the sorting detection model corresponding to the industrial product data is sent to the edge end; thus, the edge end can obtain a sorting detection model corresponding to the industrial product data; and in the sorting detection model database, the sorting detection model is correspondingly bound and stored with the industrial product data.
The method comprises the steps that a plurality of sorting detection models to be trained are needed to be obtained simultaneously, and the sorting detection models can be imported into a model warehouse or are obtained in a mode of uploading by a dock mirror image according to the type of sorting products, wherein the sorting detection models to be trained at least comprise two sorting detection models to be trained, and each sorting detection model to be trained is different; selecting annotation data corresponding to a plurality of sorting detection models to be trained from an annotation data storage database of the cloud to form a training data set, wherein the annotation data are target images for training, which are marked with different states, and the different states mainly comprise a defect state and a normal state; the image size of the target image for training can be 640 x 640, and can be specifically adjusted according to the model requirement; node parameter setting processing is carried out on a plurality of sorting detection models to be trained, training processing is carried out on the plurality of sorting detection models to be trained after setting by utilizing training data in a training data set, and a plurality of sorting detection models with training convergence are obtained; selecting a corresponding evaluation type on a cloud model evaluation platform, and setting to use different test data for the same model during evaluation, and using the same test data for different models for evaluation; performing model evaluation on a model evaluation platform, and selecting evaluation types, different data of the same model, different models of the same data or customization; the different data of the same model are mainly the evaluation results of the research model on the different data, for example, the evaluation results of the model A on the data of different AOI devices or different products can obtain which type of products or which devices the model is more suitable for; the same data and different models are mainly used for comparing the evaluation results of the same equipment or the same product and different models, so that a model which is more applicable to the product or the equipment is selected; the customization can select and compare a plurality of data attributes including product names, product batches, AOI equipment, quality inspection tasks and the like, and a plurality of models, namely a many-to-many relationship; selecting corresponding test data from a data platform of the cloud to form a plurality of test data sets; selecting corresponding evaluation indexes and evaluation scoring rules according to the properties and requirements of the sorting tasks, and performing test evaluation processing on a plurality of sorting detection models converged by training by utilizing a plurality of test data sets under the same computing environment to obtain a plurality of test evaluation results; selecting an evaluation index and giving a score according to the properties and requirements of the task; the indexes comprise a miscibility rate, a omission rate, an accuracy rate, a miscibility rate and an accuracy rate, and users can assign points according to own requirements, and the total points are not more than 100; then generating a comparison form table corresponding to a plurality of sorting detection models which are converged in training according to a plurality of test evaluation results, wherein the comparison form table comprises the score of each evaluation index and the comprehensive score; and finally, sorting from high to low according to each evaluation index score and the comprehensive score according to the comparison form table, and respectively selecting the training converged sorting detection model with the highest sorting of each evaluation index score and the highest sorting of the comprehensive score to store in a sorting detection model database.
When the model is evaluated, the model is evaluated under the same computing force environment, so that the accuracy and comparability of an evaluation result are ensured; the method comprises the following steps of unified hardware environment, software environment, data set and data preprocessing, random number seed and the like: a hardware environment: the hardware environment of all the evaluation models needs to be the same, including hardware specifications and configurations of the CPU, GPU, etc. The unified hardware environment can be built at the cloud end, and a designated hardware configuration list can be provided for the user, so that the user is required to perform corresponding configuration in the local environment to ensure consistency. b software environment: the software environment of all assessment models needs to be the same, except for the hardware environment. Therefore, a fixed software environment including an operating system, a framework version, a dependency library, etc. needs to be provided in the cloud, and periodic update and maintenance are required. c data set and data preprocessing: the same data set and data preprocessing are important factors to ensure comparability of model evaluations. Thus, there is a need to provide a unified data set and data preprocessing method at the data platform to ensure that each user uses the same data set and preprocessing method at the time of evaluation. d, random number seed: in some cases, the model evaluation results may be affected by randomness. Thus, the same random number seed needs to be used in model training and evaluation to ensure repeatability of the results.
S12: after the edge end obtains a sorting detection model corresponding to the industrial product data, a sorting control instruction for controlling the starting of the sorting equipment end is generated, and the sorting control instruction is issued to the sorting equipment end;
in the implementation process of the invention, the generating the sorting control instruction for controlling the starting of the sorting equipment end comprises the following steps: and the edge end is triggered based on a sorting detection model corresponding to the industrial product data, and a sorting control instruction for controlling the starting of the sorting equipment end is generated.
Specifically, after the edge end obtains a sorting detection model corresponding to the industrial product data, the edge end triggers and generates a sorting control instruction for controlling the starting of the sorting equipment end according to the sorting detection model corresponding to the industrial product data; and issuing sorting control instructions to the sorting equipment end.
S13: the sorting equipment starts a visual sensor to perform image acquisition and target tracking processing on the target industrial products to be sorted based on the sorting control instruction, and obtains images of the target industrial products to be sorted and tracking results of the target industrial products to be sorted;
in the specific implementation process of the invention, the sorting equipment starts a visual sensor to perform image acquisition and target tracking processing on the target industrial product to be sorted based on the sorting control instruction to obtain an image of the target industrial product to be sorted and a tracking result of the target industrial product to be sorted, and the method comprises the following steps: the sorting equipment starts a visual sensor to acquire and process images of the target industrial products to be sorted according to preset interval time based on the sorting control instruction, and continuous multi-frame images of the target industrial products to be sorted are obtained; performing foreground and background separation processing on the continuous multi-frame to-be-sorted target industrial product images to obtain foreground target images of the continuous multi-frame to-be-sorted target industrial product images; performing contour similarity matching processing on the contour model corresponding to the industrial product data and the foreground target image of the continuous multi-frame to-be-sorted target industrial product image to obtain a contour similarity matching result; selecting and obtaining target industrial product images to be sorted from the continuous multi-frame target industrial product images to be sorted based on the contour similarity matching result; and carrying out target tracking processing on the target industrial product to be sorted based on the continuous multi-frame target industrial product image to be sorted, and obtaining a target industrial product tracking result to be sorted.
Further, the performing target tracking processing on the target industrial product to be sorted based on the continuous multi-frame target industrial product image to be sorted to obtain a target industrial product tracking result to be sorted, including: analyzing and processing the motion trail of the target industrial product to be sorted in the continuous multi-frame target industrial product image to be sorted by adopting a Kalman filtering algorithm to obtain a motion trail analysis result; and carrying out target tracking processing on the target industrial products to be sorted based on the motion trail analysis result to obtain a target industrial product tracking result to be sorted.
Specifically, the sorting equipment starts a visual sensor arranged on the sorting equipment through a sorting control instruction to acquire and process images of the target industrial products to be sorted according to a preset interval time, so that continuous multi-frame images of the target industrial products to be sorted can be obtained; then, foreground and background separation processing is needed to be carried out on the continuous multi-frame to-be-sorted target industrial product images, so that the foreground target images of the continuous multi-frame to-be-sorted target industrial product images can be obtained; then, carrying out contour similarity matching processing on a contour model corresponding to the industrial product data and a foreground target image of a continuous multi-frame to-be-sorted target industrial product image to obtain a contour similarity matching result; selecting a continuous multi-frame target industrial product image to be sorted according to the contour similarity matching result to obtain a target industrial product image to be sorted; selecting an image with highest matching degree in the contour similarity matching result as an image of a target industrial product to be sorted; and finally, carrying out target tracking treatment on the target industrial products to be sorted according to the continuous multi-frame target industrial product images to be sorted, and obtaining the target industrial product tracking result to be sorted.
When the target tracking processing is carried out, a Kalman filtering algorithm is adopted to analyze and process the motion trail of the target industrial product to be sorted in the continuous multi-frame target industrial product image to be sorted, and a motion trail analysis result can be obtained; and then carrying out target tracking treatment on the target industrial products to be sorted according to the analysis result of the motion trail, and finally obtaining the tracking result of the target industrial products to be sorted.
S14: uploading the target industrial product image to be sorted to the edge end, and carrying out target image extraction processing on the target industrial product to be sorted in the received target industrial product image to be sorted by the edge end to obtain an extraction target image;
in the implementation process of the invention, the edge end performs target image extraction processing on the target industrial product to be sorted in the received target industrial product image to be sorted to obtain an extraction target image, which comprises the following steps: the edge end extracts the outline of the target industrial product to be sorted in the target industrial product image to be sorted based on an image outline extraction algorithm to obtain extracted outline information; and carrying out target image extraction processing on the target industrial products to be sorted in the target industrial product images to be sorted based on the extraction contour information, and obtaining an extraction target image.
Specifically, in order to improve the subsequent image processing efficiency, the image of the target industrial product to be sorted needs to be subjected to target extraction in the edge end; and then carrying out target image extraction processing on the target industrial products to be sorted in the target industrial product images to be sorted according to the extraction contour information, and finally obtaining an extraction target image.
S15: inputting the extracted target image into a sorting detection model corresponding to the industrial product data for sorting detection processing, obtaining a sorting detection result, and issuing the sorting detection result to the sorting equipment end;
in the specific implementation process of the application, only the extracted target image is input into a sorting detection model corresponding to industrial product data for sorting detection treatment, namely, the sorting detection result is output and obtained, and then the sorting detection result is issued to a sorting equipment end; and uploading the sorting detection result to the cloud for storage, wherein the size of the image of the extracted target image is consistent with the size of the image in the labeling data.
S16: and the sorting equipment end performs sorting control treatment on defective products of the target industrial products to be sorted based on the sorting detection result and the tracking result of the target industrial products to be sorted.
In the implementation process of the invention, the sorting equipment end performs sorting control processing on defective products of the target industrial products to be sorted based on the sorting detection result and the tracking result of the target industrial products to be sorted, and the sorting control processing comprises the following steps: when the sorting detection result of the target industrial product to be sorted is a defective product, the sorting equipment end obtains the current position of the target industrial product to be sorted based on the tracking result of the target industrial product to be sorted; and the sorting equipment end controls the sorting mechanical arm arranged on the sorting equipment end to sort defective products based on the current position of the target industrial product to be sorted.
Specifically, after the sorting equipment receives the sorting detection result, firstly identifying whether the sorting result is a defective product or a good product, if the sorting result is a defective product, then the sorting treatment is not needed, and if the sorting detection result is a defective product, then the current position of the target industrial product to be sorted is needed to be obtained according to the tracking result of the target industrial product to be sorted; and then the sorting equipment end controls a sorting mechanical arm arranged on the sorting equipment end to sort defective products according to the current position of the target industrial product to be sorted.
In the embodiment of the invention, the sorting equipment end, the edge end and the cloud end are arranged, so that the hardware cost of the sorting equipment end is greatly reduced, the sorting detection model matched on the cloud end is utilized for sorting detection, the product defects can be automatically identified, and a large amount of data analysis and processing work can be completed in a short time, thereby greatly improving the detection efficiency and accuracy; meanwhile, the full-automatic quality inspection process can be realized, so that the quality inspection cost is greatly reduced; meanwhile, customized development and deployment can be carried out at the cloud according to different enterprise demands, and meanwhile, the sorting efficiency and accuracy can be continuously improved by continuously updating and upgrading so as to meet different customer demands.
Referring to fig. 2, fig. 2 is a schematic structural diagram of an industrial product sorting control device based on Yun Bian cooperation in an embodiment of the invention.
As shown in fig. 2, an industrial product sorting control device based on cloud edge cooperation is applied to a sorting equipment end, an edge end and a cloud end, wherein the sorting equipment end is connected with the edge end, and the edge end is connected with the cloud end; the device comprises:
Obtaining module 21: the sorting equipment end is used for uploading industrial product data to be sorted to the edge end, and the edge end obtains a sorting detection model corresponding to the industrial product data on the cloud end based on the industrial product data;
in the implementation process of the invention, the edge end obtains a sorting detection model corresponding to the industrial product data at the cloud end based on the industrial product data, and the sorting detection model comprises the following steps: the edge end uploads the industrial product data to the cloud end; the cloud end matches the received industrial product data with a sorting detection model corresponding to the industrial product data in a sorting detection model database, and sends the sorting detection model corresponding to the industrial product data to the edge end; and the sorting detection model in the sorting detection model database is correspondingly bound and stored with the industrial product data.
Further, the training process of the sorting detection model in the cloud comprises the following steps: a plurality of sorting detection models to be trained are obtained in a model warehouse in a mode of importing or uploading by a dock mirror image according to the requirements of sorting product types, wherein the sorting detection models to be trained at least comprise two sorting detection models to be trained, and each sorting detection model to be trained is different; selecting annotation data corresponding to the plurality of sorting detection models to be trained from an annotation data storage database of the cloud to form a training data set; node parameter setting processing is carried out on a plurality of sorting detection models to be trained, training processing is carried out on the plurality of sorting detection models to be trained after setting by utilizing training data in the training data set, and a plurality of sorting detection models with converged training are obtained; selecting a corresponding evaluation type on the cloud model evaluation platform, and setting different test data for the same model and the same test data for different models for evaluation during evaluation; selecting corresponding test data from the cloud data platform to form a plurality of test data sets; selecting corresponding evaluation indexes and evaluation scoring rules according to the properties and requirements of the sorting tasks, and performing test evaluation processing on a plurality of sorting detection models converged by training by utilizing a plurality of test data sets under the same computing environment to obtain a plurality of test evaluation results; generating a comparison form table corresponding to a plurality of sorting detection models converged in training based on the plurality of test evaluation results, wherein the comparison form table comprises the score of each evaluation index and the comprehensive score; and sorting from high to low according to each evaluation index score and the comprehensive score based on the comparison form table, and respectively selecting a training converged sorting detection model with the highest evaluation index score and the highest comprehensive score sorting to store in a sorting detection model database.
Specifically, when the sorting equipment end needs sorting operation, industrial product data needing sorting is firstly required to be obtained, the industrial product data is uploaded to the edge end, after the industrial product data is received by the edge end, the industrial product data is uploaded to the cloud end, then the far end performs matching processing in a sorting detection model database by utilizing the industrial product data, the sorting detection model corresponding to the industrial product data is matched, and the sorting detection model corresponding to the industrial product data is sent to the edge end; thus, the edge end can obtain a sorting detection model corresponding to the industrial product data; and in the sorting detection model database, the sorting detection model is correspondingly bound and stored with the industrial product data.
The method comprises the steps that a plurality of sorting detection models to be trained are needed to be obtained simultaneously, and the sorting detection models can be imported into a model warehouse or are obtained in a mode of uploading by a dock mirror image according to the type of sorting products, wherein the sorting detection models to be trained at least comprise two sorting detection models to be trained, and each sorting detection model to be trained is different; selecting annotation data corresponding to a plurality of sorting detection models to be trained from an annotation data storage database of the cloud to form a training data set, wherein the annotation data are target images for training, which are marked with different states, and the different states mainly comprise a defect state and a normal state; the image size of the target image for training can be 640 x 640, and can be specifically adjusted according to the model requirement; node parameter setting processing is carried out on a plurality of sorting detection models to be trained, training processing is carried out on the plurality of sorting detection models to be trained after setting by utilizing training data in a training data set, and a plurality of sorting detection models with training convergence are obtained; selecting a corresponding evaluation type on a cloud model evaluation platform, and setting to use different test data for the same model during evaluation, and using the same test data for different models for evaluation; performing model evaluation on a model evaluation platform, and selecting evaluation types, different data of the same model, different models of the same data or customization; the different data of the same model are mainly the evaluation results of the research model on the different data, for example, the evaluation results of the model A on the data of different AOI devices or different products can obtain which type of products or which devices the model is more suitable for; the same data and different models are mainly used for comparing the evaluation results of the same equipment or the same product and different models, so that a model which is more applicable to the product or the equipment is selected; the customization can select and compare a plurality of data attributes including product names, product batches, AOI equipment, quality inspection tasks and the like, and a plurality of models, namely a many-to-many relationship; selecting corresponding test data from a data platform of the cloud to form a plurality of test data sets; selecting corresponding evaluation indexes and evaluation scoring rules according to the properties and requirements of the sorting tasks, and performing test evaluation processing on a plurality of sorting detection models converged by training by utilizing a plurality of test data sets under the same computing environment to obtain a plurality of test evaluation results; selecting an evaluation index and giving a score according to the properties and requirements of the task; the indexes comprise a miscibility rate, a omission rate, an accuracy rate, a miscibility rate and an accuracy rate, and users can assign points according to own requirements, and the total points are not more than 100; then generating a comparison form table corresponding to a plurality of sorting detection models which are converged in training according to a plurality of test evaluation results, wherein the comparison form table comprises the score of each evaluation index and the comprehensive score; and finally, sorting from high to low according to each evaluation index score and the comprehensive score according to the comparison form table, and respectively selecting the training converged sorting detection model with the highest sorting of each evaluation index score and the highest sorting of the comprehensive score to store in a sorting detection model database.
When the model is evaluated, the model is evaluated under the same computing force environment, so that the accuracy and comparability of an evaluation result are ensured; the method comprises the following steps of unified hardware environment, software environment, data set and data preprocessing, random number seed and the like: a hardware environment: the hardware environment of all the evaluation models needs to be the same, including hardware specifications and configurations of the CPU, GPU, etc. The unified hardware environment can be built at the cloud end, and a designated hardware configuration list can be provided for the user, so that the user is required to perform corresponding configuration in the local environment to ensure consistency. b software environment: the software environment of all assessment models needs to be the same, except for the hardware environment. Therefore, a fixed software environment including an operating system, a framework version, a dependency library, etc. needs to be provided in the cloud, and periodic update and maintenance are required. c data set and data preprocessing: the same data set and data preprocessing are important factors to ensure comparability of model evaluations. Thus, there is a need to provide a unified data set and data preprocessing method at the data platform to ensure that each user uses the same data set and preprocessing method at the time of evaluation. d, random number seed: in some cases, the model evaluation results may be affected by randomness. Thus, the same random number seed needs to be used in model training and evaluation to ensure repeatability of the results.
The generation module 22: after the edge end obtains a sorting detection model corresponding to the industrial product data, a sorting control instruction for controlling the starting of the sorting equipment end is generated, and the sorting control instruction is issued to the sorting equipment end;
in the implementation process of the invention, the generating the sorting control instruction for controlling the starting of the sorting equipment end comprises the following steps: and the edge end is triggered based on a sorting detection model corresponding to the industrial product data, and a sorting control instruction for controlling the starting of the sorting equipment end is generated.
Specifically, after the edge end obtains a sorting detection model corresponding to the industrial product data, the edge end triggers and generates a sorting control instruction for controlling the starting of the sorting equipment end according to the sorting detection model corresponding to the industrial product data; and issuing sorting control instructions to the sorting equipment end.
Image acquisition and target tracking module 23: the sorting equipment end starts a visual sensor to perform image acquisition and target tracking processing on the target industrial products to be sorted based on the sorting control instruction, and obtains images of the target industrial products to be sorted and tracking results of the target industrial products to be sorted;
In the specific implementation process of the invention, the sorting equipment starts a visual sensor to perform image acquisition and target tracking processing on the target industrial product to be sorted based on the sorting control instruction to obtain an image of the target industrial product to be sorted and a tracking result of the target industrial product to be sorted, and the method comprises the following steps: the sorting equipment starts a visual sensor to acquire and process images of the target industrial products to be sorted according to preset interval time based on the sorting control instruction, and continuous multi-frame images of the target industrial products to be sorted are obtained; performing foreground and background separation processing on the continuous multi-frame to-be-sorted target industrial product images to obtain foreground target images of the continuous multi-frame to-be-sorted target industrial product images; performing contour similarity matching processing on the contour model corresponding to the industrial product data and the foreground target image of the continuous multi-frame to-be-sorted target industrial product image to obtain a contour similarity matching result; selecting and obtaining target industrial product images to be sorted from the continuous multi-frame target industrial product images to be sorted based on the contour similarity matching result; and carrying out target tracking processing on the target industrial product to be sorted based on the continuous multi-frame target industrial product image to be sorted, and obtaining a target industrial product tracking result to be sorted.
Further, the performing target tracking processing on the target industrial product to be sorted based on the continuous multi-frame target industrial product image to be sorted to obtain a target industrial product tracking result to be sorted, including: analyzing and processing the motion trail of the target industrial product to be sorted in the continuous multi-frame target industrial product image to be sorted by adopting a Kalman filtering algorithm to obtain a motion trail analysis result; and carrying out target tracking processing on the target industrial products to be sorted based on the motion trail analysis result to obtain a target industrial product tracking result to be sorted.
Specifically, the sorting equipment starts a visual sensor arranged on the sorting equipment through a sorting control instruction to acquire and process images of the target industrial products to be sorted according to a preset interval time, so that continuous multi-frame images of the target industrial products to be sorted can be obtained; then, foreground and background separation processing is needed to be carried out on the continuous multi-frame to-be-sorted target industrial product images, so that the foreground target images of the continuous multi-frame to-be-sorted target industrial product images can be obtained; then, carrying out contour similarity matching processing on a contour model corresponding to the industrial product data and a foreground target image of a continuous multi-frame to-be-sorted target industrial product image to obtain a contour similarity matching result; selecting a continuous multi-frame target industrial product image to be sorted according to the contour similarity matching result to obtain a target industrial product image to be sorted; selecting an image with highest matching degree in the contour similarity matching result as an image of a target industrial product to be sorted; and finally, carrying out target tracking treatment on the target industrial products to be sorted according to the continuous multi-frame target industrial product images to be sorted, and obtaining the target industrial product tracking result to be sorted.
When the target tracking processing is carried out, a Kalman filtering algorithm is adopted to analyze and process the motion trail of the target industrial product to be sorted in the continuous multi-frame target industrial product image to be sorted, and a motion trail analysis result can be obtained; and then carrying out target tracking treatment on the target industrial products to be sorted according to the analysis result of the motion trail, and finally obtaining the tracking result of the target industrial products to be sorted.
The target extraction module 24: the method comprises the steps that the target industrial product image to be sorted is uploaded to the edge end, and the edge end performs target image extraction processing on the target industrial product to be sorted in the received target industrial product image to be sorted, so that an extraction target image is obtained;
in the implementation process of the invention, the edge end performs target image extraction processing on the target industrial product to be sorted in the received target industrial product image to be sorted to obtain an extraction target image, which comprises the following steps: the edge end extracts the outline of the target industrial product to be sorted in the target industrial product image to be sorted based on an image outline extraction algorithm to obtain extracted outline information; and carrying out target image extraction processing on the target industrial products to be sorted in the target industrial product images to be sorted based on the extraction contour information, and obtaining an extraction target image.
Specifically, in order to improve the subsequent image processing efficiency, the image of the target industrial product to be sorted needs to be subjected to target extraction in the edge end; and then carrying out target image extraction processing on the target industrial products to be sorted in the target industrial product images to be sorted according to the extraction contour information, and finally obtaining an extraction target image.
Sorting detection module 25: the extraction target image is input into a sorting detection model corresponding to the industrial product data for sorting detection processing, a sorting detection result is obtained, and the sorting detection result is issued to the sorting equipment end;
in the specific implementation process of the application, only the extracted target image is input into a sorting detection model corresponding to industrial product data for sorting detection treatment, namely, the sorting detection result is output and obtained, and then the sorting detection result is issued to a sorting equipment end; and uploading the sorting detection result to the cloud for storage, wherein the size of the image of the extracted target image is consistent with the size of the image in the labeling data.
Sorting control module 26: and the sorting equipment end is used for sorting control treatment of defective products of the target industrial products to be sorted based on the sorting detection result and the tracking result of the target industrial products to be sorted.
In the implementation process of the invention, the sorting equipment end performs sorting control processing on defective products of the target industrial products to be sorted based on the sorting detection result and the tracking result of the target industrial products to be sorted, and the sorting control processing comprises the following steps: when the sorting detection result of the target industrial product to be sorted is a defective product, the sorting equipment end obtains the current position of the target industrial product to be sorted based on the tracking result of the target industrial product to be sorted; and the sorting equipment end controls the sorting mechanical arm arranged on the sorting equipment end to sort defective products based on the current position of the target industrial product to be sorted.
Specifically, after the sorting equipment receives the sorting detection result, firstly identifying whether the sorting result is a defective product or a good product, if the sorting result is a defective product, then the sorting treatment is not needed, and if the sorting detection result is a defective product, then the current position of the target industrial product to be sorted is needed to be obtained according to the tracking result of the target industrial product to be sorted; and then the sorting equipment end controls a sorting mechanical arm arranged on the sorting equipment end to sort defective products according to the current position of the target industrial product to be sorted.
In the embodiment of the invention, the sorting equipment end, the edge end and the cloud end are arranged, so that the hardware cost of the sorting equipment end is greatly reduced, the sorting detection model matched on the cloud end is utilized for sorting detection, the product defects can be automatically identified, and a large amount of data analysis and processing work can be completed in a short time, thereby greatly improving the detection efficiency and accuracy; meanwhile, the full-automatic quality inspection process can be realized, so that the quality inspection cost is greatly reduced; meanwhile, customized development and deployment can be carried out at the cloud according to different enterprise demands, and meanwhile, the sorting efficiency and accuracy can be continuously improved by continuously updating and upgrading so as to meet different customer demands.
Referring to fig. 3, fig. 3 is a schematic structural diagram of an industrial product sorting control system based on Yun Bian cooperation in an embodiment of the invention.
As shown in fig. 3, an industrial product sorting control system based on cloud edge cooperation comprises a sorting equipment end 31, an edge end 32 and a cloud end 33, wherein the sorting equipment end 31 is connected with the edge end 32, and the edge end 32 is connected with the cloud end 33; the system is configured to perform the industrial product sort control method of any one of the above.
In the implementation process of the present invention, the specific implementation of the system part may refer to the above embodiments, and will not be described herein.
Those of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be implemented by a program to instruct related hardware, the program may be stored in a computer readable storage medium, and the storage medium may include: read Only Memory (ROM), random access Memory (RAM, random Access Memory), magnetic or optical disk, and the like.
In addition, the method, the device and the system for controlling the sorting of the industrial products based on cloud-edge cooperation provided by the embodiment of the invention are described in detail, and specific examples are adopted to illustrate the principle and the implementation mode of the invention, and the description of the above embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.

Claims (7)

1. The industrial product sorting control method based on cloud edge cooperation is characterized by being applied to a sorting equipment end, an edge end and a cloud end, wherein the sorting equipment end is connected with the edge end, and the edge end is connected with the cloud end; the method comprises the following steps:
The sorting equipment end uploads industrial product data to be sorted to the edge end, and the edge end obtains a sorting detection model corresponding to the industrial product data on the cloud end based on the industrial product data;
after the edge end obtains a sorting detection model corresponding to the industrial product data, a sorting control instruction for controlling the starting of the sorting equipment end is generated, and the sorting control instruction is issued to the sorting equipment end;
the sorting equipment starts a visual sensor to perform image acquisition and target tracking processing on the target industrial products to be sorted based on the sorting control instruction, and obtains images of the target industrial products to be sorted and tracking results of the target industrial products to be sorted;
uploading the target industrial product image to be sorted to the edge end, and carrying out target image extraction processing on the target industrial product to be sorted in the received target industrial product image to be sorted by the edge end to obtain an extraction target image;
inputting the extracted target image into a sorting detection model corresponding to the industrial product data for sorting detection processing, obtaining a sorting detection result, and issuing the sorting detection result to the sorting equipment end;
The sorting equipment end performs sorting control treatment on defective products of the target industrial products to be sorted based on the sorting detection result and the tracking result of the target industrial products to be sorted;
the training process of the sorting detection model in the cloud comprises the following steps:
a plurality of sorting detection models to be trained are obtained in a model warehouse in a mode of importing or uploading by a dock mirror image according to the requirements of sorting product types, wherein the sorting detection models to be trained at least comprise two sorting detection models to be trained, and each sorting detection model to be trained is different;
selecting annotation data corresponding to the plurality of sorting detection models to be trained from an annotation data storage database of the cloud to form a training data set;
node parameter setting processing is carried out on a plurality of sorting detection models to be trained, training processing is carried out on the plurality of sorting detection models to be trained after setting by utilizing training data in the training data set, and a plurality of sorting detection models with converged training are obtained;
selecting a corresponding evaluation type on the cloud model evaluation platform, and setting different test data for the same model and the same test data for different models for evaluation during evaluation;
Selecting corresponding test data from the cloud data platform to form a plurality of test data sets;
selecting corresponding evaluation indexes and evaluation scoring rules according to the properties and requirements of the sorting tasks, and performing test evaluation processing on a plurality of sorting detection models converged by training by utilizing a plurality of test data sets under the same computing environment to obtain a plurality of test evaluation results;
generating a comparison form table corresponding to a plurality of sorting detection models converged in training based on the plurality of test evaluation results, wherein the comparison form table comprises the score of each evaluation index and the comprehensive score;
sorting from high to low according to each evaluation index score and the comprehensive score based on the comparison form table, and respectively selecting a training converged sorting detection model with the highest evaluation index score and the highest comprehensive score sorting to store in a sorting detection model database;
the sorting equipment end starts a visual sensor to perform image acquisition and target tracking processing on the target industrial products to be sorted based on the sorting control instruction to obtain images of the target industrial products to be sorted and tracking results of the target industrial products to be sorted, and the method comprises the following steps:
The sorting equipment starts a visual sensor to acquire and process images of the target industrial products to be sorted according to preset interval time based on the sorting control instruction, and continuous multi-frame images of the target industrial products to be sorted are obtained;
performing foreground and background separation processing on the continuous multi-frame to-be-sorted target industrial product images to obtain foreground target images of the continuous multi-frame to-be-sorted target industrial product images;
performing contour similarity matching processing on the contour model corresponding to the industrial product data and the foreground target image of the continuous multi-frame to-be-sorted target industrial product image to obtain a contour similarity matching result;
selecting and obtaining target industrial product images to be sorted from the continuous multi-frame target industrial product images to be sorted based on the contour similarity matching result;
performing target tracking processing on the target industrial products to be sorted based on the continuous multi-frame target industrial product images to be sorted, and obtaining tracking results of the target industrial products to be sorted;
the target tracking processing is performed on the target industrial product to be sorted based on the continuous multi-frame target industrial product image to be sorted, and a target industrial product tracking result to be sorted is obtained, including:
Analyzing and processing the motion trail of the target industrial product to be sorted in the continuous multi-frame target industrial product image to be sorted by adopting a Kalman filtering algorithm to obtain a motion trail analysis result;
and carrying out target tracking processing on the target industrial products to be sorted based on the motion trail analysis result to obtain a target industrial product tracking result to be sorted.
2. The industrial product sorting control method according to claim 1, wherein the obtaining, by the edge, a sorting detection model corresponding to the industrial product data at the cloud based on the industrial product data includes:
the edge end uploads the industrial product data to the cloud end;
the cloud end matches the received industrial product data with a sorting detection model corresponding to the industrial product data in a sorting detection model database, and sends the sorting detection model corresponding to the industrial product data to the edge end;
and the sorting detection model in the sorting detection model database is correspondingly bound and stored with the industrial product data.
3. The industrial product sort control method according to claim 1, wherein the generating sort control instructions that control the sort equipment side activation comprises:
And the edge end is triggered based on a sorting detection model corresponding to the industrial product data, and a sorting control instruction for controlling the starting of the sorting equipment end is generated.
4. The industrial product sorting control method according to claim 1, wherein the edge end performs target image extraction processing on the target industrial product to be sorted in the received target industrial product image to be sorted, and obtains an extraction target image, including:
the edge end extracts the outline of the target industrial product to be sorted in the target industrial product image to be sorted based on an image outline extraction algorithm to obtain extracted outline information;
and carrying out target image extraction processing on the target industrial products to be sorted in the target industrial product images to be sorted based on the extraction contour information, and obtaining an extraction target image.
5. The industrial product sorting control method according to claim 1, wherein the sorting equipment performs sorting control processing of defective products on the target industrial product to be sorted based on the sorting detection result and the target industrial product tracking result to be sorted, and the method comprises:
when the sorting detection result of the target industrial product to be sorted is a defective product, the sorting equipment end obtains the current position of the target industrial product to be sorted based on the tracking result of the target industrial product to be sorted;
And the sorting equipment end controls the sorting mechanical arm arranged on the sorting equipment end to sort defective products based on the current position of the target industrial product to be sorted.
6. The industrial product sorting control device based on cloud edge cooperation is characterized by being applied to a sorting equipment end, an edge end and a cloud end, wherein the sorting equipment end is connected with the edge end, and the edge end is connected with the cloud end; the device comprises:
the obtaining module is as follows: the sorting equipment end is used for uploading industrial product data to be sorted to the edge end and obtaining a sorting detection model corresponding to the industrial product data at the cloud end based on the industrial product data;
the generation module is used for: after the edge end obtains a sorting detection model corresponding to the industrial product data, a sorting control instruction for controlling the starting of the sorting equipment end is generated, and the sorting control instruction is issued to the sorting equipment end;
image acquisition and target tracking module: the sorting equipment end starts a visual sensor to perform image acquisition and target tracking processing on the target industrial products to be sorted based on the sorting control instruction, and obtains images of the target industrial products to be sorted and tracking results of the target industrial products to be sorted;
The target extraction module: the method comprises the steps of uploading the to-be-sorted target industrial product image to the edge end, and carrying out target image extraction processing on the to-be-sorted target industrial product in the received to-be-sorted target industrial product image by the edge end to obtain an extraction target image;
sorting detection module: the extraction target image is input into a sorting detection model corresponding to the industrial product data for sorting detection processing, a sorting detection result is obtained, and the sorting detection result is issued to the sorting equipment end;
sorting control module: the sorting equipment end is used for sorting and controlling defective products of the target industrial products to be sorted based on the sorting detection result and the tracking result of the target industrial products to be sorted;
the training process of the sorting detection model in the cloud comprises the following steps:
a plurality of sorting detection models to be trained are obtained in a model warehouse in a mode of importing or uploading by a dock mirror image according to the requirements of sorting product types, wherein the sorting detection models to be trained at least comprise two sorting detection models to be trained, and each sorting detection model to be trained is different;
Selecting annotation data corresponding to the plurality of sorting detection models to be trained from an annotation data storage database of the cloud to form a training data set;
node parameter setting processing is carried out on a plurality of sorting detection models to be trained, training processing is carried out on the plurality of sorting detection models to be trained after setting by utilizing training data in the training data set, and a plurality of sorting detection models with converged training are obtained;
selecting a corresponding evaluation type on the cloud model evaluation platform, and setting different test data for the same model and the same test data for different models for evaluation during evaluation;
selecting corresponding test data from the cloud data platform to form a plurality of test data sets;
selecting corresponding evaluation indexes and evaluation scoring rules according to the properties and requirements of the sorting tasks, and performing test evaluation processing on a plurality of sorting detection models converged by training by utilizing a plurality of test data sets under the same computing environment to obtain a plurality of test evaluation results;
generating a comparison form table corresponding to a plurality of sorting detection models converged in training based on the plurality of test evaluation results, wherein the comparison form table comprises the score of each evaluation index and the comprehensive score;
Sorting from high to low according to each evaluation index score and the comprehensive score based on the comparison form table, and respectively selecting a training converged sorting detection model with the highest evaluation index score and the highest comprehensive score sorting to store in a sorting detection model database;
the sorting equipment end starts a visual sensor to perform image acquisition and target tracking processing on the target industrial products to be sorted based on the sorting control instruction to obtain images of the target industrial products to be sorted and tracking results of the target industrial products to be sorted, and the method comprises the following steps:
the sorting equipment starts a visual sensor to acquire and process images of the target industrial products to be sorted according to preset interval time based on the sorting control instruction, and continuous multi-frame images of the target industrial products to be sorted are obtained;
performing foreground and background separation processing on the continuous multi-frame to-be-sorted target industrial product images to obtain foreground target images of the continuous multi-frame to-be-sorted target industrial product images;
performing contour similarity matching processing on the contour model corresponding to the industrial product data and the foreground target image of the continuous multi-frame to-be-sorted target industrial product image to obtain a contour similarity matching result;
Selecting and obtaining target industrial product images to be sorted from the continuous multi-frame target industrial product images to be sorted based on the contour similarity matching result;
performing target tracking processing on the target industrial products to be sorted based on the continuous multi-frame target industrial product images to be sorted, and obtaining tracking results of the target industrial products to be sorted;
the target tracking processing is performed on the target industrial product to be sorted based on the continuous multi-frame target industrial product image to be sorted, and a target industrial product tracking result to be sorted is obtained, including:
analyzing and processing the motion trail of the target industrial product to be sorted in the continuous multi-frame target industrial product image to be sorted by adopting a Kalman filtering algorithm to obtain a motion trail analysis result;
and carrying out target tracking processing on the target industrial products to be sorted based on the motion trail analysis result to obtain a target industrial product tracking result to be sorted.
7. The industrial product sorting control system based on cloud edge cooperation is characterized by comprising a sorting equipment end, an edge end and a cloud end, wherein the sorting equipment end is connected with the edge end, and the edge end is connected with the cloud end; the system is configured for performing the industrial product sort control method of any one of claims 1-5.
CN202310752435.0A 2023-06-26 2023-06-26 Industrial product sorting control method, device and system based on cloud edge cooperation Active CN116475081B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310752435.0A CN116475081B (en) 2023-06-26 2023-06-26 Industrial product sorting control method, device and system based on cloud edge cooperation

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310752435.0A CN116475081B (en) 2023-06-26 2023-06-26 Industrial product sorting control method, device and system based on cloud edge cooperation

Publications (2)

Publication Number Publication Date
CN116475081A CN116475081A (en) 2023-07-25
CN116475081B true CN116475081B (en) 2023-08-15

Family

ID=87218183

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310752435.0A Active CN116475081B (en) 2023-06-26 2023-06-26 Industrial product sorting control method, device and system based on cloud edge cooperation

Country Status (1)

Country Link
CN (1) CN116475081B (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103903278A (en) * 2012-12-28 2014-07-02 重庆凯泽科技有限公司 Moving target detection and tracking system
CN109894383A (en) * 2019-02-21 2019-06-18 南方科技大学 A kind of article sorting method, device, storage medium and electronic equipment
CN110841927A (en) * 2019-11-15 2020-02-28 上海威士顿信息技术股份有限公司 Sorting device, system and method and electronic equipment
EP3838427A1 (en) * 2019-12-20 2021-06-23 IHP Systems A/S A method for sorting objects travelling on a conveyor belt
CN114925387A (en) * 2022-04-02 2022-08-19 北方工业大学 Sorting system and method based on end edge cloud architecture and readable storage medium
CN115447924A (en) * 2022-09-05 2022-12-09 广东交通职业技术学院 Machine vision-based garbage classification and sorting method, system, device and medium

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103903278A (en) * 2012-12-28 2014-07-02 重庆凯泽科技有限公司 Moving target detection and tracking system
CN109894383A (en) * 2019-02-21 2019-06-18 南方科技大学 A kind of article sorting method, device, storage medium and electronic equipment
CN110841927A (en) * 2019-11-15 2020-02-28 上海威士顿信息技术股份有限公司 Sorting device, system and method and electronic equipment
EP3838427A1 (en) * 2019-12-20 2021-06-23 IHP Systems A/S A method for sorting objects travelling on a conveyor belt
CN114925387A (en) * 2022-04-02 2022-08-19 北方工业大学 Sorting system and method based on end edge cloud architecture and readable storage medium
CN115447924A (en) * 2022-09-05 2022-12-09 广东交通职业技术学院 Machine vision-based garbage classification and sorting method, system, device and medium

Also Published As

Publication number Publication date
CN116475081A (en) 2023-07-25

Similar Documents

Publication Publication Date Title
CN112966772A (en) Multi-person online image semi-automatic labeling method and system
CN113632099A (en) Distributed product defect analysis system, method and computer readable storage medium
CN110610193A (en) Method and device for processing labeled data
CN111260073A (en) Data processing method, device and computer readable storage medium
US20220366244A1 (en) Modeling Human Behavior in Work Environment Using Neural Networks
CN111709371A (en) Artificial intelligence based classification method, device, server and storage medium
KR20210050168A (en) Method For Applying Learning Data Augmentaion To Deep Learning Model, Apparatus And Method For Classifying Images Using Deep Learning
CN112527676A (en) Model automation test method, device and storage medium
CN114494168A (en) Model determination, image recognition and industrial quality inspection method, equipment and storage medium
CN114863554A (en) Sign language recognition system and method based on deep learning model
CN110473530A (en) Instruction classification method, apparatus, electronic equipment and computer readable storage medium
CN114338423A (en) Continuous iterative automatic network target construction technology
CN116475081B (en) Industrial product sorting control method, device and system based on cloud edge cooperation
CN117235527A (en) End-to-end containerized big data model construction method, device, equipment and medium
CN110555384A (en) Beef marbling automatic grading system and method based on image data
CN110716778A (en) Application compatibility testing method, device and system
CN115543820A (en) Tesseract-orc-based software automated testing method and system
CN113628077A (en) Method for generating non-repeated examination questions, terminal and readable storage medium
US20220091576A1 (en) Detection of defect in edge device manufacturing by artificial intelligence
CN115035044A (en) Be applied to intelligent AI platform of industry quality inspection
CN111160330B (en) Training method for improving image recognition accuracy with assistance of electronic tag recognition
KR20220161601A (en) System for determining defect of image inspection target using deep learning model
CN113392867A (en) Image identification method and device, computer equipment and storage medium
Zheng et al. Interface Identification of Automatic Verification System Based on Deep Learning
Walia Application of machine learning for GUI test automation

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant