CN113963280B - Identification method and device for intelligent detection and judgment of material and part and storage medium - Google Patents

Abstract

The application relates to an identification method and device for intelligent detection and judgment of a material part and a storage medium. The method comprises the following steps: obtaining an image dataset; identifying the material carrier in the image data set through the material carrier detection model and obtaining a mark of the material carrier, and identifying the material carrying operation area in the image data set through the material carrying operation area detection model and obtaining a mark of the material carrying operation area; determining a motion rule of the material part carrier relative to the material part carrying operation area according to the mark of the material part carrier and the mark of the material part carrying operation area, and determining a plurality of operation cycles corresponding to the operation flow of the material part carrier according to the motion rule; obtaining a plurality of significance maps from the image dataset according to a plurality of job cycles; and determining at least one piece of associated information of the material set to be conveyed corresponding to the operation flow of the material conveyor according to the effective graphs. Thus, the production efficiency is improved, the cost is reduced, and the operation safety is improved.

Description

Identification method and device for intelligent detection and judgment of material and part and storage medium

Technical Field

The application relates to the technical field of computer vision, in particular to an identification method and device for intelligent detection and judgment of a material part and a storage medium.

Background

With the development of artificial intelligence technology, the deep learning technology has made a great development in the field of computer vision technology, and made a great breakthrough in the aspects of image classification, image target detection, image segmentation, and the like. The face recognition product based on the computer vision technology is widely applied to places such as entry and exit ports, railway stations, airport halls and the like, and the purpose of identity detection and judgment is achieved by extracting face features from collected images, comparing and searching. In the industrial application field, such as automatic cargo sorting and port automation in logistics centers, etc., intelligent automatic detection and judgment of target cargos can be realized by means of artificial intelligence technology and products based on computer vision technology, and corresponding operations of carrying, sorting, packaging, etc. can be adopted according to detection and judgment results. In addition, in the waste steel recycling link, various waste steel products with complex sources, various types and large material difference need to be graded and corresponding operations are adopted, so that intelligent automatic detection and judgment of the waste steel products can be realized by means of an artificial intelligence technology and a product based on a computer vision technology. Compared with the traditional manual measurement and manual detection, the intelligent automatic detection and judgment of the target goods or the waste steel has the advantages of objective and stable detection and judgment standard, high informatization degree, reduction of potential safety hazards, labor cost and the like, and is favorable for improving the production efficiency and the operation safety.

However, the existing computer vision technology for face recognition cannot be conveniently used in the industrial application field because of problems such as stacking and shielding of goods or waste steel materials in practical applications. Therefore, an identification method, an identification device and a storage medium for intelligent detection and judgment of a material part are needed, and intelligent automatic detection and judgment of goods, waste steel and the like can be realized based on a computer vision technology, so that a basis is provided for decision making and subsequent processing, and the method, the device and the storage medium are beneficial to improving the production efficiency, reducing the cost and improving the operation safety.

Disclosure of Invention

In a first aspect, an embodiment of the present application provides an identification method, which is used for intelligent detection and judgment of a material part. The identification method comprises the following steps: obtaining an image dataset; identifying a material carrier in the image dataset and obtaining a mark of the material carrier through a material carrier detection model, and identifying a material carrying operation area in the image dataset and obtaining a mark of the material carrying operation area through a material carrying operation area detection model; determining a motion rule of the material part carrier relative to the material part carrying operation area according to the mark of the material part carrier and the mark of the material part carrying operation area, and determining a plurality of operation cycles corresponding to an operation flow of the material part carrier according to the motion rule, wherein the operation flow of the material part carrier is associated with the material part carrying operation area; obtaining a plurality of effective graphs from the image data set according to the plurality of work periods, wherein the plurality of effective graphs correspond to the plurality of work periods in a one-to-one mode; and determining at least one piece of associated information of the material set to be conveyed corresponding to the operation flow of the material conveyor according to the effective graphs.

According to the technical scheme described in the first aspect, the movement law of the material handling device relative to the material handling operation area is determined, and then the plurality of operation periods are determined according to the movement law, so that the simplification of the processing flow and the calculation complexity of degradation are facilitated, the information of the originally shielded or covered material is obtained according to the plurality of effective graphs, the problem of difficulty in identification caused by superposition and shielding among the materials of the material set to be handled is solved, the intelligent automatic detection and judgment of the materials such as goods and waste steel products based on a computer vision technology is facilitated, a basis is provided for decision making and subsequent processing, and the purposes of improving the production efficiency, reducing the cost and improving the operation safety are achieved.

According to a possible implementation manner of the technical solution of the first aspect, an embodiment of the present application further provides that the at least one piece of associated information of the set of materials to be handled includes at least one of: contour information, category information, source information, coordinate information, area information, pixel feature information.

According to a possible implementation manner of the technical solution of the first aspect, an embodiment of the present application further provides that determining, according to the plurality of effective graphs, at least one piece of associated information of a set of material to be handled, which corresponds to a workflow of the material handler, includes: for each effective graph of the effective graphs, determining a change area of the effective graph by comparing the effective graph with an effective graph corresponding to a previous working cycle relative to the working cycle corresponding to the effective graph, and determining a material part segmentation identification result corresponding to the change area of the effective graph through a material part segmentation model; and determining at least one piece of associated information of the material set to be conveyed according to the material segmentation identification result corresponding to the change area of each effective graph.

According to a possible implementation manner of the technical solution of the first aspect, an embodiment of the present application further provides that determining, according to the plurality of effective graphs, at least one piece of associated information of a set of material to be handled, which corresponds to a workflow of the material handler, includes: respectively carrying out material part segmentation recognition on the plurality of effective graphs through a material part segmentation model so as to obtain a plurality of material part segmentation graphs, wherein the plurality of material part segmentation graphs correspond to the plurality of effective graphs one by one; and determining at least one piece of associated information of the material set to be carried according to the plurality of material segmentation maps.

According to a possible implementation manner of the technical solution of the first aspect, an embodiment of the present application further provides that the performing, by the material segmentation model, material segmentation and identification on the plurality of effective graphs to obtain the plurality of material segmentation graphs includes: and for each effective graph of the plurality of effective graphs, firstly, carrying out area limitation on the effective graph according to the mark of the material conveying operation area to obtain an effective graph with area limitation, then carrying out material segmentation identification on the effective graph with area limitation through the material segmentation model to obtain a material segmentation identification result of the effective graph with area limitation, and thus obtaining a material segmentation graph corresponding to the effective graph.

According to a possible implementation manner of the technical solution of the first aspect, an embodiment of the present application further provides that the performing, by the material segmentation model, material segmentation and identification on the plurality of effective graphs to obtain the plurality of material segmentation graphs includes: and aiming at each effective graph of the plurality of effective graphs, firstly, performing material part segmentation recognition on the effective graph through the material part segmentation model to obtain a material part segmentation recognition result of the effective graph, and then performing area limitation on the material part segmentation recognition result of the effective graph according to the mark of the material part carrying operation area, so as to obtain the material part segmentation graph corresponding to the effective graph.

According to a possible implementation manner of the technical solution of the first aspect, an embodiment of the present application further provides that determining, according to the multiple material segmentation maps, at least one piece of associated information of the material set to be handled includes: and comparing the part segmentation drawing with a part segmentation drawing corresponding to a previous work cycle of the work cycle corresponding to the part segmentation drawing in the part segmentation drawing through a part duplication removal model, and performing part duplication removal operation on each part segmentation drawing of the part segmentation drawings, so as to obtain at least one type of associated information of the part set to be carried.

According to a possible implementation manner of the technical solution of the first aspect, an embodiment of the present application further provides that the at least one piece of associated information of the set of materials to be handled includes category information, and the identification method further includes: and inputting the type information of the material set to be conveyed into a material weight estimation model so as to obtain specific gravity information corresponding to the materials of different types of information of the material set to be conveyed.

According to a possible implementation manner of the technical solution of the first aspect, an embodiment of the present application further provides that the at least one piece of associated information of the set of materials to be handled includes category information, and the identification method further includes: and inputting the type information of the material set to be conveyed into a material quality identification model, thereby obtaining the quality information of the material set to be conveyed.

According to a possible implementation manner of the technical solution of the first aspect, an embodiment of the present application further provides that the identification method further includes: and determining the overall price of the material set to be carried according to the quality information of the material set to be carried.

According to a possible implementation manner of the technical solution of the first aspect, an embodiment of the present application further provides that, during the operation process of the material handling device, at least one material is added to the material handling operation area and becomes a part of a set of materials to be handled corresponding to the operation process of the material handling device.

According to a possible implementation manner of the technical solution of the first aspect, an embodiment of the present application further provides that the identification method further includes: and respectively carrying out ultra-long material identification on the effective graphs through an ultra-long material model and giving out an alarm after the ultra-long material is identified.

According to a possible implementation manner of the technical solution of the first aspect, an embodiment of the present application further provides that the identification method further includes: and respectively carrying out seal identification on the plurality of effective graphs through seal models and giving out a warning after the seal is identified.

According to a possible implementation manner of the technical solution of the first aspect, the embodiment of the present application further provides that the identification method is used for automatic identification of the scrap steel material, the material handler is a suction cup, a gripper or a scrap steel material handler for handling the scrap steel material,

the material handling operation area is a carriage of a vehicle for loading the steel scrap, and the operation flow of the material handler is that the suction cups, the grippers or the steel scrap handler for handling the steel scrap carry all the steel scrap on the carriage away from the carriage.

According to a possible implementation manner of the technical scheme of the first aspect, the embodiment of the application further provides that the material carrier is a suction cup, the material carrier detection model is a suction cup detection model, the material carrying operation area detection model is a carriage detection model, and the motion rule is that the suction cup is relative to the motion rule of the carriage.

According to a possible implementation manner of the technical solution of the first aspect, an embodiment of the present application further provides that the identification method is used for automatically identifying the goods to be transported, the material handler is a suction cup, a grab handle or a cargo handler used for transporting the goods, the material transporting operation area is a cargo accommodating area of a carrier used for loading the goods or a designated area used for stacking the goods, and the operation flow of the material handler is that the suction cup, the grab handle or the cargo handler used for transporting the goods transports all the goods in the cargo accommodating area or the designated area away from the cargo accommodating area or the designated area.

According to a possible implementation manner of the technical solution of the first aspect, an embodiment of the present application further provides that the material handling device detection model is a target detection model, and the material handling operation area detection model is an image semantic segmentation model.

According to a possible implementation manner of the technical solution of the first aspect, an embodiment of the present application further provides that the marking of the material handling device is a detection frame of the material handling device, and the marking of the material handling operation area is a detection frame of the material handling operation area, wherein determining a motion rule of the material handling device relative to the material handling operation area according to the detection frame of the material handling device and the detection frame of the material handling operation area includes: and determining the movement rule of the material part carrier entering and leaving the material part carrying operation area according to the change of the overlapping area between the detection frame of the material part carrier and the detection frame of the material part carrying operation area.

According to a possible implementation manner of the technical solution of the first aspect, an embodiment of the present application further provides that the image data set is obtained by sampling or capturing an image of a video data stream according to a preset sampling frequency.

According to a possible implementation manner of the technical solution of the first aspect, an embodiment of the present application further provides that the image data set is subjected to a data enhancement operation, and the data enhancement operation includes at least one of: random inversion, rotation, inversion and rotation, random transformation, random scaling, random clipping, fuzzification, Gaussian noise addition and filling.

In a second aspect, embodiments of the present application provide a non-transitory computer-readable storage medium. The computer readable storage medium stores computer instructions which, when executed by a processor, implement the identification method according to any one of the first aspect.

According to the technical scheme described in the second aspect, the movement law of the material handling device relative to the material handling operation area is determined, and then a plurality of operation periods are determined according to the movement law, so that the processing flow and the degradation calculation complexity are simplified, the information of the originally shielded or covered material is obtained according to a plurality of effective graphs, the problem of difficulty in identification caused by superposition and shielding of the materials of the material set to be handled is solved, the intelligent automatic detection and judgment of the materials such as goods and waste steel products based on a computer vision technology is facilitated, a basis is provided for decision making and subsequent processing, and the purposes of improving the production efficiency, reducing the cost and improving the operation safety are achieved.

In a third aspect, an embodiment of the present application provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor implements the identification method according to any one of the first aspect by executing the executable instructions.

According to the technical scheme described in the third aspect, the movement law of the material part carrier relative to the material part carrying operation area is determined, and then a plurality of operation periods are determined according to the movement law, so that the processing flow and the degradation calculation complexity are facilitated to be simplified, the information of the originally shielded or covered material parts is obtained according to a plurality of effective graphs, the problem of difficulty in identification caused by superposition and shielding of the material parts of the material part set to be carried is solved, the intelligent automatic detection and judgment of the material parts such as goods and waste steel products based on the computer vision technology is facilitated to be realized, a basis is provided for decision making and subsequent processing, and the purposes of improving the production efficiency, reducing the cost and improving the operation safety are achieved.

In a fourth aspect, an embodiment of the present application provides an identification apparatus, which is used for intelligently checking and judging a material. The identification device comprises: a part handler detection model for identifying part handlers in the image dataset and obtaining markings of the part handlers; a material handling operation area detection model for identifying the material handling operation area in the image data set and obtaining a mark of the material handling operation area; the operation cycle algorithm generation model is used for determining a motion rule of the material part carrier relative to the material part carrying operation area according to the mark of the material part carrier and the mark of the material part carrying operation area, and determining a plurality of operation cycles corresponding to an operation process of the material part carrier according to the motion rule, wherein the operation process of the material part carrier is associated with the material part carrying operation area; an effective graph generation model for obtaining a plurality of effective graphs from the image data set according to the plurality of work periods, wherein the plurality of effective graphs correspond to the plurality of work periods one by one; and the part identification network branch is used for determining at least one type of associated information of a part set to be carried corresponding to the operation flow of the part carrier according to the effective graphs.

According to the technical scheme described in the fourth aspect, the movement law of the material handling device relative to the material handling operation area is determined, and then a plurality of operation periods are determined according to the movement law, so that the simplification of the processing flow and the calculation complexity of degradation are facilitated, the information of the originally shielded or covered material is obtained according to a plurality of effective graphs, the problem of difficulty in identification caused by superposition and shielding among the materials of the material set to be handled is solved, the intelligent automatic detection and judgment of the materials such as goods and waste steel products based on the computer vision technology is facilitated, a basis is provided for decision making and subsequent processing, and the purposes of improving the production efficiency, reducing the cost and improving the operation safety are achieved.

According to a possible implementation manner of the technical solution of the fourth aspect, an embodiment of the present application further provides that the at least one piece of associated information of the set of materials to be handled includes at least one of: contour information, category information, source information, coordinate information, area information, pixel feature information.

According to a possible implementation manner of the technical solution of the fourth aspect, an embodiment of the present application further provides that the material part identification network branch includes a material part segmentation model, and determines, according to the plurality of effective graphs, at least one type of associated information of a material part set to be carried, which corresponds to a workflow of the material part carrier, including: respectively carrying out material part segmentation recognition on the plurality of effective graphs through the material part segmentation model so as to obtain a plurality of material part segmentation graphs, wherein the plurality of material part segmentation graphs correspond to the plurality of effective graphs one by one; and determining at least one piece of associated information of the material set to be carried according to the plurality of material segmentation maps.

According to a possible implementation manner of the technical solution of the fourth aspect, an embodiment of the present application further provides that the material identification network branch includes a material deduplication model, and the determining, according to the multiple material segmentation maps, at least one piece of associated information of the material set to be handled includes: and for each material part segmentation drawing of the plurality of material part segmentation drawings, comparing the material part segmentation drawing with a material part segmentation drawing corresponding to the previous work cycle of the work cycle corresponding to the material part segmentation drawing in the plurality of material part segmentation drawings through the material part duplication removal model, and performing material part duplication removal operation, so as to obtain the type information of the material part set to be conveyed.

According to a possible implementation manner of the technical solution of the fourth aspect, an embodiment of the present application further provides that the material identification network branch includes an ultra-long material model and a sealing element model, where the ultra-long material model is configured to perform ultra-long material identification on the multiple effective graphs respectively and issue an alarm after the ultra-long material is identified, and the sealing element model is configured to perform sealing element identification on the multiple effective graphs respectively and issue an alarm after the sealing element is identified.

According to a possible implementation manner of the technical solution of the fourth aspect, an embodiment of the present application further provides that the identification device is used for automatically identifying the steel scrap, the material carrier is a suction cup, a gripper or a steel scrap carrier for carrying the steel scrap, the material carrying operation area is a carriage of a vehicle for loading the steel scrap, and the operation flow of the material carrier is that the suction cup, the gripper or the steel scrap carrier for carrying the steel scrap carries all the steel scrap on the carriage away from the carriage.

Drawings

In order to explain the technical solutions in the embodiments or background art of the present application, the drawings used in the embodiments or background art of the present application will be described below.

Fig. 1 shows an application scenario of intelligent automatic detection and judgment of waste steel provided by the embodiment of the present application.

Fig. 2 shows a schematic flow chart of an identification method for intelligent detection and judgment of a material part, provided by the embodiment of the application.

Fig. 3 shows a block diagram of an electronic device used in the identification method shown in fig. 2 according to an embodiment of the present application.

Fig. 4 shows a block diagram of an identification apparatus for intelligent detection and judgment of a material part according to an embodiment of the present application.

Detailed Description

The embodiment of the application provides an identification method and device for intelligent detection and judgment of materials and parts and a storage medium, in order to solve the technical problem of how to realize intelligent automatic detection and judgment of goods, waste steel and the like based on a computer vision technology. The identification method comprises the following steps: obtaining an image dataset; identifying a material carrier in the image dataset and obtaining a mark of the material carrier through a material carrier detection model, and identifying a material carrying operation area in the image dataset and obtaining a mark of the material carrying operation area through a material carrying operation area detection model; determining a motion rule of the material part carrier relative to the material part carrying operation area according to the mark of the material part carrier and the mark of the material part carrying operation area, and determining a plurality of operation cycles corresponding to an operation flow of the material part carrier according to the motion rule, wherein the operation flow of the material part carrier is associated with the material part carrying operation area; obtaining a plurality of effective graphs from the image data set according to the plurality of work periods, wherein the plurality of effective graphs correspond to the plurality of work periods in a one-to-one mode; and determining at least one piece of associated information of the material set to be conveyed corresponding to the operation flow of the material conveyor according to the effective graphs. Therefore, the movement law of the material part carrier relative to the material part carrying operation area is determined, a plurality of operation periods are determined according to the movement law, the simplification of the processing flow and the calculation complexity of degradation are facilitated, the information of the originally shielded or covered material parts is obtained according to a plurality of effective graphs, the problem of difficulty in identification caused by superposition and shielding of the material parts of the material part set to be carried is solved, the intelligent automatic detection and judgment of the material parts such as goods and waste steel products based on a computer vision technology is facilitated, a basis is provided for decision making and subsequent processing, and the purposes of improving the production efficiency, reducing the cost and improving the operation safety are achieved.

The embodiment of the application can be applied to the following application scenes, including but not limited to, industrial automation, goods sorting in logistics centers, port automation, intelligent automatic goods inspection and judgment, waste steel recovery, intelligent automatic waste steel inspection and judgment, and any application scenes, such as coal automatic sorting, garbage recovery, garbage automatic sorting and the like, which can improve the production efficiency and reduce the labor cost through the identification method and device for intelligent material inspection and judgment.

The embodiments of the present application may be modified and improved according to specific application environments, and are not limited herein.

In order to make the technical field of the present application better understand, embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.

Aspects of the present application and various embodiments and implementations mentioned below relate to concepts of artificial intelligence, machine learning, and neural networks. In general, Artificial Intelligence (AI) studies the nature of human Intelligence and builds intelligent machines that can react in a manner similar to human Intelligence. Research in the field of artificial intelligence applications includes robotics, speech recognition, natural language processing, image recognition, decision reasoning, human-computer interaction, expert systems, and the like. Machine Learning (ML) studies how artificial intelligence systems model or implement human Learning behavior, acquire new knowledge or skills, reorganize existing knowledge structures, and improve self-competency. Machine learning learns rules from a large number of samples, data, or experiences through various algorithms to identify new samples or to make decisions and predictions about events. Examples of machine learning algorithms include decision tree learning, bayesian classification, support vector machines, clustering algorithms, and the like. Deep Learning (DL) refers to the natural Deep structures of the human brain and cognitive processes graded by depth, studies how to input large amounts of data into complex models, and "trains" the models to learn how to grab features. Neural Networks (NN) can be divided into Artificial Neural Networks (ANN) and Spiking Neural Networks (SNN). The SNN simulates a pulse neuron model of a biological nerve working mechanism, and pulse coding information is adopted in the calculation process. Currently, ANN is widely used. The neural network NN referred to herein generally refers to an artificial neural network, i.e., an ANN, unless specified otherwise or indicated otherwise or a different interpretation is made in conjunction with the context.

The ANN refers to an algorithmic mathematical model which is established by the inspiration of a brain neuron structure and a nerve conduction principle, and has a network structure which imitates animal neural network behavior characteristics to process information. Neural networks comprise a large number of interconnected nodes or neurons, sometimes referred to as artificial neurons or perceptrons, which are inspired by the structure of neurons in the brain. The Shallow Neural Network (shadow Neural Network) only comprises an input layer and an output layer, wherein the input layer is responsible for receiving input signals, and the output layer is responsible for outputting calculation results of the Neural Network. After the input signals are linearly combined, an Activation Function (Activation Function) is applied to the input signals for transformation to obtain a result of an output layer. The complex model used in Deep learning is mainly a multi-layer Neural Network, sometimes referred to as Deep Neural Network (DNN). The multi-layer neural network includes hidden layers in addition to an input layer and an output layer, each hidden layer includes an arbitrary number of neurons which are connected as nodes with a node of a previous layer in a network structure, and each neuron can be regarded as a linear combiner and assigns a weight to each connected input value for weighted linear combination. The activation function is a nonlinear mapping after weighted linear combination of input signals, which in a multilayer neural network can be understood as a functional relationship between the output of a neuron in a previous layer and the input of a neuron in a next layer. Each hidden layer may have a different activation function. Common activation functions are ReLU, Sigmoid, Tanh, etc. The neural network passes the information of each layer to the next layer through the mesh structure. The forward propagation is a process of calculating layer by layer from an input layer to an output layer, the weighted linear combination and the transformation are repeatedly carried out in the forward propagation process, and finally, a Loss Function (Loss Function) is calculated and used for measuring the deviation degree between the predicted value and the true value of the model. The back propagation is to propagate from the output layer to the hidden layer to the input layer, and the neural network parameters are corrected according to the error between the actual output and the expected output in the back propagation process. DNN can be classified into Convolutional Neural Network (CNN), Fully Connected Neural Network (FCN), and Recurrent Neural Network (RNN) according to the composition of a base layer. The CNN is composed of a convolutional layer, a pooling layer and a full-link layer. The FCN consists of multiple fully connected layers. The RNN consists of fully connected layers but with feedback paths and gating operations between layers, also called recursive layers. Different types of neural network base layers have different computational characteristics and computational requirements, for example, the computation proportion of convolutional layers in some neural networks is high and the computation amount of each convolutional layer is large. In addition, the calculation parameters of each convolution layer of the neural network, such as the convolution kernel size and the input/output characteristic diagram size, vary widely.

Fig. 1 shows an application scenario of intelligent automatic detection and judgment of waste steel provided by the embodiment of the present application. As shown in fig. 1, an application scenario of intelligent automatic detection and judgment of waste steel products, hereinafter referred to as an automatic detection and judgment of waste steel, includes a suction cup 110 and a carriage 120. In a scrap recycling process, various scrap steels to be recycled are carried to a certain place by a carrier such as a vehicle, and then all the scrap steels on the vehicle are unloaded or transported to other places by a device such as an electromagnet chuck. For the sake of distinction, the individual pieces of scrap steel are referred to as scrap pieces, each scrap piece being understood to mean an individual piece of scrap steel which can be handled either individually or separately from the other scrap pieces. Various waste steel products to be recycled are generally a collection of a plurality of waste steel parts, which may be called a collection of waste steel parts. In rare cases, the collection of scrap pieces may comprise only one scrap piece or only one piece of scrap steel to be recycled, for example a vehicle for carrying scrap steel may be loaded with only one individual piece of scrap steel, such as a large bearing piece recovered from a train or a single piece of aircraft outer cabin portion recovered from an aircraft. In order to realize the intelligent automatic detection and judgment of the waste steel, each waste steel part to be recycled needs to be distinguished, and basic information such as the number, the size, the type and the like of the waste steel parts included in a waste steel part set needs to be identified. The dimensions of the scrap pieces generally include at least the thickness of the scrap pieces, for example, the different scrap pieces are classified by their respective thicknesses into dimensions such as 3 mm, 4 mm, 6 mm, 8 mm and 10 mm. The type of scrap generally indicates the source of the scrap, for example from the wheel hub of a train or from a building. By obtaining the basic information of the number, the size, the type and the like of the scrap pieces included in the scrap piece set, a basis can be provided for subsequent decision and processing, for example, the specific gravity, the price and the like of the scrap piece set can be estimated. And moreover, the intelligent automatic detection and judgment requirement also comprises the automatic detection and judgment in the whole process of finishing the transportation of all the scrap steel parts in the scrap steel part set. Taking the application scenario of automatic inspection and judgment of scrap steel shown in fig. 1 as an example, the intelligent automatic inspection and judgment of scrap steel requires that all scrap steel pieces to be recovered, namely scrap steel pieces loaded on the carriage 120, are automatically inspected and judged in the whole process of completing the transportation of the scrap steel pieces set by the suction cup 110.

Under the application scenario of intelligent automatic detection and judgment of waste steel, for example, the application scenario of automatic detection and judgment of waste steel shown in fig. 1, various technical problems are faced. In the first aspect, it is difficult for a vehicle in charge of transporting scrap steel parts to be precisely parked in a designated vehicle parking area and the orientation of the body of the vehicle after the parking is difficult to determine. As shown in FIG. 1, the car 120 represents the outer contour or car area of a car loaded with scrap steel on a vehicle responsible for transporting the scrap steel, and the vehicle parking area 122 represents a designated vehicle parking area. Ideally, the car 120 is parked just to the vehicle parking area 122. In addition, an intelligent automatic inspection method based on computer vision technology often acquires a video or a captured image by a monitoring device, a camera, a closed circuit television or other capturing devices, and these capturing devices for acquiring an image or a video are generally disposed at places having fixed positions and orientations, for example, at higher altitudes such as on a utility pole so as to capture an image or a video with the vehicle parking area 122 as a reference in a bird's eye view or in an overhead view. The actual position and body orientation of the parked vehicle are difficult to determine relative to the collection devices disposed in fixed positions and orientations, and it is not possible to ensure that the actually parked vehicle meets the pre-planned parking area, as shown in fig. 1, in which only a portion of the car 120 overlaps the parking area 122, or in which the car 120 does not overlap the parking area 122. This makes it unreliable to capture the image or video with the vehicle parking area 122 as a reference in the image or video captured by the capturing device, and makes it difficult to predict the size, orientation, etc. of the body of the vehicle loaded with scrap pieces, and it is necessary to adjust the capturing device to be able to capture the image of the car 120 in conjunction with the related control technology, if necessary. In the second aspect, the condition of the scrap pieces piled on the carriage carrying the scrap pieces is difficult to predict, and there may be a case where a part of the scrap pieces goes beyond the outer contour of the carriage or the edge of the carriage. As shown in fig. 1, the scrap pieces loaded on the car 120 include a scrap piece to be carried 130, a scrap piece to be carried 132, and a scrap piece to be carried 134. Wherein the scrap pieces 132 to be handled extend beyond the outer contour of the car 120. The scrap pieces 132 to be handled may be long strips of scrap steel having a length exceeding the side length of the car 120 so as to extend beyond the edge of the car 120 when loaded on the car 120. Therefore, in the image or video acquired by the acquisition equipment, the distribution of the scrap steel part set to be recycled, namely the scrap steel part set to be identified, may not be completely in the carriage loaded with the scrap steel parts or the extension of the scrap steel part set to be recycled is not consistent with the outer contour of the carriage. In a third aspect, various interference factors exist in the automatic intelligent detection and judgment process, for example, other vehicles which are also loaded with the waste steel parts may exist near the position where the vehicle loaded with the waste steel parts is parked, other stacked waste steel parts may exist near a carriage, and moving individuals such as workers, animals and the like may also exist. As shown in fig. 1, the scrap loaded on the car 120 includes a scrap to be carried 130, a scrap to be carried 132, and a scrap to be carried 134, which belong to a collection of scrap that needs to be recovered and automatically checked. However, there are also nearby scrap pieces 140 and 142, which are interference factors and interference from nearby scrap pieces 140 and 142 should be excluded when performing automatic scrap detection. It should be noted that the nearby scrap piece 142 is not located in the area of the car represented by the car 120, that is, the nearby scrap piece 142 is not loaded in the car, but the nearby scrap piece 142 is located in the designated parking area represented by the parking area 122. In a fourth aspect, the scrap pieces are generally stacked one on top of another in the vehicle cabin, and the scrap pieces located below are generally hidden by the scrap pieces located above, so that in the image or video acquired by the acquisition device, it is difficult to see the scrap pieces below or only a part of the scrap pieces below can be seen due to the hiding. Only the lower scrap pieces are exposed as the upper scrap pieces are removed one by one. In the fifth aspect, the automatic detection and judgment of the scrap steel based on the computer vision technology generally needs to extract relevant image features from an image or a video acquired by the acquisition device, but in practical application, the composition, the number and the stacking condition of the scrap steel parts included in the scrap steel part set needing to be identified are unpredictable, the shielding, the layer-by-layer stacking and the like among the scrap steel parts also bring great challenges to accurate identification, and the condition of the scrap steel parts adsorbed and taken away by the suction cups 110 each time is also unpredictable. Therefore, high requirements are put forward on the detection precision, the detection process is also relatively complex, and high requirements are put forward on the calculation performance and the storage resources.

In summary, in an application scenario of intelligent automatic detection and judgment of waste steel products, for example, the application scenario of automatic detection and judgment of waste steel products shown in fig. 1, an image or a video acquired by an acquisition device is processed based on a computer vision technology and through a machine learning model, so that various technical difficulties are faced, for example, a parking position, a size and an orientation of a vehicle body of a vehicle loaded with a set of waste steel products to be identified are difficult to predict, a stacking condition of the waste steel products on the vehicle body is difficult to predict and may extend beyond an outer contour of the vehicle body, interference is brought by nearby vehicles and nearby waste steel products, identification difficulty is caused by stacking and shielding of the waste steel products, and requirements in terms of high detection accuracy and complex detection process are brought by various uncertainties in the whole operation process.

It should be understood that the suction cups 110 shown in fig. 1 may also be replaced by grippers or any suitable scrap handler for handling scrap. Also, the suction cup 110 may be an electro-magnet suction cup or any suitable device having a suction effect that may be used to carry scrap pieces. The carriage 120 represents the outer contour or carriage area of a carriage carrying scrap pieces on a vehicle responsible for transporting the scrap pieces. The vehicle carrying the scrap may be replaced by any suitable vehicle such as a barge or tricycle or any suitable vehicle, and the compartment 120 may be more broadly understood as the area on the vehicle carrying the scrap for carrying the scrap. Therefore, the application scenario of automatic scrap inspection and judgment shown in fig. 1 can be more generally understood as that, in the process that the suction cup 110 or other scrap handlers complete the handling of the to-be-handled scrap assembly loaded on the carriage 120 or other carriers, the intelligent automatic inspection and judgment of the scrap assembly is realized based on the computer vision technology, so as to provide a basis for decision making and subsequent processing, which is beneficial to improving the production efficiency, reducing the cost and improving the operation safety.

Furthermore, the application scenario of intelligent automatic detection and judgment of the waste steel and the similar application scenario face similar technical problems in the aspect of realizing intelligent automatic detection and judgment based on the computer vision technology. The similar application scenes comprise industrial automation, goods sorting in logistics centers, port automation, intelligent automatic goods checking and judging, waste steel recycling, intelligent automatic waste steel checking and judging, and any application scenes such as automatic coal sorting, garbage recycling, automatic garbage sorting and the like, wherein the application scenes can improve the production efficiency and reduce the labor cost through an identification method and a device for intelligent material checking and judging. In the fields of intelligent automatic inspection and similar application of waste steel, it is often necessary to transfer accumulated waste steel or goods from one place to another by using a suction cup or a similar device. The artificial intelligence technology based on computer vision can be used for monitoring the operation progress, checking and judging the operation quality, recording the cargo information and providing an analysis report in the operation process of carrying or unloading the cargo, and compared with the traditional method of completing the operations through manual work, the artificial intelligence technology can improve the automation level, reduce the labor cost and achieve objective standardization. For convenience of description, goods or waste steel materials, etc. stacked to be transferred are collectively referred to as "material pieces", and suction cups or similar devices for transferring material pieces are referred to as "material carrier". The parts are stacked at one place to be transported to another place by the parts carrier, so the parts carrier needs to transport all the parts at one time through one transport operation, or transport a part of the parts through multiple transport operations each time, and thus an operation area for the parts carrier to carry out the transport operation needs to be defined, and the operation area is called as a 'parts transport operation area'. The whole process that the material carrier carries all the materials to be carried in the material carrying operation area is the operation flow of the material carrier. Therefore, the application scene of the intelligent automatic detection and judgment of the waste steel products and the similar application scene are described as that the intelligent automatic detection and judgment of the material set to be carried is realized based on the computer vision technology in the operation process of the material carrier or in the process that the material carrier carries all the materials (namely the material set to be carried) to be carried on the material carrying operation area through one or more carrying operations, so that a basis is provided for decision making and subsequent processing. Therefore, the common technical problem faced by the application scenario of intelligent automatic detection and judgment of waste steel and the similar application scenario can also be described as follows: the parking position, the size and the direction of a vehicle loading the material set to be transported are difficult to predict, so that the material transporting operation area is difficult to predict, the stacking condition of the material set to be transported is difficult to predict and may extend beyond the outline of the vehicle, interference is brought by nearby vehicles and nearby materials, identification difficulty is caused by superposition and shielding of the material set to be transported, and high detection precision and complex detection process requirements are brought by various uncertainties in the whole operation process. With reference to fig. 2 and related embodiments and implementations, how to solve the above-mentioned common technical problem faced by the implementation of intelligent automatic inspection and judgment of a to-be-handled material set based on a computer vision technology in an application scenario of intelligent automatic inspection and judgment of waste steel and similar application scenarios will be described below.

Fig. 2 shows a schematic flow chart of an identification method for intelligent detection and judgment of a material part, provided by the embodiment of the application. The identification method is used for intelligently detecting and judging the material pieces, is suitable for waste steel recycling links, application scenes of intelligently and automatically detecting and judging the waste steel and similar application scenes such as industrial automation, logistics center cargo sorting, port automation, intelligent and automatically detecting and judging the cargo, and the like, and can also be suitable for any application scenes which can improve the production efficiency and reduce the labor cost through the identification method for intelligently detecting and judging the material pieces, such as coal automatic sorting, garbage recovery, automatic garbage sorting and the like. The above identification methods are applicable to various application scenarios involving operations of transporting objects stacked in one place to another place, and the stacked objects need to be accurately identified and analyzed to count relevant information of the stacked objects, such as how many kinds of the objects are, the respective number of each object, etc., that is, the stacked objects need to be checked. Compared with manual detection and judgment, the intelligent automatic detection and judgment can improve the production efficiency and ensure objective standardization. For convenience of description, the stacked objects to be transported or objects to be detected are called "material pieces", for example, the waste steel to be transported in the application scene of intelligent automatic detection and detection of waste steel is called waste steel material pieces; each material piece is a single individual, and can be carried independently or together with other material pieces, all the material pieces to be carried or all the material pieces needing to be checked are called a material piece set to be carried, and under a specific application scene, all the waste steel materials needing to be recycled are called a waste steel material piece set. In some cases, there may be only one part that needs to be handled, and the set of parts to be handled includes only one part. In addition, the apparatus for conveying the material is referred to as a "material carrier", and for example, a suction cup for sucking and transferring the waste steel material in an application scene of intelligent automatic inspection of the waste steel material is referred to as a material carrier. Depending on the requirements of the particular application scenario, the parts handler may be an electro-magnet chuck, a gripper or similar, for example in a port automation application scenario the goods to be handled are parts and the mechanical gripper is a parts handler. In addition, since the parts to be transported are stacked at one place and wait to be transported to another place by the parts transporter, the parts transporter transports all the parts at one time through one transporting operation, or transports a part of the parts through multiple transporting operations each time, so that it is necessary to define an operation area where the parts transporter transports, which is called a "parts transporting operation area". Depending on the specific application scenario, the material handling operation area may be associated with a specific vehicle, for example in the application scenario of intelligent automated inspection of waste steel material, the material handling operation area refers to the cabin area of the vehicle used to load the waste steel material, and for example in the application scenario of port automation, the material handling operation area may refer to the cabin area on the barge used to load the cargo to be handled. The operation flow of the material handling device refers to the whole process of the material handling device for handling all the materials to be handled in the material handling operation area, that is, the whole process of the material to be handled, which is completed by gathering and handling the materials, for example, the operation flow of the material handling device or the operation flow of the suction cup refers to the whole process of the suction cup for handling all the waste steel loaded on the carriage, that is, the whole process of the waste steel gathering and handling the materials. Moreover, these application scenarios face similar technical problems in realizing intelligent automatic detection and judgment based on a computer vision technology, for example, parking positions, sizes and orientations of vehicles loading a to-be-transported material assembly are difficult to predict, so that a material transport operation area is difficult to predict, stacking conditions of the to-be-transported material assembly are difficult to predict and may extend beyond the outline of the vehicles, interference is caused by nearby vehicles and nearby materials, identification is difficult due to overlapping and shielding of the to-be-transported materials of the to-be-transported material assembly, and high detection accuracy and complex detection process requirements are caused by various uncertainties in the whole operation process. How to overcome these technical problems will be described below with reference to the specific steps of the identification method shown in fig. 2.

Step S202: an image data set is obtained.

Wherein the image dataset is derived from a video or captured image captured by a monitoring device, a camera or a closed circuit television or like capture device. These capturing devices for obtaining images or videos are generally disposed in places having fixed positions and orientations, for example, at higher altitudes such as on utility poles so as to capture images or videos at an aerial or overhead angle. For videos, video streams or video data acquired by the acquisition device, corresponding images can be acquired by any suitable means such as sampling, screenshot, frame extraction and the like. For example, video data may be sampled, captured, or image frames may be decimated at a preset sampling frequency to obtain an image data set. In some embodiments, the image data sets come from the same acquisition device and the same acquisition device remains unchanged in position and orientation at least during recording of the entire workflow. In other embodiments, the image data sets are from different acquisition devices, or the image data sets are from the same acquisition device but the position and/or orientation of the same acquisition device has changed during the recording of the entire workflow. In some embodiments, the image data sets come from the same acquisition device and the same acquisition device maintains the same camera parameters, such as zoom scale, focal length, detection distance, etc., at least during recording of the entire workflow, which may affect the imaging process. In other embodiments, the image data sets are from the same capture device but the same capture device has made an adjustment to at least one factor or camera parameter that may affect the imaging process during the recording of the entire workflow.

Step S204: identifying a part handler in the image dataset and obtaining a marking of the part handler via a part handler detection model, and identifying a part handling work area in the image dataset and obtaining a marking of the part handling work area via a part handling work area detection model.

The part handler inspection model may be based on an object inspection technology or any suitable technology, such as yolov3 object inspection model, yolov4 object inspection model, and yolov5 object inspection model. The parts handler inspection model is used to identify the parts handler in the image dataset and to obtain the markings of the parts handler. The label of the part handler may be a prediction output by the part handler detection model that is representative of the part handler, such as a predicted frame or coordinates. The marking of the parts handler may also be any judgment made by the parts handler detection model based on the image dataset on the information of the position, size, extent, etc. of the parts handler in the image dataset, such as by feature extraction and neural network calculations. Taking the application scenario of automatic scrap inspection as shown in FIG. 1 as an example, the part handler inspection model can be used to identify the suction cups 110, i.e., the suction cups 110 used to handle the scrap pieces 132 to be handled. In addition, the image data set only represents information acquired by the acquisition device within a limited viewing angle, such as a Field of View (FOV) of an optical instrument, or the image data set only represents a limited range, so that there may be a case where the part handler to be identified is not included in the image input to the part handler detection model for detection. In this case, the parts handler detection model may generate a specific mark to indicate that no parts handler is present in the marked image. For example, the suction cup 110 shown in FIG. 1 may be located so far from the car 120 that only the car 120 but not the suction cup 110 is included in the captured image.

The material part conveying operation area detection model is used for identifying the material part conveying operation area in the image data set and obtaining a mark of the material part conveying operation area. The mark of the work area for conveying the material may be a prediction result, such as a prediction frame, representing the work area for conveying the material, which is output by the work area for conveying the material detection model. In order to identify a material conveying operation area, such as a carriage area represented by the carriage 120 shown in fig. 1, problems of vehicle body inclination, distance, material exceeding the edge of the vehicle body, and uncertainty of the vehicle body placement position and orientation generally need to be considered, so that an image segmentation technology, such as a deplab v3 image semantic segmentation model, can be used.

Referring to step S202 and step S204, the two steps are used to obtain an image data set and label the obtained image data set, so as to obtain a labeled image data set. In some embodiments, there may be image data sets that have been previously labeled, such that steps S202 and S204 may be skipped, i.e., the image data sets need not be acquired and labeled, but rather the subsequent steps are performed directly with the image data sets that have been labeled in advance. Wherein the labeling of the acquired image dataset is to obtain a mark of the material part carrier and a mark of the material part carrying operation area. In some embodiments, in order to reduce the amount of calculation, the image with higher similarity may be deleted by checking the similarity between the images in the image data set after the image data set is obtained at step S202, i.e. the image with higher similarity is removed, and then the remaining images are labeled. In other embodiments, all images in the image dataset obtained at step S202 may be annotated. In addition, the parts handler inspection model and the parts handling operation area inspection model are based on machine learning algorithms or machine learning models, such as ANN. The parts carrier detection model and the parts carrier working area detection model used in step S204 are machine learning models that have been trained. The training mode can be supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning or any suitable training mode, as long as the training mode is matched with a specifically adopted machine learning model. In addition, the labels of the data sets for training the parts carrier detection model and the parts transport operation area detection model may be identical to or slightly different from the marks of the parts carrier and the parts transport operation area output by the parts carrier detection model and the parts transport operation area detection model, respectively, and these labels may be adjusted according to the model architecture, model parameters, training method, and the like actually adopted. For example, the training process for the part handler detection model may be trained through three parts, namely a training set, a verification set and a test set, and the labels of the training data set may include, but are not limited to, the position, name, width, height, number of image channels, label category, labeling information, and the like of the image. The marks of the part handler output by the trained part handler detection model in step S204 include at least a result of predicting the position of the part handler, for example, in the form of a predicted frame of the part handler or coordinates of the part handler. In contrast, the training process of the material handling operation area detection model can be trained through a training set, a verification set and a test set, and the labels of the training data set can include, but are not limited to, the position, name, width, height, image channel number, label category, labeling information and the like of the image. The marks of the material handling operation area output by the trained material handling operation area detection model in step S204 at least include a prediction frame representing the material handling operation area, which is a prediction result of the occupied area or the periphery of the material handling operation area. It should be understood that the mark of the work area for conveying the material or the prediction result of the occupied area of the work area for conveying the material, which is output by the model for detecting the work area for conveying the material, does not necessarily coincide with the contour of the vehicle compartment of the vehicle, because it is necessary to take into account the possibility that the material exceeds the edge of the vehicle body, and therefore the material exceeding the edge of the vehicle body, that is, exceeding the contour of the vehicle compartment, is also included in the set of the materials to be conveyed. In some embodiments, in consideration of the problems of vehicle body inclination, distance, material exceeding the edge of the vehicle body, and vehicle body placement position and orientation uncertainty, the marking of the material handling operation area can be more accurately performed by additional vehicle body contour extraction modules or optimization algorithms, or by integrating these additional modules or algorithms into the material handling operation area detection model.

Step S206: determining a motion rule of the material part carrier relative to the material part carrying operation area according to the mark of the material part carrier and the mark of the material part carrying operation area, and determining a plurality of operation cycles corresponding to an operation process of the material part carrier according to the motion rule, wherein the operation process of the material part carrier is associated with the material part carrying operation area.

In step S204, the mark of the material part carrier and the mark of the material part carrying operation area are obtained, and the motion law of the material part carrier relative to the material part carrying operation area can be determined according to the mark of the material part carrier and the mark of the material part carrying operation area. The movement law of the material part carrier relative to the material part carrying operation area at least comprises the movement law of the material part carrier entering and leaving the material part carrying operation area. As mentioned above, the operation flow of the material carrier refers to the whole process of the material carrier transporting all the materials to be transported in the material transporting operation area, i.e. the materials to be transported together. The operation flow of the material handling device usually involves multiple handling operations, each handling operation requires the material handling device to enter the material handling operation area, then to adsorb or grab or separate a part of the material from other materials in any way, and then to carry away the separated part of the material, that is, the material handling device leaves the material handling operation area. In practice, there may be situations where the parts handler is out of order in a single handling operation, i.e. the parts handler fails to separate any parts in a single handling operation, but the parts handler still needs to enter and leave the parts handling work area. And once the material carrier enters the material carrying operation area, the material carrier can be considered to belong to the same carrying operation period before leaving the material carrying operation area, no matter how much time the material carrier spends leaving the material carrying operation area. In this way, by determining the law of motion of the parts handler entering and leaving the parts handling work area, it is possible to distinguish between different handling operations, and this distinction can be generalized, i.e. it is not necessary to measure precisely the time at which each handling operation in particular starts and ends, as long as it is sufficient to distinguish between the end of the last handling operation and the start of the next handling operation. This may be accomplished by determining an event (representing the beginning of the next transfer operation relative to the previous transfer operation) that the parts handler first entered the parts transfer work area after the event (representing the end of the previous transfer operation) that the parts handler left the parts transfer work area. Therefore, by determining the movement law of the material part carrier entering and leaving the material part carrying operation area, a sufficient basis can be provided for distinguishing different carrying operations or distinguishing the end of the last carrying operation and the start of the next carrying operation, and therefore, a work cycle algorithm can be established for dividing the work flow of the material part carrier into a plurality of work cycles. In summary, by determining the movement law of the material handler relative to the material handling operation area (including at least the movement law of the material handler entering and leaving the material handling operation area), a plurality of operation cycles corresponding to the operation flow of the material handler can be determined according to the movement law. In addition, the work flow of the material carrier is associated with the material carrying work area. This is because the operation flow of the material handler refers to the whole process of the material handler to finish the transportation of all the materials to be transported on the material transporting operation area, that is, the collection of the materials to be transported, and therefore the operation flow of the material handler is determined according to the materials on the specific material transporting operation area.

In step S206, a movement law of the parts handler relative to the parts handling operation area is determined, wherein the movement law at least includes a movement law of the parts handler entering and leaving the parts handling operation area. Under the application scene of intelligent automatic detection and judgment of waste steel, the material part carrier can correspond to the suction cups, the material part carrying operation area can correspond to the carriage area of a vehicle for loading the waste steel, and then the material part carrier corresponds to the motion law of the material part carrying operation area, corresponds to the motion law of the suction cups relative to the carriage area, and at least comprises the motion law of the suction cups entering and leaving the carriage area. And a plurality of operation cycles corresponding to the operation flow of the material part carrier and determined according to the motion law, wherein each operation cycle of the plurality of operation cycles represents a carrying operation or an operation of carrying a part of the material part by the material part carrier. Assuming that a plurality of operation cycles corresponding to the operation flow of the material carrier have N operation cycles in total, this means that the material carrier transports all the materials to be transported, i.e. the materials to be transported, on the material transporting operation area through N transporting operations in total.

In some embodiments, the mark of the part carrier is a predicted frame representing the part carrier and can also be called a detection frame of the part carrier, and the mark of the part carrying operation area is a predicted frame representing the part carrying operation area and can also be called a detection frame of the part carrying operation area. The movement rule of the material part carrier relative to the material part carrying operation area is determined according to the marks of the material part carrier and the marks of the material part carrying operation area, and the movement rule can be realized through the intersection condition between the detection frames of the material part carrier and the detection frames of the material part carrying operation area, the size of the overlapping area, the proportion of the overlapping area and the like. Under the application scene of intelligent automatic detection and judgment of waste steel, the motion rule of the suction cup relative to the carriage area can be determined according to the intersection condition between the detection frame of the suction cup and the detection frame of the carriage area, the size of the overlapping area, the proportion of the overlapping area and the like.

Referring to step S202, step S204, and step S206, by obtaining an image data set and labeling the obtained image data set, and then determining a motion law of the material handler relative to the material handling operation area according to the mark of the material handler and the mark of the material handling operation area, a motion law capable of reflecting the progress is determined along with the progress of the handling operation of the material handler (the change of the remaining material to be handled on the material handling operation area is brought by each handling operation of the material handler), and then a plurality of operation cycles can be determined according to the motion law. It should be understood that the determined motion law of the material handling device relative to the material handling operation area only needs to be used for representing the progress of the handling operation of the material handling device, and does not need to accurately measure the specific time of starting and ending each handling operation, but only needs to distinguish the ending of the last handling operation and the starting of the next handling operation so as to represent the progress of the handling operation of the material handling device. In this way, the computational complexity involved in determining the law of motion and establishing a work cycle algorithm (for determining a plurality of work cycles) may be effectively reduced, and any suitable algorithm may be employed, for example, based on the intersection between the detection boxes of the parts handler and the detection boxes of the parts handling work area, or the size of the overlapping area, or the proportion of the overlapping area, etc.

Step S208: obtaining a plurality of effective graphs from the image data set according to the plurality of work periods, wherein the plurality of effective graphs correspond to the plurality of work periods in a one-to-one mode.

The plurality of work cycles determined in step S206 represents the progress of the conveying operation of the material handling device, and each work cycle of the plurality of work cycles represents one conveying operation or an operation of the material handling device to convey a part of the material. Assuming that a plurality of operation cycles corresponding to the operation flow of the material carrier have N operation cycles in total, this means that the material carrier transports all the materials to be transported, i.e. the materials to be transported, on the material transporting operation area through N transporting operations in total. Therefore, the effective graphs can be used for representing the change process of the remaining to-be-transported parts in the part transporting operation area in the operation flow of the part transporter, including that all the to-be-transported parts in the initial state are not transported, and all the to-be-transported parts in the end state are transported after each transporting operation, namely, each operation cycle has a part of the to-be-transported parts changed after being transported. The plurality of effective graphs correspond to the plurality of working cycles one by one, and the N working cycles correspond to the N effective graphs one by one and also correspond to N times of carrying operation. The N effective diagrams are used to represent changes caused by each of the N conveying operations. For example, the first one of the N effective maps records a case where all the pieces to be conveyed in the initial state are not conveyed, and then each time the conveying operation is performed, the change caused by the conveying operation is recorded by one effective map until all the pieces to be conveyed in the end state are conveyed, so that the total N effective maps obtained record the change process of the set of pieces to be conveyed in the material conveying operation area from the initial state to the end state in the operation flow of the material conveyor. Furthermore, as mentioned above, the movement pattern of the material handler relative to the material handling operation area for determining a plurality of operation cycles does not require precise measurement of the specific time at which each handling operation starts and ends, but rather only the end of the last handling operation and the start of the next handling operation need to be distinguished. Thus, the overall process flow of obtaining a plurality of significance maps from the image dataset is greatly simplified and has good suitability. Specifically, the whole process flow of obtaining a plurality of effective graphs from the image data set includes obtaining the image data set and labeling the obtained image data set, then determining a motion rule of the material part carrier relative to the material part carrying operation area according to the mark of the material part carrier and the mark of the material part carrying operation area, and then determining a plurality of operation cycles according to the motion rule, so that a plurality of effective graphs can be determined according to the plurality of operation cycles. Such a processing flow may be adapted to the image dataset obtained in step S202 in a flexible and adjustable manner, may also be adapted to the label of the material handler and the label of the material handling work area obtained in step S204 in a flexible and variable detection technique, and may also be adapted to the motion law determined in step S206 in a flexible manner and the established work cycle algorithm, so that various artificial intelligence techniques and machine learning algorithms may be utilized according to actual needs and scene requirements, with good adaptability; and the processing flow does not require accurate measurement of specific time for starting and ending each conveying operation, so that the change process of the material set to be conveyed in the material conveying operation area from the initial state to the end state in the operation flow of the material conveyor is realized, and the calculation complexity is greatly reduced.

In some embodiments, a plurality of significance maps are obtained from the image dataset according to the plurality of job cycles, which may be based on the timing of the capturing or images in the image dataset. For example, in addition to the first significant map for recording a case where all the pieces of the set of pieces to be conveyed in the initial state are not conveyed, the shooting timing of the other significant maps may be selected after the completion of the last conveying operation but before the start of the next conveying operation, which may be based on the plurality of work cycles (for distinguishing the end of the last conveying operation from the start of the next conveying operation).

In some embodiments, after obtaining the plurality of effective graphs, the effective graphs may be further processed for interference resistance and noise removal.

Step S210: and determining at least one piece of associated information of the material set to be conveyed corresponding to the operation flow of the material conveyor according to the effective graphs.

As mentioned above, the plurality of effective maps record the change process of the material to be transported in the material transporting operation area from the initial state to the end state in the operation flow of the material transporter, and the information recorded by the plurality of effective maps includes that all the material to be transported in the initial state is not transported, and all the material to be transported in the end state is transported after each transporting operation, that is, each operation cycle has a change after a part of the material is transported. This means that, with the information recorded by the plurality of effective maps, along with the progress of the operation flow of the material part carrier or with each carrying operation or each operation cycle, it can be determined that the material part which is originally shielded or covered is exposed after being carried away along with other material parts which shield or cover the material part, that is, the information of all the material parts in the material part set to be carried, including the information of the initially shielded or covered material part, can be obtained according to the plurality of effective maps, so that the identification difficulty caused by superposition and shielding between the material parts to be carried in the material part set to be carried is solved. In addition, the work flow of the material part carrier is related to the material part carrying operation area, and is used for determining the motion law of the material part carrier relative to the material part carrying operation area in the plurality of periods, which means that the mark of the material part carrying operation area or the prediction result of the occupied area of the material part carrying operation area obtained based on the material part carrying operation area detection model is not based on the actual contour of the carriage of the vehicle or the actual edge of the vehicle body, so that the possibility that the material part exceeds the edge of the vehicle body is considered. In addition, the marks of the parts carrier and the marks of the parts carrying operation area are obtained through the parts carrier detection model and the parts carrying operation area detection model respectively, which means that the marks can be carried out under the condition that the parking position, the size and the direction of the vehicle loaded with the parts to be carried are difficult to predict without being limited to the preset designated parking area.

In some embodiments, at least one piece of association information of the to-be-handled material set can be determined according to the plurality of effective graphs based on an image semantic segmentation technology. Image semantic segmentation techniques generally refer to pixel-level classification of an image or labeling each pixel on an image with its corresponding class. Image semantic segmentation techniques interpret pixel-level images by semantic segmentation of full pixels, identifying, grouping, and assigning classes to each pixel. The process of obtaining semantic segmentation recognition results of an image may include image classification (i.e., recognizing content present in the image), object recognition (i.e., recognizing content and location present in the image and generating bounding boxes), and semantic segmentation (recognizing content and location present in the image and finding all pixels assigned to corresponding classes). Through the image semantic segmentation technology, the content of a certain effective graph (for example, whether the material is included or not and how many materials are) in the effective graph can be identified from the effective graphs, the position of the content (the position of each material) is determined, and pixels belonging to different contents (for example, pixels belonging to each material) are determined. The semantic segmentation recognition result of the effective graph obtained in the method is a pixel-level prediction result, namely the pixel points belonging to each material part and the positions or coordinates thereof are predicted), and the categories corresponding to the pixel points on the effective graph are also predicted. In one possible embodiment, the image semantic segmentation technique used to interpret the multiple significance maps may be the above-mentioned full-pixel semantic segmentation, i.e., classifying each pixel into a respective class. In another possible implementation, the image semantic segmentation technique for interpreting the multiple significance maps may be instance-aware semantic segmentation, i.e. classifying each pixel into an object class to which it belongs and assigning an entity number of the object class. In another possible embodiment, the image semantic segmentation technique for interpreting the plurality of effective images may be any specific technique belonging to the category of image semantic segmentation techniques, including concepts that are proposed after the filing date of the present application or derived concepts added to the scope of the image semantic segmentation technique. Determining at least one piece of associated information of the material set to be carried from the plurality of effective graphs through an image semantic segmentation technology, such as an image semantic segmentation model based on deep learning, wherein the at least one piece of associated information of the material set to be carried is based on a semantic segmentation recognition result of each effective graph. It should be understood that the semantic segmentation recognition result of each significance map includes a prediction result of a category corresponding to each pixel point on the significance map, that is, a pixel-level prediction result, and specifically, the manner of grouping and assigning the categories to the pixel points is determined based on a training process of an actually used image semantic segmentation model, model parameters, and the like, and can be used for predicting one or more categories. As mentioned above, the intelligent automatic detection and judgment of the material set to be carried is realized based on the computer vision technology, so that a basis is provided for decision-making and subsequent processing. Therefore, in general, the semantic segmentation recognition result of each active graph includes whether each pixel point on the active graph belongs to one material in the to-be-transported material set and the type information of the to-be-transported material to which the pixel point belongs. Taking an application scenario of intelligent automatic detection and judgment of waste steel products as an example, the semantic segmentation and identification result of each effective graph comprises whether each pixel point on the effective graph belongs to the waste steel parts to be carried or not and the type information of the waste steel parts to be carried. For example, in the application scenario of automatic scrap detection and judgment shown in fig. 1, the semantic segmentation and recognition result of the significance map includes determining pixels belonging to the scrap 132 to be transported and type information of the scrap 132 to be transported, and may also determine pixels belonging to another scrap, such as the scrap 130 to be transported and type information of the scrap 130 to be transported. Thus, based on the semantic segmentation recognition result of each active graph (which generally includes information about whether each pixel point on the active graph belongs to one material in the set of materials to be handled and the type of the material to be handled to which the pixel point belongs), at least one type of association information of the set of materials to be handled may be determined, for example, how many types of materials and the number of each type of materials are included in the set of materials to be handled. Taking an application scenario of intelligent automatic detection and judgment of waste steel as an example, at least one piece of associated information of the material set to be carried includes, for example, how many kinds of waste steel materials need to be recovered and the number of various waste steel materials, such as waste steel materials obtained by recovering 10 train wheels, waste steel materials obtained by recovering 20 automobile bearings, waste steel materials obtained by recovering 30 screws for construction, and the like, of the material set to be carried. It should be understood that the semantic segmentation recognition result of each effective graph obtained based on the image semantic segmentation technology and the at least one piece of related information of the material set to be handled may also include other aspects of information, such as contour information, source information, coordinate information, area information, pixel feature information, and the like, which may be implemented by combining actual needs and by adjusting the training process and model parameters of the image semantic segmentation model.

At step S210, at least one piece of association information of the set of parts to be handled may be determined according to the plurality of active maps based on any suitable computer vision technology. For example, feature extraction, detection segmentation, and the like of the significance map may be implemented by any suitable image classification technique, object detection technique, semantic segmentation technique, and instance segmentation technique, or a combination thereof, and may also be implemented in combination with any suitable preprocessing technique to achieve effects such as noise reduction or improvement of contrast of related information, and may also be implemented in combination with effects such as scene reconstruction technique and image restoration technique to improve recognition detection. As mentioned above, the plurality of effective maps record the change process of the material to be transported in the material transporting operation area from the initial state to the end state in the operation flow of the material transporter, and the information recorded by the plurality of effective maps includes that all the material to be transported in the initial state is not transported, and all the material to be transported in the end state is transported after each transporting operation, that is, each operation cycle has a change after a part of the material is transported. Therefore, based on a suitable computer vision technical solution, for example, based on an image semantic segmentation technique of deep learning, the information recorded by the plurality of effective graphs can be extracted, in particular, the information reflects the change of the remaining material pieces of the material piece set to be transported in the material piece transporting operation area with each transporting operation (or with the progress of the operation progress of the material piece transporter), while with each transporting operation taking away a part of the material pieces, the originally blocked or covered material pieces are exposed after being transported away with other material pieces blocking or covering the material pieces, and the material pieces can also be recorded by the subsequent effective graphs. Therefore, the information of all the parts of the set of parts to be carried (such as the type information of the parts to be carried) and further the at least one piece of related information of the set of parts to be carried (such as how many kinds of parts the set of parts to be carried comprises and the number of each kind of parts) can be determined according to the plurality of effective graphs based on the computer vision technology, and the difficulty in identification caused by superposition and obstruction among the parts to be carried of the set of parts to be carried is solved.

Please refer to the above steps S202 to S210, the identification method is used for intelligent material part detection and judgment, and can solve the problems encountered in the aspect of intelligent automatic detection and judgment based on the computer vision technology in the following application scenarios, including but not limited to, the recycling link of waste steel, the application scenario of intelligent automatic detection and judgment of waste steel, and similar application scenarios, such as industrial automation, goods sorting in logistics centers, port automation, intelligent automatic detection and judgment of goods, and any application scenarios, such as automatic coal sorting, garbage collection, automatic garbage sorting, etc., that may improve the production efficiency and reduce the labor cost through the identification method for intelligent material part detection and judgment. Specifically, the identification method includes acquiring an image data set, labeling the acquired image data set, determining a motion law of the material handling device relative to the material handling operation area according to the label of the material handling device and the label of the material handling operation area, so as to determine the motion law capable of reflecting the progress along with the progress of the handling operation of the material handling device (each handling operation of the material handling device brings the change of the remaining materials to be handled on the material handling operation area), and further determine a plurality of operation cycles according to the motion law, so that the specific starting time and the specific ending time of each handling operation do not need to be accurately measured, but only the ending time of the previous handling operation and the starting time of the next handling operation need to be distinguished, so as to reflect the progress of the handling operation of the material handling device, the method is beneficial to simplifying the processing flow and degrading the computational complexity; the information of all the materials in the material set to be conveyed is obtained according to the effective graphs, and the information comprises the information of the materials which are originally shielded or covered, so that the problem of difficulty in identification caused by superposition and shielding among the materials to be conveyed in the material set to be conveyed is solved; in addition, the possibility that the material part exceeds the edge of the vehicle body is considered on the basis of the mark of the material part conveying operation area obtained by the material part conveying operation area detection model or the prediction result of the occupied area of the material part conveying operation area, and not on the basis of the actual contour of the carriage of the vehicle, the actual edge of the vehicle body and the like; in addition, the marks of the material part carriers and the marks of the material part carrying operation areas are obtained through a material part carrier detection model and a material part carrying operation area detection model respectively, which means that the method is not limited to a preset designated parking area, and the problem that the parking position, the size and the direction of a vehicle for loading a material part set to be carried are difficult to predict is solved. Moreover, the processing flow of the identification method may be adapted to the image dataset obtained in step S202 in a flexible and adjustable manner, may also be adapted to the label of the material handler and the label of the material handling work area obtained in step S204 in a flexible and variable detection technique, and may also be adapted to the motion law determined in step S206 in a flexible manner and the established work cycle algorithm, so that various artificial intelligence techniques and machine learning algorithms may be utilized according to actual needs and scene needs, with good adaptability. In a word, the identification method for intelligently detecting and judging the material solves the common technical problem faced when the intelligent automatic detection and judgment of the waste steel is realized on the basis of the computer vision technology under the application scene of the intelligent automatic detection and judgment of the waste steel and similar application scenes, and realizes the intelligent automatic detection and judgment of goods, waste steel and the like on the basis of the computer vision technology, thereby providing a basis for decision making and subsequent processing, and being beneficial to improving the production efficiency, reducing the cost and improving the operation safety.

In a possible embodiment, the at least one piece of associated information of the set of material pieces to be handled includes at least one of: contour information, category information, source information, coordinate information, area information, pixel feature information. The information recorded by the effective graphs, particularly the information of the material which is originally shielded or covered, is extracted, so that the information of all the materials of the material set to be carried is determined and then integrated to obtain the associated information of the material set to be carried. And obtaining a semantic segmentation recognition result of each effective graph of the plurality of effective graphs based on any suitable computer vision technology, for example, through an image semantic segmentation model based on deep learning, wherein the semantic segmentation recognition result comprises a prediction result of a category corresponding to each pixel point on the effective graph, namely a pixel-level prediction result. The specific content of the category of each pixel point in the finally obtained pixel-level prediction result, including one or more categories corresponding to each pixel point and the semantic meaning of each category, can be determined by adjusting relevant details of the computer vision technology, for example, by adjusting model parameters and a training process of an image semantic segmentation model. Therefore, the semantic segmentation recognition result of each effective graph obtained based on the image semantic segmentation technology can obtain rich associated information about the material set to be carried. In some embodiments, the associated information of the material set to be handled includes profile information, where the profile information indicates a profile of each material of the material set to be handled, and the profile may be a result of matching with multiple preset profile types, or may be a semantic description (such as a side length, a curvature, and the like) performed in a numerical manner, or may be a more general semantic description (such as a disc shape, a strip shape, and the like). In some embodiments, the related information of the set of materials to be handled includes category information, where the category information indicates how many types of materials the set of materials to be handled includes and the number of each type of material, and these information can be used to further analyze and extract more information, so that the related information of the set of materials to be handled generally includes at least category information. The type information of the material set to be handled may indicate that the material set to be handled includes scrap steel material, and there are 10 train wheels, 20 car bearings, 30 screws, and the like in total. The type information of the material set to be carried can be used for evaluating the material quality, such as the quality of a railway wheel, the quality of an automobile bearing and the quality of a screw, so that the quality information of the material set to be carried can be established. Moreover, the quality information of the material set to be transported can be combined with prices corresponding to materials with different qualities to estimate the overall price of the material set to be transported. The type information of the material set to be carried can also be used for being combined with the outline information of the material set to be carried to carry out material weight estimation. For example, if the type information of a material is a train wheel, the approximate volume of the material can be calculated by combining the profile information of the material, and the weight of the material can be estimated by combining the empirical knowledge or the prior knowledge of the density of the train wheel. Therefore, the proportion information corresponding to the materials of different types of information in the material set to be conveyed can be obtained. In some embodiments, the associated information of the set of parts to be handled may further include one or more items of source information, coordinate information, area information, and pixel feature information. Wherein the source information indicates from which location a certain material comes, for example from a train or a barge. The coordinate information indicates the coordinates of a certain part on the active map. The area information indicates the area of a certain part identified on the active map. The pixel characteristic information indicates characteristics of all pixels to which a certain piece belongs. It should be understood that, by adjusting the relevant details of the computer vision technology, for example, by adjusting the model parameters and the training process of the image semantic segmentation model, the semantic information included in the finally obtained pixel-level prediction result can be determined, so that more abundant associated information of the material set to be handled can be obtained. The above listed examples of association information are illustrative only and not limiting. Therefore, abundant associated information of the material assembly to be carried is obtained, and basis is provided for decision making and subsequent processing.

In a possible embodiment, determining at least one piece of related information of a set of pieces to be handled corresponding to a workflow of the piece handler according to the plurality of active maps includes: for each effective graph of the effective graphs, determining a change area of the effective graph by comparing the effective graph with an effective graph corresponding to a previous working cycle relative to the working cycle corresponding to the effective graph, and determining a material part segmentation identification result corresponding to the change area of the effective graph through a material part segmentation model; and determining at least one piece of associated information of the material set to be conveyed according to the material segmentation identification result corresponding to the change area of each effective graph. The material segmentation model can be an image semantic segmentation model based on deep learning, and is used for providing pixel-level prediction results, namely prediction results of categories corresponding to each pixel point, wherein each pixel point corresponds to one or more categories and semantic meanings of each category. By determining the change area of the effective graph, the time cost and the resource loss caused by predicting the pixel points of all areas on the effective graph through the material part segmentation model can be saved, and only the pixel points of the change area of the effective graph need to be predicted. Therefore, the change area is determined by comparing the front effective graph and the rear effective graph, and the material part segmentation identification result corresponding to the change area of the effective graph is determined by the material part segmentation model, so that the calculation amount and the calculation complexity are effectively reduced. And finally, integrating the part segmentation identification results corresponding to the change areas of the effective graphs to determine at least one type of associated information of the to-be-conveyed part set.

In a possible embodiment, determining at least one piece of related information of a set of pieces to be handled corresponding to a workflow of the piece handler according to the plurality of active maps includes: respectively carrying out material part segmentation recognition on the plurality of effective graphs through a material part segmentation model so as to obtain a plurality of material part segmentation graphs, wherein the plurality of material part segmentation graphs correspond to the plurality of effective graphs one by one; and determining at least one piece of associated information of the material set to be carried according to the plurality of material segmentation maps. The material segmentation model can be an image semantic segmentation model based on deep learning, and is used for providing pixel-level prediction results, namely prediction results of categories corresponding to each pixel point, wherein each pixel point corresponds to one or more categories and semantic meanings of each category. Therefore, for each effective graph of the effective graphs, the material part segmentation model is used for performing material part segmentation identification to obtain a material part segmentation graph corresponding to the effective graph, and the material part segmentation graph is a pixel-level prediction result, namely a category corresponding to each pixel point on the effective graph is predicted. Semantic information contained in the finally obtained pixel-level prediction result can be determined by adjusting model parameters and a training process of the material segmentation model, and finally abundant associated information of the material set to be carried can be obtained by integrating the plurality of material segmentation maps. Further, in some implementations, the performing a part segmentation recognition on the plurality of effective maps through the part segmentation model respectively to obtain the plurality of part segmentation maps includes: and for each effective graph of the plurality of effective graphs, firstly, carrying out area limitation on the effective graph according to the mark of the material conveying operation area to obtain an effective graph with area limitation, then carrying out material segmentation identification on the effective graph with area limitation through the material segmentation model to obtain a material segmentation identification result of the effective graph with area limitation, and thus obtaining a material segmentation graph corresponding to the effective graph. This means that the frame representing the material handling operation area on the effective map can be determined according to the mark of the material handling operation area, so that the area limitation can be performed based on the frame of the material handling operation area on the effective map, so that the material part segmentation model only needs to perform material part segmentation identification on the effective map with the area limitation. That is to say, the material part segmentation model only predicts the pixel points in the frame of the material part carrying operation area on the effective graph, so that the time cost and the resource loss caused by predicting each pixel point on the effective graph are saved, and the calculation amount and the calculation complexity are effectively reduced. Further, in another implementation, the performing a part segmentation recognition on the plurality of effective maps through the part segmentation model respectively to obtain the plurality of part segmentation maps includes: and aiming at each effective graph of the plurality of effective graphs, firstly, performing material part segmentation recognition on the effective graph through the material part segmentation model to obtain a material part segmentation recognition result of the effective graph, and then performing area limitation on the material part segmentation recognition result of the effective graph according to the mark of the material part carrying operation area, so as to obtain the material part segmentation graph corresponding to the effective graph. This means that each pixel point on the effective graph is predicted by the material part segmentation model, and then the frame representing the material part conveying operation area on the effective graph determined according to the mark of the material part conveying operation area is subjected to area limitation. That is, the prediction results of the pixel points in the frame of the material handling operation area on the effective graph are divided from the pixel-level prediction results of the entire effective graph according to the frame of the material handling operation area. Considering that the size of each effective graph is consistent and the change of the front effective graph and the rear effective graph is not large (generally, only a part of the material is transported away and changes), the material segmentation recognition result of the effective graph is obtained by performing material segmentation recognition on the effective graph, and then the material segmentation recognition result of the effective graph is subjected to area limitation according to the mark of the material transportation operation area to obtain the material segmentation graph corresponding to the effective graph, which is beneficial to efficient processing. Further, in some implementations, determining at least one piece of related information of the set of pieces to be handled according to the plurality of piece segmentation maps includes: and comparing the part segmentation drawing with a part segmentation drawing corresponding to a previous work cycle of the work cycle corresponding to the part segmentation drawing in the part segmentation drawing through a part duplication removal model, and performing part duplication removal operation on each part segmentation drawing of the part segmentation drawings, so as to obtain at least one type of associated information of the part set to be carried. The part deduplication model may be a deep learning based neural network. By comparing the two material segmentation maps corresponding to the two effective maps, the material duplication elimination operation can be performed, that is, the material with duplication or similarity higher than the threshold value is eliminated, so that the material with duplication or changed material can be more highlighted, and the calculation amount of finally integrating the multiple material segmentation maps to obtain the associated information of the material set to be carried is facilitated.

In some implementations, the at least one piece of related information of the set of parts to be handled includes category information, and the identification method further includes: and inputting the type information of the material set to be conveyed into a material weight estimation model so as to obtain specific gravity information corresponding to the materials of different types of information of the material set to be conveyed. The type information of the material set to be conveyed indicates how many types of materials and the number of each type of materials are included in the material set to be conveyed, and the information can be used for estimating the weights of different types of materials by combining with the experience common knowledge or the prior knowledge in the aspect of density so as to obtain the specific gravity information corresponding to the materials of the material set to be conveyed with different types of information. The part weight estimation model may be a deep learning based neural network and trained to make predictions based on the input category information. In other implementations, the at least one piece of related information of the set of parts to be handled includes category information, and the identification method further includes: and inputting the type information of the material set to be conveyed into a material quality identification model, thereby obtaining the quality information of the material set to be conveyed. And optionally, the identification method further comprises: and determining the overall price of the material set to be carried according to the quality information of the material set to be carried. The type information of the material set to be transported can be used for evaluating the quality of the material, for example, the material is divided into different qualities such as fine products, common products and waste products according to different types of the material, so as to establish the quality information of the material set to be transported. And the quality information of the material set to be transported can be combined with prices corresponding to the materials with different qualities to estimate the overall price of the material set to be transported. The part quality recognition model may be a deep learning based neural network and trained to make predictions based on the input category information.

In a possible embodiment, during the course of the operation of the parts handler, at least one part is added to the parts handling operation area and becomes part of the set of parts to be handled corresponding to the operation of the parts handler. Thus, the condition of material adding is considered.

In one possible implementation, the identification method further includes: and respectively carrying out ultra-long material identification on the effective graphs through an ultra-long material model and giving out an alarm after the ultra-long material is identified. An ultra-long piece refers to a piece that is oversized so as to extend beyond the profile of the vehicle, such as a larger piece of scrap steel. The very long pieces may cause recognition errors because they extend beyond the contour of the vehicle, and in scrap recycling applications, very long pieces such as larger pieces of scrap also require special handling such as cutting before they can be fed into the furnace, for which reason they need to be handled separately. The position and the type of the overlong material part are identified through the overlong material part model, and a warning is given out, so that the detection reliability can be improved. The ultra-long material part model can be a trained neural network based on deep learning, and can comprise functions of ultra-long part recognition, ultra-long part contour extraction and the like.

In one possible implementation, the identification method further includes: and respectively carrying out seal identification on the plurality of effective graphs through seal models and giving out a warning after the seal is identified. Seals refer to hazardous materials such as oxygen bottles, fire extinguishers, etc. which require special attention and which remain sealed during handling and therefore often require separate or preferential disposal. The operation safety can be improved by identifying the position and the type of the sealing element through the sealing element model and giving a warning. The seal model may be a deep learning based trained neural network that may include seal identification, seal profile extraction, and the like. It should be understood that the seal model may also be used to identify hazardous materials in a more general sense, such as bullet-shaped, cylindrical objects, projectile-shaped, or any suspect-shaped piece of material.

In one possible embodiment, the identification method is used for automatic identification of a scrap, the scrap handler is a suction cup, a gripper or a scrap handler for handling the scrap, the scrap handling work area is a carriage of a vehicle for loading the scrap, and the operation flow of the scrap handler is that the suction cup, the gripper or the scrap handler for handling the scrap all carries the scrap on the carriage away from the carriage. In some embodiments, the parts handler is a suction cup, the parts handler inspection model is a suction cup inspection model, the parts handling work area inspection model is a car inspection model, and the motion law is a motion law of the suction cup relative to the car. Therefore, the applicability of the identification method in the scenes of automatic identification of the scrap steel parts and the like is reflected.

In one possible embodiment, the identification method is used for automatic identification of goods to be handled, the material handler is a suction cup, a gripper or a goods handler for handling goods, the material handling operation area is a goods accommodating area of a carrier for loading goods or a designated area for stacking goods, and the material handler has an operation flow that the suction cup, the gripper or the goods handler for handling goods conveys all the goods in the goods accommodating area or the designated area away from the goods accommodating area or the designated area. Therefore, the applicability of the identification method in the scenes of automatic identification of goods and the like is embodied, and the method can be used for industrial automation, goods sorting in logistics centers, port automation and the like.

In one possible embodiment, the parts handler detection model is an object detection model and the parts handling operation area detection model is an image semantic segmentation model.

In one possible embodiment, the marking of the material handler is a detection frame of the material handler, and the marking of the material handling operation area is a detection frame of the material handling operation area, wherein the determining of the motion rule of the material handler relative to the material handling operation area according to the detection frame of the material handler and the detection frame of the material handling operation area comprises: and determining the movement rule of the material part carrier entering and leaving the material part carrying operation area according to the change of the overlapping area between the detection frame of the material part carrier and the detection frame of the material part carrying operation area.

In a possible embodiment, the image data set is obtained by sampling or capturing a video data stream at a predetermined sampling frequency.

In one possible embodiment, the image data set is subjected to a data enhancement operation comprising at least one of: random inversion, rotation, inversion and rotation, random transformation, random scaling, random clipping, fuzzification, Gaussian noise addition and filling. It should be understood that the image dataset may also be subjected to any suitable pre-processing or data enhancement operation, and is not specifically limited herein.

It is to be understood that the above-described method may be implemented by a corresponding execution body or carrier. In some exemplary embodiments, a non-transitory computer readable storage medium stores computer instructions that, when executed by a processor, implement the above-described method and any of the above-described embodiments, implementations, or combinations thereof. In some example embodiments, an electronic device includes: a processor; a memory for storing processor-executable instructions; wherein the processor implements the above method and any of the above embodiments, implementations, or combinations thereof by executing the executable instructions.

Fig. 3 shows a block diagram of an electronic device used in the identification method shown in fig. 2 according to an embodiment of the present application. As shown in fig. 3, the electronic device includes a main processor 302, an internal bus 304, a network interface 306, a main memory 308, and secondary processor 310 and secondary memory 312, as well as a secondary processor 320 and secondary memory 322. The main processor 302 is connected to the main memory 308, and the main memory 308 may be used for storing computer instructions executable by the main processor 302, so that the identification method shown in fig. 2 may be implemented, including some or all of the steps, and any possible combination or combination of the steps, and possible alternatives or variations thereof. The network interface 306 is used to provide network connectivity and to transmit and receive data over a network. The internal bus 304 is used to provide internal data interaction between the main processor 302, the network interface 306, the auxiliary processor 310, and the auxiliary processor 320. The secondary processor 310 is coupled to the secondary memory 312 and provides secondary computing power, and the secondary processor 320 is coupled to the secondary memory 322 and provides secondary computing power. The auxiliary processors 310 and 320 may provide the same or different auxiliary computing capabilities including, but not limited to, computing capabilities optimized for particular computing requirements such as parallel processing capabilities or tensor computing capabilities, computing capabilities optimized for particular algorithms or logic structures such as iterative computing capabilities or graph computing capabilities, and the like. The secondary processor 310 and the secondary processor 320 may include one or more processors of a particular type, such as a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or the like, so that customized functions and structures may be provided. In some exemplary embodiments, the electronic device may not include an auxiliary processor, may include only one auxiliary processor, and may include any number of auxiliary processors and each have a corresponding customized function and structure, which are not specifically limited herein. The architecture of the two auxiliary processors shown in FIG. 3 is for illustration only and should not be construed as limiting. In addition, the main processor 302 may include a single-core or multi-core computing unit to provide the functions and operations necessary for embodiments of the present application. In addition, the main processor 302 and the auxiliary processors (such as the auxiliary processor 310 and the auxiliary processor 320 in fig. 3) may have different architectures, that is, the electronic device may be a heterogeneous architecture based system, for example, the main processor 302 may be a general-purpose processor based on an instruction set operating system, such as a CPU, and the auxiliary processor may be a graphics processor GPU suitable for parallelized computation or a dedicated accelerator suitable for neural network model-related operations. The auxiliary memory (e.g., auxiliary memory 312 and auxiliary memory 322 shown in fig. 3) may be used to implement customized functions and structures with the respective auxiliary processors. While main memory 308 is operative to store the necessary instructions, software, configurations, data, etc. to provide the functionality and operations necessary for embodiments of the subject application in conjunction with main processor 302. In some exemplary embodiments, the electronic device may not include the auxiliary memory, may include only one auxiliary memory, and may further include any number of auxiliary memories, which is not specifically limited herein. The architecture of the two auxiliary memories shown in fig. 3 is illustrative only and should not be construed as limiting. Main memory 308, and possibly secondary memory, may include one or more of the following features: volatile, nonvolatile, dynamic, static, readable/writable, read-only, random-access, sequential-access, location-addressability, file-addressability, and content-addressability, and may include random-access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, a hard disk, a removable disk, a recordable and/or rewriteable Compact Disc (CD), a Digital Versatile Disc (DVD), a mass storage media device, or any other form of suitable storage media. The internal bus 304 may include any of a variety of different bus structures or combinations of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and/or a processor or local bus that utilizes any of a variety of bus architectures. It should be understood that the electronic device shown in fig. 3, the illustrated structure of which does not constitute a specific limitation as to the apparatus or system, may in some exemplary embodiments include more or less components than the specific embodiments and figures, or combine certain components, or split certain components, or have a different arrangement of components.

With continued reference to fig. 3, in one possible implementation, the auxiliary processor 310 and/or the auxiliary processor 320 may have a computing architecture that is custom designed for the characteristics of neural network computing, such as a neural network accelerator. Moreover, the electronic device may include any number of auxiliary processors each having a computing architecture that is custom designed for the characteristics of neural network computations, or the electronic device may include any number of neural network accelerators. In some embodiments, for illustrative purposes only, an exemplary neural network accelerator may be: the neural network accelerator is provided with a time domain computing architecture based on a control flow, and the instruction flow of an instruction set is customized based on a neural network algorithm to perform centralized control on computing resources and storage resources; alternatively, neural network accelerators with a data-flow based spatial computation architecture, such as two-dimensional spatial computation arrays based on Row Stationary (RS) data flows, two-dimensional matrix multiplication arrays using Systolic arrays (Systolic Array), and the like; or any neural network accelerator having any suitable custom designed computational architecture.

Fig. 4 shows a block diagram of an identification apparatus for intelligent detection and judgment of a material part according to an embodiment of the present application. The identification means as shown in fig. 4 comprises two network branches, a first network branch 410 and a part identification network branch 450 (which may also be called a second network branch). The first network branch 410 includes a parts handler detection model 412, a parts handling work area detection model 414, a work cycle algorithm generation model 420 and an active map generation model 430. The parts handler detection model 412 is configured to receive the image dataset 402, identify the parts handler in the image dataset 402, and obtain the label 413 of the parts handler. The parts handling operation area detection model 414 is configured to receive the image dataset 402, identify the parts handling operation area in the image dataset 402, and obtain an indication 415 of the parts handling operation area. The parts handler detection model 412 and the parts handling work area detection model 414 are connected to the work cycle algorithm generation model 420 and send the marks 413 and 415 of the parts handler and the parts handling work area to the work cycle algorithm generation model 420, respectively. The work cycle algorithm generation model 420 is configured to receive the image data set 402, determine a motion rule of the material handler relative to the material handling work area according to the label 413 of the material handler and the label 415 of the material handling work area, and determine a plurality of work cycles corresponding to a work flow of the material handler according to the motion rule, that is, to establish a work cycle algorithm or generate a work cycle 421. Wherein the operation process of the material carrier is associated with the material carrying operation area. The work cycle algorithm generation model 420 is connected to the significance map generation model 430 and transmits the generated work cycle 421 (a plurality of work cycles corresponding to the work flow of the material handler) to the significance map generation model 430. The significance map generation model 430 is configured to receive the image data set 402 and obtain a plurality of significance maps, i.e., significance maps 431, from the image data set 402 according to a plurality of job cycles, i.e., job cycles 421. The active maps correspond to the operation cycles one to one, that is, the active map 431 corresponds to the operation cycle 421 one to one. In this way, the first network branch 410 implements obtaining of the significance map 431, i.e. a plurality of significance maps, from the image data set 402, the significance maps record the change process of the material to be transported set in the material transporting operation area from the initial state to the end state in the operation flow of the material transporter, and the information recorded by the significance maps includes that all the materials to be transported in the initial state are not transported, and all the materials to be transported in the end state are transported after each transporting operation, i.e. after a part of the materials are transported in each operation cycle. This means that, with the information recorded by the plurality of effective maps, along with the progress of the operation flow of the material part carrier or with each carrying operation or each operation cycle, it can be determined that the material part which is originally shielded or covered is exposed after being carried away along with other material parts which shield or cover the material part, that is, the information of all the material parts in the material part set to be carried, including the information of the initially shielded or covered material part, can be obtained according to the plurality of effective maps, so that the identification difficulty caused by superposition and shielding between the material parts to be carried in the material part set to be carried is solved. The activity map generation model 430 of the first network branch 410 is coupled to the part identification network branch 450 and sends the generated activity map 431 to the part identification network branch 450. The part identification network branch 450 is configured to determine at least one kind of related information of a set of parts to be handled corresponding to the workflow of the part handler according to the significance map 431, that is, the significance maps.

In some embodiments, the parts identification network branch 450 may be based on an image semantic segmentation technique to determine at least one associated information of a set of parts to be handled from the plurality of active maps. That is, the part recognition network branch 450 may be an image semantic segmentation model. In some embodiments, the parts identification network branch 450 may be based on any suitable computer vision technique to determine at least one type of associated information for a set of parts to be handled based on the plurality of significance maps. For example, feature extraction, detection segmentation, and the like of the significance map may be implemented by any suitable image classification technique, object detection technique, semantic segmentation technique, and instance segmentation technique, or a combination thereof, and may also be implemented in combination with any suitable preprocessing technique to achieve effects such as noise reduction or improvement of contrast of related information, and may also be implemented in combination with effects such as scene reconstruction technique and image restoration technique to improve recognition detection. As mentioned above, the plurality of effective maps record the change process of the material to be transported in the material transporting operation area from the initial state to the end state in the operation flow of the material transporter, and the information recorded by the plurality of effective maps includes that all the material to be transported in the initial state is not transported, and all the material to be transported in the end state is transported after each transporting operation, that is, each operation cycle has a change after a part of the material is transported. Therefore, the part identification network branch 450 can extract the information recorded by the plurality of effective graphs based on a suitable computer vision technical scheme, for example, an image semantic segmentation technology based on deep learning, in particular, the information reflects the change of the remaining parts of the set of parts to be transported in the part transportation operation area with each transportation operation (or with the progress of the operation progress of the part transporter), while with each transportation operation taking away a part of the parts, the originally blocked or covered parts are exposed after being transported away with other parts blocking or covering the parts, and the parts are also recorded by the subsequent effective graphs. Therefore, the information of all the parts of the set of parts to be carried (such as the type information of the parts to be carried) and further the at least one piece of related information of the set of parts to be carried (such as how many kinds of parts the set of parts to be carried comprises and the number of each kind of parts) can be determined according to the plurality of effective graphs based on the computer vision technology, and the difficulty in identification caused by superposition and obstruction among the parts to be carried of the set of parts to be carried is solved.

With continuing reference to fig. 4, the recognition device is used for intelligent material piece inspection and judgment, and can solve the problems encountered in the aspect of intelligent automatic inspection and judgment based on the computer vision technology in the following application scenarios, including but not limited to, waste steel recycling links, application scenarios of intelligent automatic inspection and judgment of waste steel, and similar application scenarios such as industrial automation, cargo sorting in logistics centers, port automation, intelligent automatic inspection and judgment of cargos, and any application scenarios that may improve the production efficiency and reduce the labor cost through the recognition device for intelligent material piece inspection and judgment, such as automatic coal sorting, garbage recycling, automatic garbage sorting, and the like. Specifically, the identification device obtains an image data set, marks the obtained image data set, and then determines the movement law of the material handling device relative to the material handling operation area according to the mark of the material handling device and the mark of the material handling operation area, so that the movement law capable of showing the progress is determined along with the progress of the handling operation of the material handling device (each handling operation of the material handling device brings the change of the remaining materials to be handled on the material handling operation area), and then a plurality of operation cycles can be determined according to the movement law, so that the specific starting time and the specific ending time of each handling operation do not need to be accurately measured, but only the ending time of the previous handling operation and the starting time of the next handling operation need to be distinguished, so that the progress of the handling operation of the material handling device can be shown, the method is beneficial to simplifying the processing flow and degrading the computational complexity; the information of all the materials in the material set to be conveyed is obtained according to the effective graphs, and the information comprises the information of the materials which are originally shielded or covered, so that the problem of difficulty in identification caused by superposition and shielding among the materials to be conveyed in the material set to be conveyed is solved; in addition, the possibility that the material part exceeds the edge of the vehicle body is considered based on the mark of the material part conveying operation area obtained by the material part conveying operation area detection model 414 or the prediction result of the occupied area of the material part conveying operation area, rather than the actual contour of the carriage of the vehicle or the actual edge of the vehicle body; in addition, the marks of the parts handler and the marks of the parts handling work area are obtained by the parts handler detection model 412 and the parts handling work area detection model 414, respectively, which means that the parking areas are not limited to the designated parking areas planned in advance, and the problem that the parking positions, sizes and orientations of the vehicles loading the parts sets to be handled are difficult to predict is solved. In addition, the recognition device can utilize various artificial intelligence technologies and machine learning algorithms according to actual needs and scene requirements, and has good adaptability. In a word, the identification device for intelligently detecting and judging the material solves the common technical problem faced when the intelligent automatic detection and judgment of the waste steel is realized on the basis of the computer vision technology under the application scene of the intelligent automatic detection and judgment of the waste steel and the similar application scene, and realizes the intelligent automatic detection and judgment of goods, waste steel and the like on the basis of the computer vision technology, thereby providing a basis for decision making and subsequent processing, being beneficial to improving the production efficiency, reducing the cost and improving the operation safety.

In a possible embodiment, the at least one piece of associated information of the set of material pieces to be handled includes at least one of: contour information, category information, source information, coordinate information, area information, pixel feature information. Therefore, abundant associated information of the material assembly to be carried is obtained, and basis is provided for decision making and subsequent processing.

In a possible embodiment, the part identification network branch 450 includes a part segmentation model, and determining at least one related information of a set of parts to be handled corresponding to the workflow of the part handler according to the plurality of active maps includes: respectively carrying out material part segmentation recognition on the plurality of effective graphs through the material part segmentation model so as to obtain a plurality of material part segmentation graphs, wherein the plurality of material part segmentation graphs correspond to the plurality of effective graphs one by one; and determining at least one piece of associated information of the material set to be carried according to the plurality of material segmentation maps. The material segmentation model can be an image semantic segmentation model based on deep learning, and is used for providing pixel-level prediction results, namely prediction results of categories corresponding to each pixel point, wherein each pixel point corresponds to one or more categories and semantic meanings of each category. Semantic information contained in the finally obtained pixel-level prediction result can be determined by adjusting model parameters and a training process of the material segmentation model, and finally abundant associated information of the material set to be carried can be obtained by integrating the plurality of material segmentation maps. In some embodiments, the component recognition network branch includes a component deduplication model, and determining at least one piece of associated information of the set of components to be handled according to the plurality of component segmentation maps includes: and for each material part segmentation drawing of the plurality of material part segmentation drawings, comparing the material part segmentation drawing with a material part segmentation drawing corresponding to the previous work cycle of the work cycle corresponding to the material part segmentation drawing in the plurality of material part segmentation drawings through the material part duplication removal model, and performing material part duplication removal operation, so as to obtain the type information of the material part set to be conveyed. The part deduplication model may be a deep learning based neural network. By comparing the two material segmentation maps corresponding to the two effective maps, the material duplication elimination operation can be performed, that is, the material with duplication or similarity higher than the threshold value is eliminated, so that the material with duplication or changed material can be more highlighted, and the calculation amount of finally integrating the multiple material segmentation maps to obtain the associated information of the material set to be carried is facilitated.

In a possible embodiment, the material part identification network branch 450 includes an extra-long material part model and a sealing element model, wherein the extra-long material part model is used for respectively carrying out extra-long material part identification on the plurality of effective graphs and sending out an alarm after the extra-long material part is identified, and the sealing element model is used for respectively carrying out sealing element identification on the plurality of effective graphs and sending out an alarm after the sealing element is identified. An ultra-long piece refers to a piece that is oversized so as to extend beyond the profile of the vehicle, such as a larger piece of scrap steel. The very long pieces may cause recognition errors because they extend beyond the contour of the vehicle, and in scrap recycling applications, very long pieces such as larger pieces of scrap also require special handling such as cutting before they can be fed into the furnace, for which reason they need to be handled separately. The position and the type of the overlong material part are identified through the overlong material part model, and a warning is given out, so that the detection reliability can be improved. The ultra-long material part model can be a trained neural network based on deep learning, and can comprise functions of ultra-long part recognition, ultra-long part contour extraction and the like. Seals refer to hazardous materials such as oxygen bottles, fire extinguishers, etc. which require special attention and which remain sealed during handling and therefore often require separate or preferential disposal. The operation safety can be improved by identifying the position and the type of the sealing element through the sealing element model and giving a warning. The seal model may be a deep learning based trained neural network that may include seal identification, seal profile extraction, and the like. It should be understood that the seal model may also be used to identify hazardous materials in a more general sense, such as bullet-shaped, cylindrical objects, projectile-shaped, or any suspect-shaped piece of material.

In one possible embodiment, the identification device is used for automatic identification of a scrap, the scrap handler is a suction cup, a gripper or a scrap handler for handling the scrap, the scrap handling area is a carriage of a vehicle for loading the scrap, and the operation flow of the scrap handler is that the suction cup, the gripper or the scrap handler for handling the scrap all carries the scrap on the carriage away from the carriage. Therefore, the applicability of the identification method in the scenes of automatic identification of the scrap steel parts and the like is reflected.

The embodiments provided herein may be implemented in any one or combination of hardware, software, firmware, or solid state logic circuitry, and may be implemented in connection with signal processing, control, and/or application specific circuitry. Particular embodiments of the present application provide an apparatus or device that may include one or more processors (e.g., microprocessors, controllers, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), etc.) that process various computer-executable instructions to control the operation of the apparatus or device. Particular embodiments of the present application provide an apparatus or device that can include a system bus or data transfer system that couples the various components together. A system bus can include any of a variety of different bus structures or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and/or a processor or local bus that utilizes any of a variety of bus architectures. The devices or apparatuses provided in the embodiments of the present application may be provided separately, or may be part of a system, or may be part of other devices or apparatuses.

Particular embodiments provided herein may include or be combined with computer-readable storage media, such as one or more storage devices capable of providing non-transitory data storage. The computer-readable storage medium/storage device may be configured to store data, programmers and/or instructions that, when executed by a processor of an apparatus or device provided by embodiments of the present application, cause the apparatus or device to perform operations associated therewith. The computer-readable storage medium/storage device may include one or more of the following features: volatile, non-volatile, dynamic, static, read/write, read-only, random access, sequential access, location addressability, file addressability, and content addressability. In one or more exemplary embodiments, the computer-readable storage medium/storage device may be integrated into a device or apparatus provided in the embodiments of the present application or belong to a common system. The computer-readable storage medium/memory device may include optical, semiconductor, and/or magnetic memory devices, etc., and may also include Random Access Memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, a hard disk, a removable disk, a recordable and/or rewriteable Compact Disc (CD), a Digital Versatile Disc (DVD), a mass storage media device, or any other form of suitable storage media.

The above is an implementation manner of the embodiments of the present application, and it should be noted that the steps in the method described in the embodiments of the present application may be sequentially adjusted, combined, and deleted according to actual needs. In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments. It is to be understood that the embodiments of the present application and the structures shown in the drawings are not to be construed as particularly limiting the devices or systems concerned. In other embodiments of the present application, an apparatus or system may include more or fewer components than the specific embodiments and figures, or may combine certain components, or may separate certain components, or may have a different arrangement of components. Those skilled in the art will understand that various modifications and changes may be made in the arrangement, operation, and details of the methods and apparatus described in the specific embodiments without departing from the spirit and scope of the embodiments herein; without departing from the principles of embodiments of the present application, several improvements and modifications may be made, and such improvements and modifications are also considered to be within the scope of the present application.

Claims (25)

1. An identification method is used for intelligent detection and judgment of a material part, and is characterized by comprising the following steps:

obtaining an image dataset;

identifying a material carrier in the image dataset and obtaining a mark of the material carrier through a material carrier detection model, and identifying a material carrying operation area in the image dataset and obtaining a mark of the material carrying operation area through a material carrying operation area detection model;

determining a motion rule of the material part carrier relative to the material part carrying operation area according to the mark of the material part carrier and the mark of the material part carrying operation area, and determining a plurality of operation cycles corresponding to an operation flow of the material part carrier according to the motion rule, wherein the operation flow of the material part carrier is associated with the material part carrying operation area;

obtaining a plurality of effective graphs from the image data set according to the plurality of work periods, wherein the plurality of effective graphs correspond to the plurality of work periods in a one-to-one mode; and

determining at least one kind of associated information of the material set to be carried corresponding to the operation flow of the material carrier according to the effective graphs,

determining at least one kind of associated information of a to-be-conveyed material set corresponding to the operation flow of the material part conveyor according to the effective graphs, wherein the method comprises the following steps: for each effective graph of the effective graphs, determining a change area of the effective graph by comparing the effective graph with an effective graph corresponding to a previous working cycle relative to the working cycle corresponding to the effective graph, and determining a material part segmentation identification result corresponding to the change area of the effective graph through a material part segmentation model; determining at least one kind of associated information of the material set to be carried according to the material segmentation identification result corresponding to the change area of each effective graph,

alternatively, the first and second electrodes may be,

determining at least one kind of associated information of a to-be-conveyed material set corresponding to the operation flow of the material part conveyor according to the effective graphs, wherein the associated information comprises: respectively carrying out material part segmentation recognition on the plurality of effective graphs through the material part segmentation model so as to obtain a plurality of material part segmentation graphs, wherein the plurality of material part segmentation graphs correspond to the plurality of effective graphs one by one; and determining at least one piece of associated information of the material set to be carried according to the plurality of material segmentation maps.

2. The identification method according to claim 1, wherein the at least one piece of associated information of the set of material pieces to be handled comprises at least one of: contour information, category information, source information, coordinate information, area information, pixel feature information.

3. The identification method according to claim 1, wherein the identifying the effective maps by material segmentation through the material segmentation model to obtain the material segmentation maps comprises:

and for each effective graph of the effective graphs, firstly, carrying out area limitation on the effective graph according to the mark of the material conveying operation area to obtain the effective graph subjected to area limitation, then carrying out material segmentation identification on the effective graph subjected to area limitation through the material segmentation model to obtain a material segmentation identification result of the effective graph subjected to area limitation, and thus obtaining the material segmentation graph corresponding to the effective graph.

4. The identification method according to claim 1, wherein the identifying the effective maps by material segmentation through the material segmentation model to obtain the material segmentation maps comprises:

and aiming at each effective graph of the plurality of effective graphs, firstly, performing material part segmentation recognition on the effective graph through the material part segmentation model to obtain a material part segmentation recognition result of the effective graph, and then performing area limitation on the material part segmentation recognition result of the effective graph according to the mark of the material part carrying operation area, so as to obtain the material part segmentation graph corresponding to the effective graph.

5. The identification method according to claim 1, wherein determining at least one piece of association information of the to-be-handled piece set according to the plurality of piece segmentation maps comprises:

and comparing the part segmentation drawing with a part segmentation drawing corresponding to a previous work cycle of the work cycle corresponding to the part segmentation drawing in the part segmentation drawing through a part duplication removal model, and performing part duplication removal operation on each part segmentation drawing of the part segmentation drawings, so as to obtain at least one type of associated information of the part set to be carried.

6. The identification method according to claim 1, wherein the at least one piece of related information of the set of material pieces to be handled includes category information, and the identification method further comprises:

and inputting the type information of the material set to be conveyed into a material weight estimation model so as to obtain specific gravity information corresponding to the materials of different types of information of the material set to be conveyed.

7. The identification method according to claim 1, wherein the at least one piece of related information of the set of material pieces to be handled includes category information, and the identification method further comprises:

and inputting the type information of the material set to be conveyed into a material quality identification model, thereby obtaining the quality information of the material set to be conveyed.

8. The identification method according to claim 7, characterized in that the identification method further comprises:

and determining the overall price of the material set to be carried according to the quality information of the material set to be carried.

9. The method according to claim 1, wherein at least one part is added to the part handling operation area and becomes part of a set of parts to be handled corresponding to the workflow of the part handler during the workflow of the part handler.

10. The identification method according to claim 1, characterized in that the identification method further comprises:

and respectively carrying out ultra-long material identification on the effective graphs through an ultra-long material model and giving out an alarm after the ultra-long material is identified.

11. The identification method according to claim 1, characterized in that the identification method further comprises:

and respectively carrying out seal identification on the plurality of effective graphs through seal models and giving out a warning after the seal is identified.

12. The identification method according to any one of claims 1 to 11,

the identification method is used for automatically identifying the scrap steel parts,

the material part carrier is a sucker, a gripper or a steel scrap material part carrier for carrying steel scrap material parts,

the material handling operation area is a carriage of a vehicle for loading scrap material,

the operation flow of the material part carrier is that the suction disc, the gripper or the steel scrap carrier for carrying the steel scrap material carries all the steel scrap materials on the carriage away from the carriage.

13. The method of claim 12, wherein the parts handler is a suction cup, the parts handler inspection model is a suction cup inspection model, the parts handling work area inspection model is a car inspection model, and the motion law is a motion law of the suction cup relative to the car.

14. The identification method according to any one of claims 1 to 11,

the identification method is used for automatic identification of the goods to be transported,

the material carrier is a suction cup, a gripper or a cargo carrier for carrying cargo,

the material handling operation area is a cargo accommodation area of a carrier for loading cargo or a designated area for stacking cargo,

the operation process of the material part carrier is that the suction cups, the grippers or the goods carrier used for carrying the goods carry all the goods in the goods containing area or the appointed area away from the goods containing area or the appointed area.

15. The identification method according to any one of claims 1 to 11, characterized in that the parts handler detection model is an object detection model and the parts handling work area detection model is an image semantic segmentation model.

16. The identification method according to any one of claims 1 to 11, wherein the marking of the parts handler is a detection frame of the parts handler and the marking of the parts handling work area is a detection frame of the parts handling work area, wherein determining the law of motion of the parts handler relative to the parts handling work area based on the detection frame of the parts handler and the detection frame of the parts handling work area comprises:

and determining the movement rule of the material part carrier entering and leaving the material part carrying operation area according to the change of the overlapping area between the detection frame of the material part carrier and the detection frame of the material part carrying operation area.

17. The identification method according to any one of claims 1 to 11,

the image data set is obtained by sampling or capturing images of a video data stream according to a preset sampling frequency.

18. The identification method according to any one of claims 1 to 11,

the image data set is subjected to a data enhancement operation comprising at least one of: random inversion, rotation, inversion and rotation, random transformation, random scaling, random clipping, fuzzification, Gaussian noise addition and filling.

19. A non-transitory computer-readable storage medium storing computer instructions which, when executed by a processor, implement the identification method according to any one of claims 1 to 18.

20. An electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor implements the identification method according to any one of claims 1 to 18 by executing the executable instructions.

21. The utility model provides an identification means for material piece intelligence is examined and is judged, its characterized in that, identification means includes:

a part handler detection model for identifying part handlers in the image dataset and obtaining markings of the part handlers;

a material handling operation area detection model for identifying the material handling operation area in the image data set and obtaining a mark of the material handling operation area;

the operation cycle algorithm generation model is used for determining a motion rule of the material part carrier relative to the material part carrying operation area according to the mark of the material part carrier and the mark of the material part carrying operation area, and determining a plurality of operation cycles corresponding to an operation process of the material part carrier according to the motion rule, wherein the operation process of the material part carrier is associated with the material part carrying operation area;

an effective graph generation model for obtaining a plurality of effective graphs from the image data set according to the plurality of work periods, wherein the plurality of effective graphs correspond to the plurality of work periods one by one; and

the part identification network branch is used for determining at least one piece of associated information of a part set to be carried corresponding to the operation flow of the part carrier according to the effective graphs,

the parts identification network branch comprises a parts segmentation model, and at least one piece of associated information of a parts set to be carried corresponding to the operation flow of the parts carrier is determined according to the effective graphs, and the method comprises the following steps: for each effective graph of the effective graphs, determining a change area of the effective graph by comparing the effective graph with an effective graph corresponding to a previous working cycle relative to the working cycle corresponding to the effective graph, and determining a material part segmentation identification result corresponding to the change area of the effective graph through the material part segmentation model; determining at least one kind of associated information of the material set to be carried according to the material segmentation identification result corresponding to the change area of each effective graph

Alternatively, the first and second electrodes may be,

determining at least one kind of associated information of a to-be-conveyed material set corresponding to the operation flow of the material part conveyor according to the effective graphs, wherein the associated information comprises: respectively carrying out material part segmentation recognition on the plurality of effective graphs through the material part segmentation model so as to obtain a plurality of material part segmentation graphs, wherein the plurality of material part segmentation graphs correspond to the plurality of effective graphs one by one; and determining at least one piece of associated information of the material set to be carried according to the plurality of material segmentation maps.

22. The identification device according to claim 21, wherein the at least one piece of related information of the set of material pieces to be handled comprises at least one of: contour information, category information, source information, coordinate information, area information, pixel feature information.

23. The identification device according to claim 21, wherein the part identification network branch comprises a part de-duplication model, and the determining at least one kind of related information of the to-be-handled part set according to the plurality of part segmentation maps comprises:

and for each material part segmentation drawing of the plurality of material part segmentation drawings, comparing the material part segmentation drawing with a material part segmentation drawing corresponding to the previous work cycle of the work cycle corresponding to the material part segmentation drawing in the plurality of material part segmentation drawings through the material part duplication removal model, and performing material part duplication removal operation, so as to obtain the type information of the material part set to be conveyed.

24. The identification device according to claim 21, wherein the part identification network branch comprises an extra-long part model and a seal model, wherein the extra-long part model is used for respectively carrying out extra-long part identification on the plurality of effective graphs and giving an alarm after identifying the extra-long part, and the seal model is used for respectively carrying out seal identification on the plurality of effective graphs and giving an alarm after identifying the seal.

25. Identification means according to any of claims 21 to 24,

the identification device is used for automatically identifying the scrap steel parts,

the material part carrier is a sucker, a gripper or a steel scrap material part carrier for carrying steel scrap material parts,

the material handling operation area is a carriage of a vehicle for loading scrap material,

the operation flow of the material part carrier is that the suction disc, the gripper or the steel scrap carrier for carrying the steel scrap material carries all the steel scrap materials on the carriage away from the carriage.

Images