WO2023241102A1 - 一种标签识别方法、装置、电子设备及存储介质 - Google Patents

一种标签识别方法、装置、电子设备及存储介质 Download PDF

Info

Publication number
WO2023241102A1
WO2023241102A1 PCT/CN2023/079023 CN2023079023W WO2023241102A1 WO 2023241102 A1 WO2023241102 A1 WO 2023241102A1 CN 2023079023 W CN2023079023 W CN 2023079023W WO 2023241102 A1 WO2023241102 A1 WO 2023241102A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
label
labels
recognized
predicted
Prior art date
Application number
PCT/CN2023/079023
Other languages
English (en)
French (fr)
Inventor
周靖超
贾淇超
刘浩
周邦国
滕辉
李志远
Original Assignee
卡奥斯工业智能研究院(青岛)有限公司
卡奥斯物联科技股份有限公司
海尔数字科技(青岛)有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 卡奥斯工业智能研究院(青岛)有限公司, 卡奥斯物联科技股份有限公司, 海尔数字科技(青岛)有限公司 filed Critical 卡奥斯工业智能研究院(青岛)有限公司
Publication of WO2023241102A1 publication Critical patent/WO2023241102A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Definitions

  • the embodiments of the present application relate to computer technology, for example, to a tag identification method, device, electronic equipment and storage medium.
  • the quality inspection link in the factory's production line is generally completed by quality inspectors to prevent defective products from flowing into the market.
  • the quality inspection link involves processes such as accessory labeling model confirmation, product component assembly quality confirmation, and product surface scratch defect detection. It requires the cooperation of multiple quality inspection personnel.
  • Most of the factory production lines adopt a mixed production method. Some products only have different internal parts and models, but the appearance size is not very different, resulting in a wide variety of product models and labels, such as energy consumption stickers, product nameplates, warning stickers, consumption stickers, etc. Battery, etc., have small fonts and long numbers, which are time-consuming for quality inspection personnel to check. Different labels on different products are easily confused. Missing inspections and misdetections also occur from time to time, and traceability is also difficult.
  • OCR Optical Character Recognition
  • This application provides a tag identification method, device, electronic equipment and storage medium to realize automatic identification of the content in the tag without human participation.
  • embodiments of the present application provide a tag identification method, including:
  • a label prediction network to predict labels in the image to be identified, and obtain multiple predicted labels in the image to be identified and the confidence of each predicted label in the multiple predicted labels;
  • a label determination model is used to determine a target label for the item based on the plurality of predicted labels and the confidence of each predicted label.
  • embodiments of the present application also provide a tag identification device, which includes:
  • the image acquisition module is configured to obtain the labeling image of the item, and preprocess the labeling image to obtain the image to be recognized;
  • a network prediction module configured to use a label prediction network to predict labels in the image to be identified, and obtain multiple predicted labels in the image to be identified and the confidence of each predicted label in the multiple predicted labels;
  • a label determination module configured to use a label determination model to determine the target label of the item based on the plurality of predicted labels and the confidence of each predicted label.
  • embodiments of the present application also provide an electronic device, which includes:
  • a storage device arranged to store at least one program
  • the at least one processor When the at least one program is executed by the at least one processor, the at least one processor implements the tag identification method.
  • embodiments of the present application also provide a computer-readable storage medium on which a computer program is stored, which implements the tag identification method when executed by a processor.
  • Figure 1 is a schematic flow chart of a tag identification method provided by an embodiment of the present application.
  • FIG. 2 is another schematic flowchart of a tag identification method provided by an embodiment of the present application.
  • FIG. 3 is a schematic structural diagram of a tag identification device provided by an embodiment of the present application.
  • Figure 4 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • Figure 1 is a schematic flowchart of a tag identification method provided by an embodiment of the present application. This method can be executed by a tag identification device provided by an embodiment of the present application.
  • the device can be implemented in software and/or hardware.
  • the device can be integrated in an electronic device, such as a server. The following embodiments will be described by taking the device integrated in an electronic device as an example. Referring to Figure 1, the method may include the following steps:
  • the labeling image can come from an image acquisition device.
  • the image acquisition device can be a camera, a video recorder, or other device with an image acquisition function.
  • the labeling image can be an image collected in real time or a pre-collected image; when the labeling image When the image is collected in real time, it can be an image collected in real time by the image collection device on the factory production line of the item or in a preset area.
  • the real-time image is used to identify the label of the item in the current scene in real time; when When the labeling image is a pre-collected image, the labeling image can be any image in the labeling image library.
  • the pre-collected labeling image is used to identify the item labels in the scene at that time.
  • the image to be recognized may be an image preprocessed by using preprocessing technology on the labeled image, and has higher definition than the labeled image.
  • the labeling image can be preprocessed to improve the quality of the labeling image and obtain the image to be recognized; for example, the obtained labeling image can be grayscaled and image enhanced. , tilt monitoring and correction, Gaussian filtering and other image processing methods to eliminate image noise, save computing resources, improve image quality and image processing speed, to ensure the accuracy and accuracy of label detection and identification of items.
  • the barcode of the item is first scanned to obtain the serial number of the item.
  • the serial number can be the number of the placement or storage location of the item, or the number corresponding to the production item.
  • the image acquisition device is used to obtain the labeling image of the item.
  • the labeling image contains the image content of the label to be identified on the item, and the serial number of the item is stored in association with the labeling image of the item to determine the item corresponding to the identified label.
  • S120 Use the label prediction network to predict labels in the image to be recognized, and obtain multiple predicted labels in the image to be recognized and the confidence of each predicted label in the multiple predicted labels.
  • the label prediction network may be a neural network used to identify labels of items in the image to be identified, where the label prediction network includes a convolutional network and a long short memory network, where the convolutional network is used to extract the labels in the image to be identified.
  • Image features and perform pixel-level image segmentation based on the image features in the image to be recognized, determine the detection area in the image to be recognized, and then obtain the feature vector in the detection area.
  • the detection area may be an image area corresponding to a label in the image to be recognized, and the feature vector in the detection area may be a vector formed by extracting text feature sequences from image features in the detection area.
  • the long short memory network can be used to capture the forward information and backward information of the feature vectors in the detection area output by the convolutional network, and fuse the feature vectors of the detection area based on the convolution layer to extract the text feature sequence.
  • Contextual features predict the label distribution of each feature vector in the text feature sequence and the probability distribution of each column of features.
  • the multiple prediction labels in the image to be recognized can be the long short-term memory network outputting multiple prediction results of the label in the image area corresponding to the label in the image to be recognized.
  • the confidence of each prediction label in the multiple prediction labels can be the long short-term memory network. Output the probability values corresponding to multiple prediction results of the label in the image area corresponding to the label in the image to be recognized.
  • the image to be recognized is input into the convolutional network in the label prediction network, the image feature information in the image to be recognized is extracted through the convolutional network, and the convolution layer in the convolutional network is upsampled to obtain the image to be recognized. Identify images with the same image size, use the one-to-one correspondence between the image to be recognized and the upsampled image, classify the upsampled image at the pixel level, recover the type of each pixel from the abstract features, and obtain the pixel level Two-dimensional heat map after classification.
  • Target detection and edge detection are performed based on the two-dimensional heat map, and the image is segmented at the pixel level based on the label features in the heat map to achieve precise positioning and segmentation of the detection area in the image to be identified, obtain the image features in the detection area, and Based on the image features in the detection area, text detection is performed to extract the text feature sequence, and then the feature vector of the image to be recognized is obtained.
  • the long and short memory network is used to capture the forward information and backward information of the text feature sequence from the feature vector of the image to be recognized, and the contextual features of the text feature sequence are extracted based on the feature vector of the image to be recognized, and the context feature pair detection of the text feature sequence is used
  • Label text in the area Prediction is performed to obtain multiple predicted labels in the image to be recognized and the confidence of each predicted label in the multiple predicted labels.
  • the convolutional network can use multiple convolutional layers to extract image features in the image to be recognized, and the convolutional network can use a fully convolutional network (FCN) to achieve any size.
  • FCN fully convolutional network
  • the image is convolved, and an image with the probability value of each pixel category can be obtained.
  • the feature map of the last convolution layer in the convolution network is upsampled, so that the image after passing the convolution layer can be restored and identified.
  • the images are of the same size, and the correlation between pixels is retained for the two images of the same size, where each pixel can generate a prediction.
  • multiple predicted labels and the confidence of each predicted label are input into the label determination model.
  • the output of the label determination model can be the content of the target label.
  • the label determination model can obtain the target of the item based on the confidence of each predicted label.
  • Label, the target label can be the text information in the detection area in the image to be recognized.
  • the text information can be the key information of the item, such as: the attribute information of the item, the functional information of the item, and the parameters of the item.
  • the label determination model can be trained based on training samples. Among them, the label content in the target label of the identified item can also be removed through spaces and duplication operations, so as to improve the quality of the target label of the identified item.
  • the labeling image of the item is obtained and the labeling image is preprocessed to obtain the image to be identified;
  • the label prediction network is used to predict the labels in the image to be identified, and multiple predicted labels in the image to be identified are obtained. and the confidence of each predicted label among the multiple predicted labels;
  • the label determination model is used to determine the target label of the item based on the multiple predicted labels and the confidence of each predicted label.
  • the quality of the labeling image is improved by preprocessing the labeling image, thereby improving the accuracy of label recognition.
  • the label prediction network is used to perform label recognition on the image to be identified, to obtain multiple prediction labels, and then the model is determined based on the labels. Screen or determine multiple prediction tags to obtain the target tag, which improves the efficiency and accuracy of identification. It can automatically identify the content in the tag without human participation and avoid the impact of the environment on the recognition effect.
  • the tag identification method provided by the embodiment of the present application is further described below. As shown in Figure 2, the method may include the following steps:
  • S220 Use the convolutional network to extract the feature vector of the detection area in the image to be recognized.
  • the detection area in the image to be recognized may be the image area corresponding to the label in the image to be recognized determined based on image segmentation at different pixel levels after feature extraction by the convolutional network.
  • the feature vector of the detection area in the image to be recognized may be a vector formed by performing text detection based on the image features of the detection area in the image to be recognized and then extracting the text feature sequence in the detection area. Input the image to be recognized into the convolutional network to obtain the image features of the image to be recognized. Based on the image features of the image to be recognized, the image after feature extraction is upsampled so that the size of the image after feature extraction is consistent with the size of the image to be recognized. And retain the correlation between the feature extracted image and the pixels of the image to be recognized.
  • the image after feature extraction is classified at the pixel level to obtain a two-dimensional heat map of the image to be identified.
  • the convolutional network includes a convolutional layer and a deconvolutional layer.
  • the convolutional network is used to extract the feature vector of the detection area in the image to be recognized, including:
  • the image features in the image to be recognized can be image feature information extracted from the image to be recognized using a convolution layer, and can be information such as boundary information, grayscale information, and text information of the objects in the image to be recognized.
  • the convolutional network includes a convolutional layer and a deconvolutional layer, where the convolutional layer is used to extract image features in the image to be recognized, and the deconvolutional layer is used to upsample the convolutional layer.
  • the convolutional image can be an image that is output after upsampling through the deconvolution layer and maintains the same size as the image to be recognized.
  • the convolutional image retains The relationship between the pixels of the image to be recognized is stored.
  • the two-dimensional heat map can be used to distinguish different pixel categories in the image to be identified. It can intuitively see the location of different pixels.
  • the two-dimensional heat map can be used to accurately locate and segment the detection area.
  • the image to be recognized is input into the convolutional network to obtain the image features of the image to be recognized.
  • the image after feature extraction is upsampled, so that the image after feature extraction and the image to be recognized are The size is consistent and the relationship between the pixels of the image after feature extraction and the image to be recognized is retained.
  • Classify the pixels of the image after feature extraction to obtain a two-dimensional heat map of the image to be identified. Perform pixel-level image segmentation on the image to be identified based on the two-dimensional heat map, determine the detection area in the image to be identified based on the label features, and perform text detection on the image features in the detection area to extract the text in the image features in the detection area. Text features to obtain the feature vector of the detection area.
  • the semantic relationship of the feature vector can be the forward information and backward information of the feature vector in the detection area, where the forward information can be the forward semantic transmission information corresponding to the label content to be predicted in the detection area, that is, The previous semantic information associated with the label content to be predicted in the detection area.
  • the backward information can be the information that is semantically transmitted backwards corresponding to the label content to be predicted in the detection area, that is, the subsequent semantic information associated with the label content to be predicted in the detection area.
  • the label information in the image is 123456X890, where "X" is the label content to be predicted, "12345” is the forward information, and "890" is the backward information.
  • the forward information can also be before the label content to be predicted.
  • the backward information may also be the characteristic information corresponding to the backward content of the label content to be predicted.
  • the text characteristics of the feature vector can be the arrangement pattern of the text sequence of the feature vector.
  • the feature vector of the detection area is input into the long and short memory network.
  • the long and short memory network captures the forward information and backward information of the character feature sequence in the feature vector of the detection area output by the convolution network, and obtains the semantic relationship of the feature vector, and According to the semantic relationship of the feature vector and the feature vector of the detection area, the text features of the label content to be predicted in the detection area are extracted after feature fusion. Prediction based on text features Labels in the image to be recognized are used to obtain multiple predicted labels in the image to be recognized and the confidence of each predicted label in the multiple predicted labels.
  • the long and short memory network is a neural network trained through training samples, in which the content of the label to be predicted in the detection area can be predicted through text features.
  • the label content to be predicted can be the label content in the detection area of the image to be recognized that has not been recognized by the convolutional network. Content prediction needs to be performed based on the long and short memory network.
  • the label content to be predicted can be a character, or it can be Multiple characters, when the content of the label to be predicted is one character, the position of the character can be predicted, that is, the multiple prediction labels and the confidence of each prediction label in the multiple prediction labels can be the prediction and confidence of the character at the position degree; when the content of the label to be predicted is multiple characters, the predicted characters and confidence of the multiple character positions can be obtained after predicting the multiple character positions respectively, and the prediction is performed based on the predicted characters and confidence of the multiple character positions. Calculate and obtain multiple predicted labels and the confidence of each predicted label in the multiple predicted labels.
  • Predict labels in the image to be recognized based on text features and obtain multiple predicted labels in the image to be recognized and the confidence of each predicted label in the multiple predicted labels.
  • using the label prediction network to predict labels in the image to be recognized also includes:
  • Input the image to be recognized into the label prediction network obtain the recognition error information of the image to be recognized, and perform fault warning based on the recognition error information.
  • the recognition error message may be the corresponding message sent when the label prediction network fails to predict the label content in the image to be recognized.
  • the recognition error message is used to prompt that the label in the image to be recognized cannot be recognized or the label of the image to be recognized cannot satisfy the label prediction.
  • the prediction conditions of the network, and triggering the alarm function based on the recognition error information for fault warning require the label prediction network to be retrained or the image to be recognized to be re-photographed or pre-processed.
  • the label determination model is obtained as follows:
  • training samples where the training samples include multiple recognition images, each recognition image includes multiple prediction labels and the confidence of each prediction label in the multiple prediction labels;
  • Backpropagation is performed according to the loss function to optimize the time series classification model and obtain the label determination model.
  • the training sample may be a set of images containing label recognition content collected according to the label recognition content, and multiple recognition images in the training sample are predicted in advance through the label prediction network.
  • Each recognition image contains multiple prediction tags and multiple The confidence of each predicted label is marked, and the actual label corresponding to each recognized image is marked in each recognized image.
  • the time series classification model may be a neural network based on the time series category, and is set to filter or identify multiple prediction labels and the confidence of each prediction label among the multiple prediction labels, and obtain the target label corresponding to each recognized image.
  • the actual label corresponding to each recognition image may be the actual label content corresponding to each recognition image in the training sample, which is used for model training of the label determination model.
  • multiple recognition images in the training sample are obtained, the multiple recognition images are preprocessed in advance, and the preprocessed multiple recognition images are input into the label prediction network to obtain multiple prediction labels and multiple recognition images for each recognition image.
  • the confidence of each predicted label is input into the time series classification model for label determination, and the target label corresponding to each recognition image is obtained.
  • the loss function is calculated based on the target label corresponding to each recognition image and the actual label corresponding to each recognition image, and the entropy value of the loss function is obtained.
  • the attribute information may be information with item attributes in the target tag content of the item, such as: energy consumption label model, nameplate number, comprehensive power consumption, etc.
  • the information database can be a database that stores information related to items. It can be used in the factory's manufacturing execution system to store the item's barcode, label image and attribute information in the value information database in the form of associated storage to facilitate subsequent traceability of the item. and query. Among them, after completing the tag identification of the target tag of the item, regularized filtering, search algorithm, etc. can be used to process the identified target tag and extract the attribute information of the item from the target tag of the item.
  • the labeling image of the item is obtained and the labeling image is preprocessed to obtain the image to be identified; the label prediction network is used to predict the labels in the image to be identified, and multiple predicted labels of the image to be identified are obtained.
  • the determination model screens multiple prediction labels or determines the target label to improve the efficiency and accuracy of identification. It can automatically identify the content in the label without human participation and avoid the impact of the environment on the recognition effect.
  • FIG 3 is a schematic structural diagram of a tag identification device provided by an embodiment of the present application. As shown in Figure 3, the tag identification device includes:
  • the image acquisition module 310 is configured to acquire the label image of the item, and preprocess the label image to obtain the image to be recognized;
  • the network prediction module 320 is configured to use the label prediction network to predict the labels in the image to be recognized, Obtain multiple prediction labels in the image to be recognized and the confidence of each prediction label in the plurality of prediction labels;
  • the label determination module 330 is configured to use a label determination model to determine the target label of the item based on the plurality of predicted labels and the confidence of each predicted label.
  • the label prediction network of the network prediction module 320 includes a convolutional network and a long and short memory network.
  • the label prediction network is used to predict labels in the image to be recognized, and multiple labels in the image to be recognized are obtained. prediction labels and the confidence of each prediction label among the plurality of prediction labels, including:
  • the long and short memory network is used to extract the text features of the feature vector according to the semantic relationship of the feature vector, and the labels in the image to be recognized are predicted based on the text features to obtain multiple predictions in the image to be recognized. label and the confidence of each of the plurality of predicted labels.
  • the convolutional network of the network prediction module 320 includes a convolutional layer and a deconvolutional layer. Using the convolutional network to extract the feature vector of the detection area in the image to be recognized includes:
  • the image to be recognized is input into a convolution layer to extract image features in the image to be recognized, and the deconvolution layer is used to upsample the convolution layer to obtain a convolution image, which is the same as the convolution image. There is a correlation between the pixels of the image to be recognized;
  • the network prediction module 320 uses the long and short memory network to extract the text features of the feature vector according to the semantic relationship of the feature vector, and predicts the text features in the image to be recognized based on the text features. label, obtain multiple predicted labels in the image to be recognized and the confidence of each predicted label in the multiple predicted labels, including:
  • the label determination module 330 uses the label determination model to determine the target label of the item based on the multiple predicted labels and the confidence of each predicted label, it also includes:
  • Attribute information is extracted from the target label of the item, and the attribute information and the label image are associated and stored in an information database, so that the attribute information of the item is queried according to the label image.
  • the label determination model in the label determination module 330 is obtained as follows:
  • training samples wherein the training samples include a plurality of recognition images, each of the recognition images includes a plurality of prediction labels and the confidence of each prediction label in the plurality of prediction labels;
  • a time series classification model is used to screen multiple prediction labels in each recognition image in the training sample and the confidence of each prediction label in the multiple prediction labels to obtain the target label corresponding to each recognition image.
  • Back propagation is performed according to the loss function to optimize the temporal class classification model to obtain the label determination model.
  • the network prediction module 320 uses a label prediction network to predict labels in the image to be recognized, and further includes:
  • the image to be recognized is input into the label prediction network, the recognition error information of the image to be recognized is obtained, and a fault warning is performed based on the recognition error information.
  • the device of the embodiment of the present application acquires the labeling image of the item and performs preprocessing on the labeling image to obtain the image to be identified; it uses the label prediction network to predict the labels in the image to be identified, and obtains multiple predicted labels in the image to be identified. and the confidence of each of the multiple predicted labels; use the labels to determine The model determines the target label of the item based on multiple predicted labels and the confidence of each predicted label.
  • the embodiment of the present application improves the quality of the labeling image by preprocessing the labeling image, thereby improving the accuracy of label recognition, and performs label recognition on the image to be recognized through the label prediction network to obtain multiple prediction labels, and then combines the labels
  • the determination model screens multiple prediction labels or determines the target label to improve the efficiency and accuracy of identification. It can automatically identify the content in the label without human participation and avoid the impact of the environment on the recognition effect.
  • FIG. 4 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. 4 illustrates a block diagram of an exemplary electronic device 12 suitable for implementing embodiments of the present application.
  • the electronic device 12 shown in FIG. 4 is only an example and should not bring any limitations to the functions and scope of use of the embodiments of the present application.
  • electronic device 12 is embodied in the form of a general-purpose computing device.
  • the components of electronic device 12 may include, but are not limited to: at least one processor or processing unit 16, system memory 28, and a bus 18 connecting various system components (including system memory 28 and processing unit 16).
  • Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics accelerated port, a processor, or a local bus using any of a variety of bus structures.
  • these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (Video Electronics Standards) Association, VESA) local bus and Peripheral Component Interconnect (PCI) bus.
  • Electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 12, including volatile and nonvolatile media, removable and non-removable media.
  • System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and/or cache memory 32 .
  • Electronic device 12 may further include other removable/non-removable, volatile/non-volatile computers System storage media.
  • storage system 34 may be used to read and write to non-removable, non-volatile magnetic media (not shown in Figure 4, commonly referred to as a "hard drive").
  • a disk drive for reading and writing to a removable non-volatile disk may be provided, as well as a disk drive for reading and writing to a removable non-volatile optical disk (such as a Compact Disc- Read Only Memory, CD-ROM), digital video disc (Digital Video Disc-Read Only Memory, DVD-ROM) or other optical media) read and write optical disc drive.
  • a removable non-volatile optical disk such as a Compact Disc- Read Only Memory, CD-ROM), digital video disc (Digital Video Disc-Read Only Memory, DVD-ROM) or other optical media
  • each drive may be connected to bus 18 via at least one data media interface.
  • System memory 28 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of various embodiments of the present application.
  • a program/utility 40 having a set of (at least one) program modules 42 may be stored, for example, in system memory 28. Each of these examples, or some combination, may include the implementation of a network environment.
  • Program modules 42 generally perform functions and/or methods in the embodiments described herein.
  • Electronic device 12 may also communicate with at least one external device 14 (e.g., keyboard, pointing device, display 24, etc.), may also communicate with at least one device that enables a user to interact with electronic device 12, and/or with electronic device 12 that enables Any device (eg, network card, modem, etc.) that can communicate with at least one other computing device. This communication may occur through an input/output (I/O) interface 22 .
  • the electronic device 12 can also communicate with at least one network (such as a local area network (Local Area Network, LAN), a wide area network (Wide Area Network, WAN) and/or a public network, such as the Internet) through the network adapter 20.
  • LAN Local Area Network
  • WAN Wide Area Network
  • public network such as the Internet
  • network adapter 20 communicates with other modules of electronic device 12 via bus 18 .
  • other hardware and/or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays). of Independent Disks, RAID) systems, tape drives and data backup storage systems, etc.
  • Processing unit 16 performs various functions by executing programs stored in system memory 28
  • Application and data processing such as implementing the tag identification method provided by the embodiment of this application, the method includes:
  • a label prediction network to predict labels in the image to be identified, and obtain multiple predicted labels in the image to be identified and the confidence of each predicted label in the multiple predicted labels;
  • a label determination model is used to determine a target label for the item based on the plurality of predicted labels and the confidence of each predicted label.
  • Embodiments of the present application also provide a computer-readable storage medium on which a computer program is stored.
  • the program When executed by a processor, it implements the tag identification method as described.
  • the method includes:
  • a label prediction network to predict labels in the image to be identified, and obtain multiple predicted labels in the image to be identified and the confidence of each predicted label in the multiple predicted labels;
  • a label determination model is used to determine a target label for the item based on the plurality of predicted labels and the confidence of each predicted label.
  • the computer storage medium in the embodiment of the present application may be any combination of one or more computer-readable media.
  • the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium.
  • the computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer readable storage media include: electrical connections having one or more conductors, portable computer disks, hard drives, random access memory (RAM), read only memory (ROM), Erasable Programmable Read-Only Memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any of the above suitable The combination.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above.
  • a computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device .
  • Program code embodied on a computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, radio frequency (Radio Frequency, RF), etc., or any suitable combination of the above.
  • any appropriate medium including but not limited to wireless, wire, optical cable, radio frequency (Radio Frequency, RF), etc., or any suitable combination of the above.
  • Computer program code for performing operations of the present application may be written in one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedures, or a combination thereof.
  • programming language - such as "C” or a similar programming language.
  • the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
  • the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as through the Internet using an Internet service provider) .
  • LAN local area network
  • WAN wide area network
  • Internet service provider such as AT&T, MCI, Sprint, EarthLink, MSN, GTE, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Image Analysis (AREA)

Abstract

本申请公开了一种标签识别方法、装置、电子设备及存储介质。该方法包括：获取物品的贴标图像，并对贴标图像进行预处理，得到待识别图像；利用标签预测网络预测待识别图像中的标签，得到待识别图像中的多个预测标签及多个预测标签中每个预测标签的置信度；利用标签确定模型基于多个预测标签及每个预测标签的置信度确定物品的目标标签。

Description

一种标签识别方法、装置、电子设备及存储介质
本申请要求在2022年6月15日提交中国专利局、申请号为202210676549.7的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及计算机技术，例如涉及一种标签识别方法、装置、电子设备及存储介质。
背景技术
工厂的产线中的质检环节，一般由质检员完成，以防止不良产品流入市场，质检环节涉及到附件贴标型号确认、产品部件装配质量确认、产品表面划痕缺陷检测等流程，需要多名质检人员配合完成。工厂产线上大多采用混产的方式，一些产品仅是内部零部件型号不同，外观尺寸区别不是很大，导致产品型号、贴标种类繁多，且诸如能耗帖、产品铭牌、警示贴、耗电量等，字体小、编号长，质检人员核对耗时，不同产品的不同贴标极易混淆，漏检、误检的情况也时有发生，追溯也比较困难。针对上述问题相关技术大多采用光学字符识别(Optical Character Recognition，OCR)方法来替代人工进行自动识别贴标图像中的文字内容，可以有效提高检测效率和准确率，由于传统OCR识别方法获取到的图像预处理步骤较多，易产生误差从而降低识别率，同时多个环节需要人工参与，难以实现自动化，而且识别灵活性较差，对复杂样本识别率较低。
发明内容
本申请提供一种标签识别方法、装置、电子设备及存储介质，以实现在无人员参与的情况下，自动识别标签中的内容。
第一方面，本申请实施例提供了一种标签识别方法，包括：
获取物品的贴标图像，并对所述贴标图像进行预处理，得到待识别图像；
利用标签预测网络预测所述待识别图像中的标签，得到所述待识别图像中的多个预测标签及所述多个预测标签中每个预测标签的置信度；
利用标签确定模型基于所述多个预测标签及所述每个预测标签的置信度确定所述物品的目标标签。
第二方面，本申请实施例还提供了一种标签识别装置，该装置包括：
图像获取模块，设置为获取物品的贴标图像，并对所述贴标图像进行预处理，得到待识别图像；
网络预测模块，设置为利用标签预测网络预测所述待识别图像中的标签，得到所述待识别图像中的多个预测标签及所述多个预测标签中每个预测标签的置信度；
标签确定模块，设置为利用标签确定模型基于所述多个预测标签及所述每个预测标签的置信度确定所述物品的目标标签。
第三方面，本申请实施例还提供了一种电子设备，该电子设备包括：
至少一个处理器；
存储装置，设置为存储至少一个程序，
当所述至少一个程序被所述至少一个处理器执行，使得所述至少一个处理器实现所述的标签识别方法。
第四方面，本申请实施例还提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现所述的标签识别方法。
附图说明
图1是本申请实施例提供的标签识别方法的一个流程示意图；
图2是本申请实施例提供的标签识别方法的另一流程示意图；
图3是本申请实施例提供的标签识别装置的一个结构示意图；
图4是本申请实施例提供的电子设备的一个结构示意图。
具体实施方式
下面结合附图和实施例对本申请作详细说明。
图1为本申请实施例提供的标签识别方法的一个流程示意图，该方法可以由本申请实施例提供的标签识别装置来执行，该装置可采用软件和/或硬件的方式实现。在一个具体的实施例中，该装置可以集成在电子设备中，电子设备比如可以是服务器。以下实施例将以该装置集成在电子设备中为例进行说明，参考图1，该方法可以包括如下步骤：
S110、获取物品的贴标图像，并对贴标图像进行预处理，得到待识别图像。
示例地，贴标图像可以来自图像采集设备，图像采集设备可以是摄像头、录像机等具有图像采集功能的设备，贴标图像可以是实时采集的图像，也可以是预先采集的图像；当贴标图像为实时采集的图像时，可以是图像采集设备在物品的工厂产线上或预设区域上实时采集的图像，这种情况下，即利用实时图像实时对当前场景中物品的标签进行识别；当贴标图像为预先采集的图像时，贴标图像可以是贴标图像库中任意一张图像，这种情况下，即对预先采集的贴标图像识别当时场景中的物品标签，可以是预先采集的贴标图像中的物品标签内容无法查找或丢失的情况下。待识别图像可以是利用预处理技术对贴标图像进行预处理后的图像，具有较贴标图像更高的清晰度。
具体实现中，在获取物品的贴标图像之后，可以对贴标图像进行预处理，以改善贴标图像质量，得到待识别图像；比如：可以对获取的贴标图像进行灰度化、图像增强、倾斜监测与校正、高斯滤波等图像处理方法，消除图像噪点、节约计算资源、提高图像质量和图像处理速度，以保证物品的标签检测和识别准确度和精度。其中，先对物品条码进行扫描获取物品的序列号，该序列号可以是物品摆放或存放位置的编号，也可以是生产物品对应的编号，同时利用图像采集设备获取物品的贴标图像，该贴标图像中包含物品待识别标签的图像内容，并将物品的序列号与物品的贴标图像关联存储，以确定识别出标签对应的物品。
S120、利用标签预测网络预测待识别图像中的标签，得到待识别图像中的多个预测标签及多个预测标签中每个预测标签的置信度。
示例地，标签预测网络可以是用于对待识别图像中物品的标签进行识别的神经网络，其中，标签预测网络包括卷积网络和长短记忆网络，其中，卷积网络用于提取待识别图像中的图像特征，并根据待识别图像中图像特征进行像素级别的图像分割，确定出待识别图像中的检测区域，进而得到检测区域内的特征向量。其中，检测区域可以是待识别图像中标签所对应的图像区域，检测区域内的特征向量可以是检测区域内通过图像特征提取文字特征序列形成的向量。其中，长短记忆网络可以是用于捕获卷积网络输出的检测区域内的特征向量的前向信息和后向信息，并在卷积层的基础上融合检测区域的特征向量以提取文字特征序列的上下文特征，预测文字特征序列中的每一个特征向量标签分布及每列特征的概率分布。待识别图像中的多个预测标签可以是长短记忆网络输出待识别图像中标签所对应的图像区域中标签的多个预测结果，多个预测标签中每个预测标签的置信度可以是长短记忆网络输出待识别图像中标签所对应的图像区域中标签的多个预测结果对应的概率值。
具体实现中，将待识别图像输入标签预测网络中的卷积网络中，通过卷积网络提取待识别图像中的图像特征信息，并通过对卷积网络中卷积层进行上采样，得到与待识别图像尺寸一致的图像，利用待识别图像与上采样后的图像进行一一对应，将上采样后的图像进行像素级别的分类，从抽象的特征中恢复每个像素所属的类型，得到像素级别分类后的二维热力图。基于二维热力图进行目标检测和边缘检测，根据热力图中标签特征对图像进行像素级别的图像分割，实现对待识别图像中检测区域的精准定位和精准分割，获取检测区域内的图像特征，并根据检测区域内的图像特征进行文本检测提取文字特征序列，进而得到待识别图像的特征向量。利用长短记忆网络从待识别图像的特征向量中捕获文字特征序列的前向信息和后向信息，并基于待识别图像的特征向量提取文字特征序列的上下文特征，利用文字特征序列的上下文特征对检测区内的标签文 本进行预测，得到待识别图像中的多个预测标签及多个预测标签中每个预测标签的置信度。
本申请实施例中，卷积网络中可以是由多个卷积层进行待识别图像中图像特征的提取，卷积网络可以采用全卷积网络(Fully Convolutional Network，FCN)，以实现对任意尺寸图像进行卷积，并可以得到标好每个像素类别概率值的图像，针对卷积网络中最后一个卷积层的特征图进行上采样，以使得通过卷积层后的图像可以恢复与待识别图像相同的尺寸大小，并对相同尺寸大小的两张图像保留像素间的关联关系，其中，每一个像素都可以产生一个预测。
S130、利用标签确定模型基于多个预测标签及每个预测标签的置信度确定物品的目标标签。
具体实现中，将多个预测标签及每个预测标签的置信度输入标签确定模型，标签确定模型输出可以是目标标签中内容，标签确定模型可以根据每个预测标签的置信度，得到物品的目标标签，目标标签可以是待识别图像中检测区域内的文本信息，该文本信息可以是物品的关键信息，比如：物品的属性信息、物品的功能信息、物品的参数。标签确定模型可以是根据训练样本训练得到的。其中，还可以将确定出的物品的目标标签中的标签内容，经过去除空格和去重操作，以便于提高识别出物品的目标标签的质量。
本申请实施例中，通过获取物品的贴标图像，并对贴标图像进行预处理，得到待识别图像；利用标签预测网络预测待识别图像中的标签，得到待识别图像中的多个预测标签及多个预测标签中每个预测标签的置信度；利用标签确定模型基于多个预测标签及每个预测标签的置信度确定物品的目标标签。本申请实施例，通过对贴标图像进行预处理改善贴标图像质量，进而提高标签识别的准确度，并通过标签预测网络对待识别图像进行标签识别，得到多个预测标签，再结合标签确定模型对多个预测标签进行筛选或确定得到目标标签，提升识别的效率和准确率，可以在无人员参与的情况下，自动识别标签中的内容，避免环境对识别效果的影响。
下面进一步描述本申请实施例提供的标签识别方法，如图2所示，该方法可以包括如下步骤：
S210、获取物品的贴标图像，并对贴标图像进行预处理，得到待识别图像。
S220、利用卷积网络提取待识别图像中检测区域的特征向量。
具体实现中，待识别图像中检测区域可以是卷积网络特征提取后根据不同像素级别图像分割确定出的待识别图像中标签所对应的图像区域。待识别图像中检测区域的特征向量可以是根据待识别图像中检测区域的图像特征进行文本检测后提取检测区域内文本特征序列形成的向量。将待识别图像输入卷积网络，得到待识别图像的图像特征，基于待识别图像的图像特征，并对特征提取后图像进行上采样，使得特征提取后的图像和待识别图像的尺寸大小一致，并保留特征提取后的图像和待识别图像像素间的关联关系。对特征提取后的图像进行像素级别的分类，得到待识别图像的二维热力图。根据二维热力图对待识别图像进行像素级别的图像分割，并根据标签特征确定出待识别图像中检测区域，并根据对检测区域内的图像特征进行文本检测，提取检测区域的特征向量。
可选的，卷积网络包括卷积层和反卷积层，利用卷积网络提取待识别图像中检测区域的特征向量，包括：
将待识别图像输入卷积层提取待识别图像中图像特征，并使用反卷积层对卷积层进行上采样，得到卷积图像，卷积图像与待识别图像的像素间存在关联；
对卷积图像进行像素分类，得到待识别图像的二维热力图，并根据二维热力图对待识别图像进行图像分割，确定出待识别图像中的检测区域，对检测区域内的图像特征进行文本检测，得到检测区域的特征向量。
示例地，待识别图像中图像特征可以是利用卷积层从待识别提取到的图像特征信息，可以是待识别图像中物体的边界信息、灰度信息和文字信息等信息。其中，卷积网络包括卷积层和反卷积层，其中，卷积层用于提取待识别图像中图像特征，反卷积层用于对卷积层进行上采样。卷积图像可以是通过反卷积层进行上采样之后的输出的与待识别图像保持同样尺寸大小的图像，卷积图像保 存了与待识别图像像素之间的关系。二维热力图可以是用于区分待识别图像中不同像素类别的图像，可以直观的看出不同像素所在的区域位置，可以利用二维热力图精准定位和分割出检测区域。
具体实现中，将待识别图像输入卷积网络，得到待识别图像的图像特征，基于待识别图像的图像特征，并对特征提取后图像进行上采样，使得特征提取后的图像和待识别图像的尺寸大小一致，并保留特征提取后的图像和待识别图像像素间的关系。对特征提取后的图像进行像素的分类，得到待识别图像的二维热力图。根据二维热力图对待识别图像进行像素级别的图像分割，并根据标签特征确定出待识别图像中检测区域，并根据对检测区域内的图像特征进行文本检测，提取检测区域内的图像特征中的文字特征，得到检测区域的特征向量。
S230、利用长短记忆网络根据特征向量的语义关系提取特征向量的文字特征，并根据文字特征预测待识别图像中的标签，得到待识别图像中的多个预测标签及多个预测标签中每个预测标签的置信度。
示例地，特征向量的语义关系可以是检测区域内的特征向量的前向信息和后向信息，其中，前向信息可以是检测区域内待预测标签内容对应的向前进行语义传递的信息，即检测区域内待预测标签内容相关联的前面语义信息，后向信息可以是检测区域内待预测标签内容对应的向后进行语义传递的信息，即检测区域内待预测标签内容相关联的后面语义信息，比如：图像中的标签信息为123456X890，其中“X”是待预测标签内容，“12345”为前向信息，“890”为后向信息，其中，前向信息也可以是待预测标签内容前向内容对应的特征信息，后向信息也可以是待预测标签内容后向内容对应的特征信息。特征向量的文字特征可以是特征向量文字序列排列规律。
具体实现中，将检测区域的特征向量输入长短记忆网络，长短记忆网络捕获卷积网络输出的检测区域的特征向量中文字特征序列的前向信息和后向信息，得到特征向量的语义关系，并根据特征向量的语义关系和检测区域的特征向量进行特征融合后提取检测区域待预测标签内容的文字特征。根据文字特征预测 待识别图像中的标签，得到待识别图像中的多个预测标签及多个预测标签中每个预测标签的置信度。其中，长短记忆网络是通过训练样本进行训练后的神经网络，其中，可以通过文字特征预测检测区域中待预测标签的内容。其中，待预测标签内容可以是通过卷积网络未识别出的待识别图像中检测区域内的标签内容，需要根据长短记忆网络进行内容预测，其中，待预测标签内容可以是一个字符，也可以是多个字符，当待预测标签内容是一个字符时，可以通过对该字符位置进行预测，即多个预测标签及多个预测标签中每个预测标签的置信度可以是对该位置字符预测和置信度；当待预测标签内容是多个字符时，可以是分别对多个字符位置进行预测后，分别得到多个字符位置的预测字符和置信度，根据多个字符位置的预测字符和置信度进行计算，得到多个预测标签及多个预测标签中每个预测标签的置信度。
可选的，利用长短记忆网络根据特征向量的语义关系提取特征向量的文字特征，并根据文字特征预测待识别图像中的标签，得到待识别图像中的多个预测标签及多个预测标签中每个预测标签的置信度，包括：
将检测区域的特征向量输入长短记忆网络，得到特征向量的语义关系，并根据特征向量的语义关系提取特征向量的文字特征；
根据文字特征预测待识别图像中的标签，得到待识别图像中的多个预测标签及多个预测标签中每个预测标签的置信度。
可选的，利用标签预测网络预测待识别图像中的标签，还包括：
将待识别图像输入至标签预测网络，得到待识别图像的识别错误信息，并根据识别错误信息进行故障预警。
示例地，识别错误信息可以是在标签预测网络未预测出待识别图像中的标签内容时对应发出的信息，识别错误信息用于提示待识别图像中标签无法识别或待识别图像标签不能满足标签预测网络的预测条件，并根据识别错误信息触发报警功能进行故障预警，需要标签预测网络进行重新训练或者对待识别图像进行重新拍摄或者预处理。
S240、利用标签确定模型基于多个预测标签及每个预测标签的置信度确定物品的目标标签。
可选的，标签确定模型按照如下方式获取：
获取训练样本，其中，训练样本包含多张识别图像，每张识别图像中包含多个预测标签及多个预测标签中每个预测标签的置信度；
利用时序类分类模型对训练样本中的每张识别图像中多个预测标签及多个预测标签中每个预测标签的置信度进行筛选，得到每张识别图像对应的目标标签；
根据每张识别图像对应的目标标签和每张识别图像对应的实际标签计算损失函数；
根据损失函数进行反向传播以优化时序类分类模型，得到标签确定模型。
示例地，训练样本可以是根据标签识别内容搜集的包含标签识别内容的图像集，并预先对训练样本中多张识别图像通过标签预测网络预测出的每张识别图像中包含多个预测标签及多个预测标签中每个预测标签的置信度进行标记，并在每张识别图像中标记每张识别图像对应的实际标签。时序类分类模型可以是根据时序类类别的神经网络，设置为对多个预测标签及多个预测标签中每个预测标签的置信度进行筛选或识别，得到每张识别图像对应的目标标签。每张识别图像对应的实际标签可以是训练样本中每张识别图像对应的实际标签内容，用于对标签确定模型进行模型训练。
具体实现中，获取训练样本中多张识别图像，预先对多张识别图像进行预处理，并将预处理后的多张识别图像输入标签预测网络，得到每张识别图像的多个预测标签及多个预测标签中每个预测标签的置信度。将每张识别图像的多个预测标签及多个预测标签中每个预测标签的置信度输入时序类分类模型进行标签确定，得到每张识别图像对应的目标标签。根据每张识别图像对应的目标标签和每张识别图像对应的实际标签计算损失函数，得到损失函数的熵值。根据损失函数的熵值确定时序类分类模型是否收敛，并根据损失函数的熵值进行 反向传播以优化时序类分类模型的参数，直至损失函数的熵值小于预设熵值阈值，确定时序类分类模型收敛，得到标签确定模型。
S250、从物品的目标标签中提取属性信息，并将属性信息和贴标图像关联存储至信息数据库中，以使得根据贴标图像查询物品的属性信息。
具体实现中，属性信息可以是物品的目标标签内容中具有物品属性的信息，比如：能耗帖型号、铭牌编号、综合耗电量等。信息数据库可以是存储物品相关信息的数据库，可以用于工厂的制造执行系统中，通过关联存储的形式将物品的条码、贴标图像和属性信息存储值信息数据库中，以便于后续对物品进行追溯和查询。其中，完成对物品的目标标签进行标签识别后，可以采用正则化过滤、搜索算法等处理识别的目标标签，提取物品的目标标签中物品的属性信息。
本申请实施例中，通过获取物品的贴标图像，并对贴标图像进行预处理，得到待识别图像；利用标签预测网络预测待识别图像中的标签，得到待识别图像的多个预测标签及多个预测标签中每个预测标签的置信度；利用标签确定模型基于多个预测标签及每个预测标签的置信度确定物品的目标标签。即，本申请实施例，通过对贴标图像进行预处理改善贴标图像质量，进而提高标签识别的准确度，并通过标签预测网络对待识别图像进行标签识别，得到多个预测标签，再结合标签确定模型对多个预测标签进行筛选或确定得到目标标签，提升识别的效率和准确率，可以在无人员参与的情况下，自动识别标签中的内容，避免环境对识别效果的影响。
图3是本申请实施例提供的标签识别装置的结构示意图，如图3所示，该标签识别装置包括：
图像获取模块310，设置为获取物品的贴标图像，并对所述贴标图像进行预处理，得到待识别图像；
网络预测模块320，设置为利用标签预测网络预测所述待识别图像中的标签， 得到所述待识别图像中的多个预测标签及所述多个预测标签中每个预测标签的置信度；
标签确定模块330，设置为利用标签确定模型基于所述多个预测标签及所述每个预测标签的置信度确定所述物品的目标标签。
一实施例中，所述网络预测模块320所述标签预测网络包括卷积网络和长短记忆网络，所述利用标签预测网络预测所述待识别图像中的标签，得到所述待识别图像中的多个预测标签及所述多个预测标签中每个预测标签的置信度，包括：
利用所述卷积网络提取所述待识别图像中检测区域的特征向量；
利用所述长短记忆网络根据所述特征向量的语义关系提取所述特征向量的文字特征，并根据所述文字特征预测所述待识别图像中的标签，得到所述待识别图像中的多个预测标签及所述多个预测标签中每个预测标签的置信度。
一实施例中，所述网络预测模块320所述卷积网络包括卷积层和反卷积层，所述利用所述卷积网络提取所述待识别图像中检测区域的特征向量，包括：
将所述待识别图像输入卷积层提取所述待识别图像中图像特征，并使用所述反卷积层对所述卷积层进行上采样，得到卷积图像，所述卷积图像与所述待识别图像的像素间存在关联；
对所述卷积图像进行像素分类，得到所述待识别图像的二维热力图，并根据所述二维热力图对所述待识别图像进行图像分割，确定出所述待识别图像中的检测区域，对所述检测区域内的图像特征进行文本检测，得到所述检测区域的特征向量。
一实施例中，所述网络预测模块320所述利用所述长短记忆网络根据所述特征向量的语义关系提取所述特征向量的文字特征，并根据所述文字特征预测所述待识别图像中的标签，得到所述待识别图像中的多个预测标签及所述多个预测标签中每个预测标签的置信度，包括：
将所述检测区域的特征向量输入所述长短记忆网络，得到所述特征向量的 语义关系，并根据所述特征向量的语义关系提取所述特征向量的文字特征；
根据所述文字特征预测所述待识别图像中的标签，得到所述待识别图像中的多个预测标签及所述多个预测标签中每个预测标签的置信度。
一实施例中，所述标签确定模块330利用标签确定模型基于所述多个预测标签及所述每个预测标签的置信度确定所述物品的目标标签之后，还包括：
从所述物品的目标标签中提取属性信息，并将所述属性信息和所述贴标图像关联存储至信息数据库中，以使得根据所述贴标图像查询所述物品的属性信息。
一实施例中，所述标签确定模块330中所述标签确定模型按照如下方式获取：
获取训练样本，其中，所述训练样本包含多张识别图像，所述每张识别图像中包含多个预测标签及所述多个预测标签中每个预测标签的置信度；
利用时序类分类模型对所述训练样本中的每张识别图像中多个预测标签及所述多个预测标签中每个预测标签的置信度进行筛选，得到所述每张识别图像对应的目标标签；
根据所述每张识别图像对应的目标标签和所述每张识别图像对应的实际标签计算损失函数；
根据所述损失函数进行反向传播以优化所述时序类分类模型，得到所述标签确定模型。
一实施例中，所述网络预测模块320利用标签预测网络预测所述待识别图像中的标签，还包括：
将所述待识别图像输入至所述标签预测网络，得到所述待识别图像的识别错误信息，并根据所述识别错误信息进行故障预警。
本申请实施例装置，通过获取物品的贴标图像，并对贴标图像进行预处理，得到待识别图像；利用标签预测网络预测待识别图像中的标签，得到待识别图像中的多个预测标签及多个预测标签中每个预测标签的置信度；利用标签确定 模型基于多个预测标签及每个预测标签的置信度确定物品的目标标签。即，本申请实施例，通过对贴标图像进行预处理改善贴标图像质量，进而提高标签识别的准确度，并通过标签预测网络对待识别图像进行标签识别，得到多个预测标签，再结合标签确定模型对多个预测标签进行筛选或确定得到目标标签，提升识别的效率和准确率，可以在无人员参与的情况下，自动识别标签中的内容，避免环境对识别效果的影响。
图4为本申请实施例提供的一种电子设备的结构示意图。图4示出了适于用来实现本申请实施方式的示例性电子设备12的框图。图4显示的电子设备12仅仅是一个示例，不应对本申请实施例的功能和使用范围带来任何限制。
如图4所示，电子设备12以通用计算设备的形式表现。电子设备12的组件可以包括但不限于：至少一个处理器或者处理单元16，系统存储器28，连接不同系统组件(包括系统存储器28和处理单元16)的总线18。
总线18表示几类总线结构中的一种或多种，包括存储器总线或者存储器控制器，外围总线，图形加速端口，处理器或者使用多种总线结构中的任意总线结构的局域总线。举例来说，这些体系结构包括但不限于工业标准体系结构(Industry Standard Architecture，ISA)总线，微通道体系结构(Micro Channel Architecture，MCA)总线，增强型ISA总线、视频电子标准协会(Video Electronics Standards Association，VESA)局域总线以及外围组件互连(Peripheral Component Interconnect，PCI)总线。
电子设备12典型地包括多种计算机系统可读介质。这些介质可以是任何能够被电子设备12访问的可用介质，包括易失性和非易失性介质，可移动的和不可移动的介质。
系统存储器28可以包括易失性存储器形式的计算机系统可读介质，例如随机存取存储器(Random Access Memory，RAM)30和/或高速缓存存储器32。电子设备12可以进一步包括其它可移动/不可移动的、易失性/非易失性计算机 系统存储介质。仅作为举例，存储系统34可以用于读写不可移动的、非易失性磁介质(图4未显示，通常称为“硬盘驱动器”)。尽管图4中未示出，可以提供用于对可移动非易失性磁盘(例如“软盘”)读写的磁盘驱动器，以及对可移动非易失性光盘(例如只读光盘(Compact Disc-Read Only Memory，CD-ROM),数字视盘(Digital Video Disc-Read Only Memory，DVD-ROM)或者其它光介质)读写的光盘驱动器。在这些情况下，每个驱动器可以通过至少一个数据介质接口与总线18相连。系统存储器28可以包括至少一个程序产品，该程序产品具有一组(例如至少一个)程序模块，这些程序模块被配置以执行本申请各实施例的功能。
具有一组(至少一个)程序模块42的程序/实用工具40，可以存储在例如系统存储器28中，这样的程序模块42包括但不限于操作系统、至少一个应用程序、其它程序模块以及程序数据，这些示例中的每一个或某种组合中可能包括网络环境的实现。程序模块42通常执行本申请所描述的实施例中的功能和/或方法。
电子设备12也可以与至少一个外部设备14(例如键盘、指向设备、显示器24等)通信，还可与至少一个使得用户能与该电子设备12交互的设备通信，和/或与使得电子设备12能与至少一个其它计算设备进行通信的任何设备(例如网卡，调制解调器等等)通信。这种通信可以通过输入/输出(Input/Output，I/O)接口22进行。并且，电子设备12还可以通过网络适配器20与至少一个网络(例如局域网(Local Area Network，LAN)，广域网(Wide Area Network，WAN)和/或公共网络，例如因特网)通信。如图所示，网络适配器20通过总线18与电子设备12的其它模块通信。应当明白，尽管图中未示出，可以结合电子设备12使用其它硬件和/或软件模块，包括但不限于：微代码、设备驱动器、冗余处理单元、外部磁盘驱动阵列、磁盘阵列(Redundant Arrays of Independent Disks，RAID)系统、磁带驱动器以及数据备份存储系统等。
处理单元16通过运行存储在系统存储器28中的程序，从而执行各种功能 应用以及数据处理，例如实现本申请实施例所提供的标签识别方法，该方法包括：
获取物品的贴标图像，并对所述贴标图像进行预处理，得到待识别图像；
利用标签预测网络预测所述待识别图像中的标签，得到所述待识别图像中的多个预测标签及所述多个预测标签中每个预测标签的置信度；
利用标签确定模型基于所述多个预测标签及所述每个预测标签的置信度确定所述物品的目标标签。
本申请实施例还提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如所述的标签识别方法，该方法包括：
获取物品的贴标图像，并对所述贴标图像进行预处理，得到待识别图像；
利用标签预测网络预测所述待识别图像中的标签，得到所述待识别图像中的多个预测标签及所述多个预测标签中每个预测标签的置信度；
利用标签确定模型基于所述多个预测标签及所述每个预测标签的置信度确定所述物品的目标标签。
本申请实施例的计算机存储介质，可以采用一个或多个计算机可读的介质的任意组合。计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质。计算机可读存储介质例如可以是但不限于电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器((Erasable Programmable Read-Only Memory，EPROM)或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本文件中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。
计算机可读的信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。
计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于无线、电线、光缆、射频(Radio Frequency，RF)等等，或者上述的任意合适的组合。
可以以一种或多种程序设计语言或其组合来编写用于执行本申请操作的计算机程序代码，所述程序设计语言包括面向对象的程序设计语言诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络包括局域网(LAN)或广域网(WAN)连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。

Claims (10)

  1. 一种标签识别方法，包括：
    获取物品的贴标图像，并对所述贴标图像进行预处理，得到待识别图像；
    利用标签预测网络预测所述待识别图像中的标签，得到所述待识别图像中的多个预测标签及所述多个预测标签中每个预测标签的置信度；
    利用标签确定模型基于所述多个预测标签及所述每个预测标签的置信度确定所述物品的目标标签。
  2. 根据权利要求1所述的方法，其中，所述标签预测网络包括卷积网络和长短记忆网络，所述利用标签预测网络预测所述待识别图像中的标签，得到所述待识别图像中的多个预测标签及所述多个预测标签中每个预测标签的置信度，包括：
    利用所述卷积网络提取所述待识别图像中检测区域的特征向量；
    利用所述长短记忆网络根据所述特征向量的语义关系提取所述特征向量的文字特征，并根据所述文字特征预测所述待识别图像中的标签，得到所述待识别图像中的多个预测标签及所述多个预测标签中每个预测标签的置信度。
  3. 根据权利要求2所述的方法，其中，所述卷积网络包括卷积层和反卷积层，所述利用所述卷积网络提取所述待识别图像中检测区域的特征向量，包括：
    将所述待识别图像输入卷积层提取所述待识别图像中图像特征，并使用所述反卷积层对所述卷积层进行上采样，得到卷积图像，所述卷积图像与所述待识别图像的像素间存在关联；
    对所述卷积图像进行像素分类，得到所述待识别图像的二维热力图，并根据所述二维热力图对所述待识别图像进行图像分割，确定出所述待识别图像中的检测区域，对所述检测区域内的图像特征进行文本检测，得到所述检测区域的特征向量。
  4. 根据权利要求2所述的方法，其中，所述利用所述长短记忆网络根据所述特征向量的语义关系提取所述特征向量的文字特征，并根据所述文字特征预测所述待识别图像中的标签，得到所述待识别图像中的多个预测标签及所述多 个预测标签中每个预测标签的置信度，包括：
    将所述检测区域的特征向量输入所述长短记忆网络，得到所述特征向量的语义关系，并根据所述特征向量的语义关系提取所述特征向量的文字特征；
    根据所述文字特征预测所述待识别图像中的标签，得到所述待识别图像中的多个预测标签及所述多个预测标签中每个预测标签的置信度。
  5. 根据权利要求1所述的方法，利用标签确定模型基于所述多个预测标签及所述每个预测标签的置信度确定所述物品的目标标签之后，还包括：
    从所述物品的目标标签中提取属性信息，并将所述属性信息和所述贴标图像关联存储至信息数据库中，以使得根据所述贴标图像查询所述物品的属性信息。
  6. 根据权利要求1所述的方法，其中，所述标签确定模型按照如下方式获取：
    获取训练样本，其中，所述训练样本包含多张识别图像，每张识别图像中包含多个预测标签及所述多个预测标签中每个预测标签的置信度；
    利用时序类分类模型对所述训练样本中的每张识别图像中多个预测标签及所述多个预测标签中每个预测标签的置信度进行筛选，得到所述每张识别图像对应的目标标签；
    根据所述每张识别图像对应的目标标签和所述每张识别图像对应的实际标签计算损失函数；
    根据所述损失函数进行反向传播以优化所述时序类分类模型，得到所述标签确定模型。
  7. 根据权利要求1所述的方法，其中，利用标签预测网络预测所述待识别图像中的标签，还包括：
    将所述待识别图像输入至所述标签预测网络，得到所述待识别图像的识别错误信息，并根据所述识别错误信息进行故障预警。
  8. 一种标签识别装置，包括：
    图像获取模块，设置为获取物品的贴标图像，并对所述贴标图像进行预处理，得到待识别图像；
    网络预测模块，设置为利用标签预测网络预测所述待识别图像中的标签，得到所述待识别图像中的多个预测标签及所述多个预测标签中每个预测标签的置信度；
    标签确定模块，设置为利用标签确定模型基于所述多个预测标签及所述每个预测标签的置信度确定所述物品的目标标签。
  9. 一种电子设备，包括：
    至少一个处理器；
    存储装置，设置为存储至少一个程序，
    当所述至少一个程序被所述至少一个处理器执行，使得所述至少一个处理器实现如权利要求1-7中任一所述的标签识别方法。
  10. 一种计算机可读存储介质，所述计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时实现如权利要求1-7中任一所述的标签识别方法。
PCT/CN2023/079023 2022-06-15 2023-03-01 一种标签识别方法、装置、电子设备及存储介质 WO2023241102A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210676549.7A CN114972880A (zh) 2022-06-15 2022-06-15 一种标签识别方法、装置、电子设备及存储介质
CN202210676549.7 2022-06-15

Publications (1)

Publication Number Publication Date
WO2023241102A1 true WO2023241102A1 (zh) 2023-12-21

Family

ID=82963761

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/079023 WO2023241102A1 (zh) 2022-06-15 2023-03-01 一种标签识别方法、装置、电子设备及存储介质

Country Status (2)

Country Link
CN (1) CN114972880A (zh)
WO (1) WO2023241102A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114972880A (zh) * 2022-06-15 2022-08-30 卡奥斯工业智能研究院(青岛)有限公司 一种标签识别方法、装置、电子设备及存储介质
CN116542250B (zh) * 2023-06-29 2024-04-19 杭州同花顺数据开发有限公司 一种信息抽取模型获取方法和系统

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120269436A1 (en) * 2011-04-20 2012-10-25 Xerox Corporation Learning structured prediction models for interactive image labeling
CN107886028A (zh) * 2016-09-29 2018-04-06 九阳股份有限公司 一种冰箱的食材录入方法及食材录入装置
CN110827247A (zh) * 2019-10-28 2020-02-21 上海悦易网络信息技术有限公司 一种识别标签的方法及设备
CN112036509A (zh) * 2020-09-30 2020-12-04 北京百度网讯科技有限公司 用于训练图像识别模型的方法和装置
CN113420757A (zh) * 2021-08-23 2021-09-21 北京每日优鲜电子商务有限公司 文本审核方法、装置、电子设备和计算机可读介质
CN114972880A (zh) * 2022-06-15 2022-08-30 卡奥斯工业智能研究院(青岛)有限公司 一种标签识别方法、装置、电子设备及存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120269436A1 (en) * 2011-04-20 2012-10-25 Xerox Corporation Learning structured prediction models for interactive image labeling
CN107886028A (zh) * 2016-09-29 2018-04-06 九阳股份有限公司 一种冰箱的食材录入方法及食材录入装置
CN110827247A (zh) * 2019-10-28 2020-02-21 上海悦易网络信息技术有限公司 一种识别标签的方法及设备
CN112036509A (zh) * 2020-09-30 2020-12-04 北京百度网讯科技有限公司 用于训练图像识别模型的方法和装置
CN113420757A (zh) * 2021-08-23 2021-09-21 北京每日优鲜电子商务有限公司 文本审核方法、装置、电子设备和计算机可读介质
CN114972880A (zh) * 2022-06-15 2022-08-30 卡奥斯工业智能研究院(青岛)有限公司 一种标签识别方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN114972880A (zh) 2022-08-30

Similar Documents

Publication Publication Date Title
WO2023241102A1 (zh) 一种标签识别方法、装置、电子设备及存储介质
CN109902622B (zh) 一种用于登机牌信息验证的文字检测识别方法
CN111860506B (zh) 识别文字的方法和装置
CN112560999B (zh) 一种目标检测模型训练方法、装置、电子设备及存储介质
WO2021103868A1 (zh) 一种行人结构化方法、装置、设备和存储介质
CN111652087B (zh) 验车方法、装置、电子设备和存储介质
CN112613569B (zh) 图像识别方法、图像分类模型的训练方法及装置
CN112580657B (zh) 一种自学习字符识别方法
CN110188766B (zh) 基于卷积神经网络的图像主目标检测方法及装置
CN114898466A (zh) 一种面向智慧工厂的视频动作识别方法及系统
CN110851605A (zh) Ota酒店的图文信息匹配的检测方法、系统及电子设备
CN114694130A (zh) 基于深度学习的铁路沿线电线杆及杆号检测方法和装置
CN110728287A (zh) 图像识别方法及装置、电子设备、存储介质
CN113239883A (zh) 分类模型的训练方法、装置、电子设备以及存储介质
CN115546824B (zh) 禁忌图片识别方法、设备及存储介质
CN116030050A (zh) 基于无人机与深度学习的风机表面缺陷在线检测分割方法
US20230267779A1 (en) Method and system for collecting and monitoring vehicle status information
CN114612971A (zh) 人脸检测方法、模型训练方法、电子设备及程序产品
CN114245232A (zh) 一种视频摘要生成方法、装置、存储介质及电子设备
CN115565201B (zh) 禁忌图片识别方法、设备及存储介质
CN111988506B (zh) 补光方法及装置、电子设备及存储介质
CN116844003A (zh) 一种基于深度学习的水涝识别方法及其装置与储存介质
CN112883876A (zh) 室内行人检测的方法、装置、设备及计算机存储介质
CN113762248A (zh) 一种目标落地检测方法、装置、电子设备和存储介质
CN117934873A (zh) 图像筛选方法、装置、计算机设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23822674

Country of ref document: EP

Kind code of ref document: A1