WO2020134099A1

WO2020134099A1 - Article identification method, device and system

Info

Publication number: WO2020134099A1
Application number: PCT/CN2019/099757
Authority: WO
Inventors: 刘朋樟; 张屹峰; 刘巍; 陈宇
Original assignee: 北京沃东天骏信息技术有限公司
Priority date: 2018-12-29
Filing date: 2019-08-08
Publication date: 2020-07-02
Also published as: CN109711473A

Abstract

The present invention relates to the field of computers, and provides an article recognition method, device and system. According to the present invention, an article is identified on the basis of an image feature comparison mode, and a novel article identification solution is implemented; on the other hand, a model for constructing a feature database can generate features of untrained articles, so that the untrained articles can also be identified; on the other hand, the model for constructing the feature database does not need to be retrained for each new article; and on the other hand, article identification can be performed in a smaller range, so that the probability of article error identification is reduced.

Description

Article identification method, equipment and system

cross reference

This application is based on the application with the CN application number 201811628625.7 and the application date is December 29, 2018, and claims its priority. The disclosure content of the CN application is hereby incorporated into this application as a whole.

Technical field

The present disclosure relates to the field of computers, and in particular, to an item identification method, device, and system.

Background technique

A related-art vending machine collects images of items taken away, processes the collected images using a neural network model, and outputs the probability that the items taken away belong to each saleable item. If there are 1,000 types of items for sale, each time the neural network model is recognized, a 1000-dimensional vector will be output. Each dimension of the vector corresponds to the probability of a type of item for sale. Items taken away are usually identified as the most probable items for sale. article.

Summary of the invention

Some embodiments of the present disclosure provide an item identification method, including:

Generating feature information of the item to be identified based on the collected image information of the item to be identified;

Comparing the generated feature information of the item to be identified with the feature information of the image of each item in the first feature database;

The type of the item to be identified is determined according to the similarity comparison result.

In some embodiments, the first feature database is received data sent by the cloud device and stored locally; the data of the first feature database corresponds to a local set of items to be identified.

In some embodiments, when the local set of items to be identified changes, the first feature database is updated accordingly.

In some embodiments, the first feature database is a subset of the second feature database located in the cloud device,

The method for establishing the second feature database includes:

The feature information of the image of each item is sent to the cloud device and added to the second feature database; or,

The image of each item is sent to the cloud device, and the cloud device calculates the feature information of the image of each item and adds it to the second feature database.

In some embodiments, generating feature information of the item to be identified includes:

Use the feature extraction model to process the input image information of the item to be identified, and output the characteristic information of the item to be identified,

The feature extraction model is obtained by training a feature-based classification model and deleting the classification layer of the classification model after training. The classification model includes a classification layer, and the classification layer is used to perform image processing based on image feature information. classification.

The method for establishing the second feature database includes:

Use the feature extraction model to process the input image information of each item, output the feature information of the corresponding item, and add it to the second feature database,

In some embodiments, the classification model is a neural network model, and the feature extraction model includes a convolutional layer, a pooling layer, and a fully connected layer.

In some embodiments, the feature extraction model is used to generate feature information of the image of the newly added item without retraining, and is added to the second feature database.

In some embodiments, the feature extraction model is retrained if the preset conditions are met,

The preset conditions include: the newly added item reaches a preset ratio or quantity, meets a preset time interval, or the accuracy of item identification is lower than a preset value.

In some embodiments, the method is performed by a vending machine or cloud device.

Training a feature-based classification model, the classification model includes a classification layer, and the classification layer is used to classify images according to image feature information;

Delete the classification layer of the classification model after training to obtain a feature extraction model;

The feature extraction model processes the input image of each item and outputs feature information of the image of the corresponding item;

The feature information of the image of each item is added to the second feature database, and the feature extraction model and the second feature database are used for item identification.

In some embodiments, the feature extraction model is used to generate feature information of images of newly added items without retraining.

In some embodiments, the image of the newly added item comes from a vending machine.

In some embodiments, the method further includes: the feature extraction model and the subset of the second feature database are delivered to each vending machine for item identification by the vending machine, and the data of the subset of the second feature database corresponds to the sales The set of items to be identified on the freighter.

In some embodiments, it also includes:

Receive the image information of the items to be identified uploaded by the vending machine;

Extract feature information of the item to be identified based on the feature extraction model;

Determining a subset of the second feature database according to the set of items to be identified of the vending machine;

Comparing the feature information of the item to be identified with the feature information of the image of each item in the subset of the second feature database;

Some embodiments of the present disclosure provide a cloud device, including:

Storage; and

A processor coupled to the memory, the processor configured to perform the method performed by the cloud device in any of the foregoing embodiments based on instructions stored in the memory.

Some embodiments of the present disclosure propose a vending machine, including:

Storage; and

A processor coupled to the memory, the processor configured to perform the method performed by the vending machine in any of the foregoing embodiments based on instructions stored in the memory.

Some embodiments of the present disclosure propose an item identification system, including:

The aforementioned cloud device; and

Several of the aforementioned vending machines.

Some embodiments of the present disclosure propose a non-transitory computer-readable storage medium on which a computer program is stored, which when executed by a processor implements the method in any of the foregoing embodiments.

BRIEF DESCRIPTION

The drawings needed to be used in the embodiments or related technical descriptions will be briefly introduced below. The present disclosure can be more clearly understood from the following detailed description with reference to the drawings,

Obviously, the drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, without paying any creative labor, other drawings can also be obtained based on these drawings.

FIG. 1 is a schematic diagram of an item identification system according to some embodiments of the present disclosure.

FIG. 2a is a schematic flowchart of item identification by a vending machine 12 according to some embodiments of the present disclosure.

FIG. 2b is a schematic flowchart of item identification performed by the cloud device 11 according to some embodiments of the present disclosure.

FIG. 3 shows a schematic diagram of establishing a second feature database according to some embodiments of the present disclosure.

FIG. 4 shows a schematic diagram of the cloud device of some embodiments of the present disclosure delivering feature information of images of some items to each vending machine.

FIG. 5 shows a schematic diagram of updating the second feature database for newly added items in some embodiments of the present disclosure.

6 is a schematic diagram of a cloud device according to some embodiments of the present disclosure.

7 is a schematic diagram of a vending machine according to some embodiments of the present disclosure.

detailed description

The technical solutions in the embodiments of the present disclosure will be described clearly and completely in conjunction with the drawings in the embodiments of the present disclosure.

The inventor found that the neural network model of the related art cannot identify items that have not been trained. If there are new items, the neural network model needs to be retrained. In addition, item recognition needs to be performed on the range of all available items. These problems cause vending machines. The operability is relatively poor.

In view of this, the present disclosure proposes an item identification solution to solve at least one of the foregoing problems and improve the operability of the vending machine.

On the one hand, the present disclosure recognizes items based on image feature comparison and implements a new type of item recognition scheme; on the other hand, the model of building a feature database can generate the characteristics of untrained items, making the untrained Items can also be identified; on the one hand, the model for building a feature database does not need to be retrained for each new item; on the other hand, item identification can be performed in a smaller range, which is helpful to reduce the probability of item misidentification.

As shown in FIG. 1, the article identification system 10 of this embodiment includes a cloud device 11 and a number of vending machines 12. These vending machines 12 can be located in different locations, for example, and can realize automatic vending. Information exchange between the cloud device 11 and each vending machine 12 is possible.

For example, the cloud device 11 may establish a feature database based on the feature extraction model. The vending machine 12 therein can collect images of items to be identified, for example. The cloud device 11 or the vending machine 12 can perform item identification based on the feature database and the image of the item to be identified. The two article identification methods are described below in conjunction with FIGS. 2a and 2b.

For convenience of description, the feature database located in the vending machine 12 is set as the first feature database, and the feature database located in the cloud device 11 is set as the second feature database. The second feature database usually includes feature information of images of all items. In some embodiments, the first feature database is a subset of the second feature database, and the data of the first feature database corresponds to the set of items to be identified locally by the vending machine (ie, items sold by the vending machine). When the local to-be-recognized item set of the vending machine changes, the first feature database is updated accordingly. Of course, if the computing resources of the vending machine are sufficient, or, regardless of the recognition efficiency, the first feature database may also be the complete set of the second feature database.

In some embodiments, if the item identification is performed by the vending machine 12, the cloud device 11 may send the feature extraction model and each first feature database (that is, feature information of an image of some items in the second feature database) to Each vending machine 12, and then, the vending machine 12 uses the feature extraction model issued by the cloud device 11 to obtain the feature information of the image of the item to be identified, and through the feature information of the image of the item to be identified and the first feature database issued by the cloud device 11 Article identification is carried out in a manner of comparing feature information of images of each article in. This embodiment is described below with reference to FIG. 2a.

As shown in FIG. 2a, this embodiment includes steps S21-S26.

In step S21, the cloud device 11 builds a second feature database based on the feature extraction model (refer to the schematic diagram of establishing a second feature database shown in FIG. 3, and id represents an item identifier).

The feature extraction model can be obtained through steps S211 to 212, for example:

In step S211, the training data is used to train a feature-based classification model. The classification model includes a classification layer, which is used to classify images based on image feature information.

The training data includes, for example, the image of the item and the type of the item annotating the image, that is, which item is annotated in the image.

In step S212, the classification layer of the feature-based classification model after training is deleted to obtain a feature extraction model.

The feature-based classification model is, for example, a neural network model. The neural network model includes, for example, a convolutional layer, a pooling layer, a fully connected layer, and a classification layer, and the corresponding feature extraction model includes a convolutional layer, a pooling layer, and a fully connected layer. .

The classification layer is, for example, a softmax layer. If there are 1000 saleable items, define a softmax layer with a dimension of 1000. The softmax layer will output a 1000-dimensional vector, and each dimension of the vector corresponds to the probability of a saleable item.

It is assumed that each image can be characterized as a D-dimensional feature vector, such as 256 dimensions, through the feature extraction model. The number of types of items is N. Each item uses K images to extract features, and N×K D-dimensional feature vectors can be obtained through the feature extraction model. Optionally, K feature vectors of each item are clustered, and m feature vectors are selected according to the clustering result to characterize the item. For example, the clustering algorithm may use the K-means algorithm. The final feature database includes feature information of N items, and the feature information of each item is m D-dimensional feature vectors.

The establishment of the second feature database may be implemented through steps S213 to 214, for example:

In step S213, the feature extraction model processes the input image of each item and outputs feature information of the image of the corresponding item.

In step S214, the feature information of the image of each item is added to the second feature database.

In step S22, the cloud device 11 extracts the corresponding subset of the second feature database as the first feature database of the vending machine 12 according to the set of items to be identified of the vending machine 12, and delivers the feature extraction model and the first feature database to the corresponding Each of the vending machines 12 is used to identify items by the vending machine.

For example, if the second feature database has feature information of 1,000 items, and a certain vending machine 1 sells 20 items, the feature information of these 20 items is sent to the vending machine as the first feature database of the vending machine 1 1. The vending machine 1 can perform item identification within the range of 20 items, without the need to perform item identification within the range of 1000 items, and the item identification within a smaller range is conducive to reducing the probability of incorrect identification of items and improving identification effectiveness. However, each vending machine has the ability to recognize all items, so each vending machine can pull feature information of any item from the second feature database as needed.

FIG. 4 shows a schematic diagram of the cloud device delivering the feature information of the image of some items to each vending machine. For example, according to the situation of the items sold by each vending machine, the characteristic information of the item with the item ID of 1/3/4 sold by the vending machine 1 is sent to the vending machine 1, and the item ID of the item sold by the vending machine 2 is 2/4/7 The characteristic information of the item of the item is sent to the vending machine 2, and the characteristic information of the item with the item id of 9/11/12 sold by the vending machine n is sent to the vending machine n. In addition, if the feature information of different items is relatively similar, the feature information of similar items can be separately delivered to different vending machines, so that the difference in feature information of different items in the same vending machine is relatively large, which is conducive to improving item recognition Accuracy.

Those skilled in the art can understand that if the problems of reducing the probability of incorrect identification of items and improving the efficiency of identification are not considered, the cloud device 11 can also send the feature information of the images of all the items in the feature database to each vending machine, and the items can also be realized Identify.

In step S23, a certain vending machine 12 collects an image of the item to be identified.

Among them, the vending machine 12 may use a camera to collect an image of the item to be identified. For example, the camera may be disposed in the vending machine 12 adjacent to the cabinet door, so as to collect images of the items.

In step S24, the vending machine 12 uses the feature extraction model to generate feature information of the image of the item to be recognized.

In step S25, the vending machine 12 compares the generated feature information of the image of the item to be recognized with the feature information of the image of each item in the first feature database stored in the machine.

For example, the Euclidean distance between the feature information of the image of the item to be recognized and the feature information of the image of each item in the feature database is calculated. The smaller the Euclidean distance, the more similar the description.

In step S26, the vending machine 12 determines the type of the item to be identified based on the similarity comparison result.

Generally, the item to be identified can be determined as the item in the first feature database that is most similar to the item to be identified.

The above-mentioned embodiment recognizes items based on the image feature comparison, and implements a new item identification scheme. In addition, each vending machine has the ability to identify all items, but it can perform item identification in the limited item types it sells. It is not necessary to perform item identification in all item types in the feature database, which is helpful to reduce the probability of item misidentification.

In other embodiments, if the cloud device 11 performs item recognition, the vending machine 12 collects the image of the item to be identified and uploads it to the cloud device 11, and then the cloud device 11 uses the feature extraction model to obtain the image of the item to be identified Feature information, determine the subset of the second feature database (that is, the first feature database) according to the set of items to be recognized of the vending machine 12 through the feature information of the image of the item to be recognized and the feature information of the image of each item in the first feature database The article identification is performed in a comparison manner, and the article identification result is returned to the vending machine 12. The embodiment will be described below with reference to FIG. 2b.

As shown in FIG. 2b, this embodiment includes the following steps: steps S31 to S36.

In step S31, as in step S21, the cloud device 11 builds a second feature database based on the feature extraction model. The cloud device 11 may save the feature extraction model and the second feature database.

In step S32, a certain vending machine 12 collects the image of the item to be identified and uploads it to the cloud device 11.

In step S33, the cloud device 11 uses the feature extraction model to acquire feature information of the image of the item to be identified.

In step S34, the cloud device 11 determines a subset of the second feature database (ie, the first feature database) according to the set of items to be identified of the vending machine 12, and compares the generated feature information of the image of the item to be identified with the first feature database The feature information of the images of each item is compared for similarity.

Those skilled in the art can understand that if the problems of reducing the probability of incorrect recognition of items and improving the efficiency of recognition are not considered, the cloud device 11 may also compare the feature information of the image of the item to be recognized with the feature information of the images of all the items in the second feature database Similarity comparison can also realize item identification.

For the method of similarity comparison, please refer to the foregoing, which will not be repeated here.

In step S35, the cloud device 11 determines the type of the item to be identified according to the similarity comparison result.

In step S36, the cloud device 11 returns the item identification result to the vending machine 12, and the vending machine 12 obtains the item identification result.

The above-mentioned embodiment recognizes items based on the image feature comparison, and implements a new item identification scheme. In addition, complex operations such as feature extraction and similarity comparison are implemented by cloud devices with powerful computing capabilities, which can improve recognition efficiency and greatly reduce the performance requirements of vending machines.

The above item recognition scheme has many advantages for the application of the feature extraction model. The following is a detailed description.

Refer to FIG. 5 for a schematic diagram of updating the second feature database for newly added items. Assuming that the feature extraction model has completed training and feature extraction for existing items, if there are newly added items, the feature extraction model may not be retrained, and the current feature extraction model is used to directly generate feature information of the images of several newly added items. The feature information of the newly added item can be directly added to the second feature database as incremental information and delivered to the vending machine that sells the newly added item.

It can be seen that in this disclosure, the model for constructing the feature database does not need to be retrained for each new item, the number of model trainings is reduced, and the model for constructing the feature database can generate features of items that have not been trained, so that no training has been done Items can also be identified. These characteristics make the solution of the present disclosure have good operability, which greatly improves the online speed of new items.

In some embodiments, the feature extraction model is retrained if it meets the preset conditions, where the preset conditions include: the newly added items reach a preset ratio or number, satisfying the preset time interval (such as the number Month), or, the accuracy of item identification is lower than the preset value.

The feature extraction model can be retrained based on the image of the original item and the image of the newly added item. Then, a new feature extraction model is used to extract the feature information of the images of all items, thereby generating a new feature database.

In some embodiments, the image of the newly added item may come from a vending machine that actually sells the item, and is not limited to having to come from a test vending machine. Vending machines that actually sell items may belong to different operators.

In some embodiments, the image of the newly added item may come from a certain operator.

The items sold by various operators may be different, but the images of these items can eventually participate in the training of the feature extraction model. Even if there are few item images uploaded by a certain operator, using all the item images uploaded by all operators can still train a feature extraction model with high accuracy. Relative to each operator independently maintaining a feature database, it is conducive to reducing costs. The capability of this feature extraction model will continue to be optimized as the number of uploaded item images increases.

As shown in FIG. 6, the cloud device 11 of this embodiment includes:

Memory 111; and

A processor 112 coupled to the memory is configured to execute the method performed by the cloud device 11 in any of the foregoing embodiments based on instructions stored in the memory.

As shown in FIG. 7, the vending machine 12 of this embodiment includes:

Memory 121; and

A processor 122 coupled to the memory, the processor configured to perform the method performed by the vending machine 12 in any of the foregoing embodiments based on instructions stored in the memory.

Among them, the memories 111, 121 may include, for example, a system memory, a fixed non-volatile storage medium, and the like. The system memory stores, for example, an operating system, application programs, a boot loader (Boot Loader), and other programs.

Some embodiments of the present disclosure propose a computer-readable storage medium on which a computer program is stored, which when executed by a processor implements the item identification method in any of the foregoing embodiments.

Those skilled in the art should understand that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer usable non-transitory storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code .

The above are only preferred embodiments of the present disclosure and are not intended to limit the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present disclosure shall be included in the protection of the present disclosure Within range.

Claims

An item identification method, including:

Generating feature information of the item to be identified based on the collected image information of the item to be identified;

Comparing the generated feature information of the item to be identified with the feature information of the image of each item in the first feature database;

The type of the item to be identified is determined according to the similarity comparison result.
The method of claim 1, wherein

The first feature database is received data sent by the cloud device and is stored locally;

The data of the first feature database corresponds to the local set of items to be identified.
The method of claim 2, wherein when the local set of items to be identified changes, the first feature database is updated accordingly.
The method of claim 1, wherein

The first feature database is a subset of the second feature database located in the cloud device,

The method for establishing the second feature database includes:

The feature information of the image of each item is sent to the cloud device and added to the second feature database;

or,

The image of each item is sent to the cloud device, and the cloud device calculates the feature information of the image of each item and adds it to the second feature database.
The method of claim 1, wherein generating feature information of the item to be identified includes:

Use the feature extraction model to process the input image information of the item to be identified, and output the characteristic information of the item to be identified,

The feature extraction model is obtained by training a feature-based classification model and deleting the classification layer of the classification model after training. The classification model includes a classification layer, and the classification layer is used to perform image processing based on image feature information. classification.
The method of claim 1, wherein

The first feature database is a subset of the second feature database located in the cloud device,

The method for establishing the second feature database includes:

Use the feature extraction model to process the input image information of each item, output the feature information of the corresponding item, and add it to the second feature database,

The feature extraction model is obtained by training a feature-based classification model and deleting the classification layer of the classification model after training. The classification model includes a classification layer, and the classification layer is used to perform image processing based on image feature information. classification.
The method according to claim 5 or 6, wherein the classification model is a neural network model, and the feature extraction model includes a convolutional layer, a pooling layer, and a fully connected layer.
The method of claim 6, wherein:

Without retraining, the feature extraction model is used to generate feature information of the image of newly added items and add it to the second feature database.
The method according to claim 5, 6 or 8, wherein the feature extraction model is retrained if the preset conditions are met,

The preset conditions include: the newly added item reaches a preset ratio or quantity, meets a preset time interval, or the accuracy of item identification is lower than a preset value.
The method of claim 1, which is performed by a vending machine or a cloud device.
An item identification method, including:

Training a feature-based classification model, the classification model includes a classification layer, and the classification layer is used to classify images according to image feature information;

Delete the classification layer of the classification model after training to obtain a feature extraction model;

The feature extraction model processes the input image of each item and outputs feature information of the image of the corresponding item;

The feature information of the image of each item is added to the second feature database, and the feature extraction model and the second feature database are used for item identification.
The method of claim 11, wherein

The feature extraction model is used to generate feature information of images of newly added items without retraining.
The method of claim 12, wherein the newly added image of the item is from a vending machine.
The method of claim 11, wherein the feature extraction model is retrained if the preset conditions are met,

The preset conditions include: the newly added item reaches a preset ratio or quantity, meets a preset time interval, or the accuracy of item identification is lower than a preset value.
The method of claim 11, further comprising:

The feature extraction model and the subset of the second feature database are distributed to each vending machine for item identification by the vending machine, and the data of the subset of the second feature database corresponds to the item set to be identified of the vending machine.
The method of claim 11, further comprising:

Receive the image information of the items to be identified uploaded by the vending machine;

Extract feature information of the item to be identified based on the feature extraction model;

Determining a subset of the second feature database according to the set of items to be identified of the vending machine;

Comparing the feature information of the item to be identified with the feature information of the image of each item in the subset of the second feature database;

The type of the item to be identified is determined according to the similarity comparison result.
A cloud device, including:

Storage; and

A processor coupled to the memory, the processor configured to perform the method of any one of claims 1, 5-9, 11-16 based on instructions stored in the memory.
A vending machine, including:

Storage; and

A processor coupled to the memory, the processor configured to perform the method of any one of claims 1-5 based on instructions stored in the memory.
An item identification system, including:

The cloud device of claim 17; and

A number of vending machines as claimed in claim 18.
A non-transitory computer-readable storage medium on which a computer program is stored, which when executed by a processor implements the method of any one of claims 1-16.