WO2020215952A1 - Procédé et système de reconnaissance d'objets - Google Patents

Procédé et système de reconnaissance d'objets Download PDF

Info

Publication number
WO2020215952A1
WO2020215952A1 PCT/CN2020/080767 CN2020080767W WO2020215952A1 WO 2020215952 A1 WO2020215952 A1 WO 2020215952A1 CN 2020080767 W CN2020080767 W CN 2020080767W WO 2020215952 A1 WO2020215952 A1 WO 2020215952A1
Authority
WO
WIPO (PCT)
Prior art keywords
item
image
category
recognized
information
Prior art date
Application number
PCT/CN2020/080767
Other languages
English (en)
Chinese (zh)
Inventor
马事伟
吴江旭
张伟华
石海龙
张洪光
徐荣图
胡淼枫
王璟璟
Original Assignee
北京京东尚科信息技术有限公司
北京京东世纪贸易有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京京东尚科信息技术有限公司, 北京京东世纪贸易有限公司 filed Critical 北京京东尚科信息技术有限公司
Publication of WO2020215952A1 publication Critical patent/WO2020215952A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting

Definitions

  • the present disclosure relates to the field of image recognition, and in particular to an item recognition method and system.
  • the dishes need to be identified first, and then settled according to the prices corresponding to the dishes.
  • dishes can be identified based on computer vision technology.
  • the sensor triggers the image acquisition device to take a picture of the dish, and then recognizes the dish; or, processes each image collected by the image acquisition device to identify the dish information in the image.
  • an item identification method which includes: acquiring one or more images to be identified, wherein the images to be identified include one or more items to be identified; using a trained pre-recognition model to determine Identify whether the probability of the image being clear and containing the complete item is greater than a threshold; and if the probability is greater than the threshold, identify the category of each item.
  • the probability of the first image being clear and containing the complete item is greater than the first threshold, and the probability of other images being clear and containing the complete item is greater than the second threshold, the The items contained in the first image among the images are classified.
  • training the pre-recognition model includes: labeling images in the sample images that are clear and containing complete items as positive sample images, and labeling images that are not positive sample images in the sample images as negative sample images; and The sample image and the negative sample image train the pre-recognition model to determine whether the probability that the image to be recognized is clear and contains the complete item is greater than the threshold according to the trained pre-recognition model.
  • identifying the category of each item includes: inputting the image to be recognized into the item detection model, extracting the area information and the first level category corresponding to each item in the image to be recognized; determining the effective category in the first level category; And input the area information of the items belonging to the valid category in the image to be recognized into the item recognition model, extract the item features corresponding to each area information, and compare the item features corresponding to each area information with the item features in the item feature library to determine The second-level category of each item in the image to be recognized.
  • training the item detection model and the item recognition model includes: labeling the area information and the first-level category corresponding to the item in the sample image, generating first label information, and labeling the item based on the sample image and the first label information.
  • the detection model is trained to determine the area information and the first-level category corresponding to each item in the image to be recognized according to the trained item detection model; and to label the item features corresponding to the area information of the effective category items in the sample image
  • the second tagging information is to train the item recognition model based on the sample image and the second tagging information, so as to extract the item features corresponding to the area information of each item in the image to be recognized based on the trained item recognition model.
  • the minimum distance between the item feature of each item and the item feature in the item feature library is determined; if the minimum distance is less than or equal to the distance threshold, the item feature corresponding to each item in the item feature library is the closest to the item feature
  • the corresponding category is regarded as the second-level category of each item; if the minimum distance is greater than the distance threshold, the user is prompted whether the item category and attribute information needs to be input; and if the item category and attribute need to be input, the item category and attribute information is added, otherwise ,
  • the category corresponding to the closest item feature in the item feature library corresponding to each item is taken as the second level category of each item.
  • the corresponding attribute information is matched according to the category of each item.
  • the image to be recognized is marked as a training image or a test image, so that the item detection model and the item recognition model are performed based on the image to be recognized. Training or testing.
  • the size information of each item is determined based on the item detection model, and the corresponding attribute information is matched according to the category and size of each item; it is determined whether multiple items in the image to be identified meet the item combination, if multiple items meet the item combination , Match the attribute information corresponding to the item combination; determine whether the attribute sum corresponding to multiple items in the image to be identified meets the preset condition, if the attribute sum satisfies the preset condition, the attribute sum is processed according to the preset condition; and determine For the matching time of item matching attribute information, the attribute information corresponding to each item is determined according to the matching time.
  • an item identification system including: an image acquisition module configured to acquire one or more images to be identified, wherein the images to be identified include one or more items to be identified;
  • the pre-recognition module is configured to use the trained pre-recognition model to determine whether the probability that the image to be recognized is clear and contains the complete item is greater than a threshold; and the item determination module is configured to recognize the probability of each item when the probability is greater than the threshold category.
  • the pre-recognition module is further configured to: if the probability of the first image being clear and containing the complete item is greater than the first threshold, and the probability of other images being clear and containing the complete item is greater than the first If the threshold is two, the first image in the consecutive multiple images is sent to the item determination module; and the item determination module is configured to perform category recognition on the items contained in the first image in the consecutive multiple images.
  • the pre-recognition module is further configured to mark images in the sample image that are clear and contain complete items as positive sample images, and to mark images in the sample images that are not positive sample images as negative sample images; and
  • the positive sample image and the negative sample image train the pre-recognition model to determine whether the probability that the image to be recognized is clear and contains the complete item is greater than the threshold according to the trained pre-recognition model.
  • the item determination module includes: an item detection module configured to input the image to be recognized into the item detection model, and extract the area information and the first-level category corresponding to each item in the image to be recognized based on the item detection model;
  • the management module is configured to determine the effective category in the first-level category;
  • the item recognition module is configured to input the area information of items belonging to the effective category in the image to be recognized into the item recognition model, and extract each area based on the item recognition model The item features corresponding to the information are compared with the item features in the item feature database to determine the second-level category of each item in the image to be recognized.
  • the item management module is configured to determine the effective category in the first-level category; the item detection module is configured to input the image to be recognized into the item detection model, and extract the area corresponding to each item in the image to be recognized Information and the first-level category, call the item management module, and input the area information of items belonging to the effective category to the item recognition module; and the item recognition module is also configured to identify the item characteristics corresponding to the area information of the effective category items in the sample image Annotation is performed to generate second annotation information, and the item recognition model is trained based on the sample image and the second annotation information, so as to extract the item features corresponding to the area information of each item in the image to be recognized according to the trained item identification model.
  • the item management module is configured to determine valid item features in the item feature library within a predetermined time; and the item identification module is further configured to compare the item features corresponding to each area information with the valid item features in the item feature library. Perform comparison to determine the second-level category of each item in the image to be identified.
  • the item identification module is configured to determine the minimum distance between the item feature of each item and the item feature in the item feature library; if the minimum distance is less than or equal to the distance threshold, the item in the item feature library corresponding to each item The category corresponding to the item feature with the closest feature distance is regarded as the second-level category of each item; if the minimum distance is greater than the distance threshold, the user is prompted whether the item category and attribute information needs to be input; if the item category and attribute need to be input, the item category is added And attribute information, otherwise, the category corresponding to the item feature closest to the item feature corresponding to each item in the item feature library is taken as the second level category of each item.
  • the attribute matching unit is configured to match corresponding attribute information according to the category of each item.
  • the item management module is further configured to, after matching the attribute information, in response to the user modifying the attribute information corresponding to the category of the item, mark the image to be recognized as a training image or a test image, so as to compare the item based on the image to be recognized.
  • the detection model and the item recognition model are trained or tested.
  • the attribute matching unit is further configured to: match corresponding attribute information according to the category and size of each item, wherein the item detection module is further configured to determine the size information of each item based on the item detection model ; Determine whether multiple items in the image to be recognized meet the item combination, if multiple items meet the item combination, match the attribute information corresponding to the item combination; determine whether the attribute sum corresponding to multiple items in the image to be recognized meets the preset conditions, If the preset conditions are met, the attributes are processed according to the preset conditions; and the matching time of the item matching attribute information is determined, and the attribute information corresponding to each item is determined according to the matching time.
  • an article identification system including: a memory; and a processor coupled to the memory, the processor configured to execute the above-mentioned article identification method based on instructions stored in the memory.
  • a non-transitory computer-readable storage medium on which computer program instructions are stored, and when the instructions are executed by a processor, the above-mentioned item identification method is realized.
  • FIG. 1 is a schematic flowchart of some embodiments of the object identification method of the present disclosure.
  • FIG. 2 is a schematic flowchart of other embodiments of the object identification method of the present disclosure.
  • FIG. 3 is a schematic structural diagram of some embodiments of the object identification system of the present disclosure.
  • Fig. 4 is a schematic structural diagram of other embodiments of the object identification system of the present disclosure.
  • FIG. 5 is a schematic structural diagram of other embodiments of the object identification system of the present disclosure.
  • FIG. 6 is a schematic structural diagram of other embodiments of the object identification system of the present disclosure.
  • FIG. 7 is a schematic structural diagram of other embodiments of the object identification system of the present disclosure.
  • FIG. 8 is a schematic structural diagram of other embodiments of the object identification system of the present disclosure.
  • FIG. 1 is a schematic flowchart of some embodiments of the object identification method of the present disclosure.
  • one or more images to be identified are acquired, where the images to be identified include one or more objects to be identified.
  • the images to be identified include one or more objects to be identified. For example, in a restaurant, when a customer purchases a dish and a bowl of rice, the dish and rice can be placed in the recognition area, and the recognition area can be photographed by a camera to obtain an image containing the dish and rice.
  • step 120 the trained pre-recognition model is used to determine whether the probability that the image to be recognized is clear and contains a complete item is greater than a threshold.
  • the pre-recognition model may be trained in advance, a certain number of sample images are collected, and the sample images are classified. For example, the image in the sample image is clear and the image containing the complete item is annotated as a positive sample image, the image in the sample image that is not a positive sample image is annotated as a negative sample image, and the pre-recognition model is performed based on the positive sample image and the negative sample image. training.
  • the output result of the pre-recognition module is compared with the corresponding information of the sample to determine whether the comparison result meets the requirements of the loss function of the pre-recognition model. Iteratively, optimize and adjust the parameters of the pre-recognition module, so that the comparison result finally meets the requirements of constructing the pre-recognition model The requirement of the loss function, save the pre-recognition model.
  • a customer purchases a dish and a bowl of rice, first places the dish and rice on a tray, and then places the tray in the recognition area.
  • the tray is constantly moving. If the tray does not completely enter the recognition area when the image capture device captures an image, the tray in the image is incomplete. For example, some dishes are not collected, or some dishes are collected only a small part, which will affect the accuracy of subsequent identification. In addition, when the tray is moving, the collected images are blurred with motion, which will also affect the accuracy of subsequent recognition. Therefore, invalid images are excluded, and dishes are recognized only for images that contain a complete tray and have clear images.
  • a certain number of images of the recognition area are collected, including images of no tray in the area, images of the tray just entering the recognition area, images of half of the tray entering the recognition area, and images of the tray completely entering the recognition area. Then classify the images, label the images with clear images and the trays in the image completely entering the recognition area as positive sample images, label other images as negative sample images, and then train the pre-recognition model based on the positive and negative sample images. If the customer does not use the tray, the image is clear, and the images containing the complete dishes, beverages and other priced goods are marked as positive sample images, and the other images are marked as negative sample images, and then the pre-recognition model is trained.
  • step 130 when the probability is greater than the threshold, the category of each item is identified. That is, in this embodiment, instead of performing item category recognition on all images, it first determines whether the image meets the requirements, and performs item recognition on the images that meet the requirements.
  • each item in the image whose image is clear and the probability of containing a complete item is greater than the threshold is recognized, instead of performing item recognition on all images, the accuracy and recognition efficiency of the recognition system can be improved.
  • the probability of each image being clear and containing a complete item in multiple consecutive images to be recognized is greater than the threshold, then the first image contained in the consecutive multiple images Items are identified by category.
  • the probability that the first image is clear and contains a complete item is greater than the first threshold, and the other images are clear and contain complete If the probability of the item is greater than the second threshold, the item contained in the first image among the consecutive multiple images is classified.
  • the item contained in the first image is identified.
  • the second to Nth images are clear and the probability of containing a complete object is greater than 0.1, the second to Nth images are not processed.
  • FIG. 2 is a schematic flowchart of other embodiments of the object identification method of the present disclosure.
  • step 210 one or more images to be identified are acquired, where the images to be identified include one or more items to be identified.
  • step 220 the trained pre-recognition model is used to determine whether the probability that the image to be recognized is clear and contains a complete item is greater than a threshold.
  • step 230 when the probability is greater than the threshold, the image to be recognized is input to the item detection model, and the region information and the first level category corresponding to each item in the image to be recognized are extracted based on the item detection model.
  • the first-level category refers to the category to which the item belongs, such as dishes, fruits, and beverages.
  • the item detection model can be pre-trained, the area information and the first-level category corresponding to the item in the sample image are annotated, the first annotation information is generated, and the item detection model is performed based on the sample image and the first annotation information. training. Compare the output result of the item detection model with the first label information to determine whether the comparison result meets the requirements of the loss function for building the item recognition model, and iterate repeatedly, optimize and adjust the parameters of the item detection model, so that the comparison result finally meets the construction of the item detection model The requirements of the loss function, save the item detection model.
  • the item detection model When training the item detection model, mark the items in the collected images as categories such as dishes, yogurt, fruits, drinks, keys, badges, wallets, mobile phones, chopsticks, spoons, and hands. Then input the image to the item detection model for training. After the item detection model is trained, when an image is input, the item detection model can output the area information and category information of each item in the image.
  • a valid category in the first-level category is determined according to the configuration information. For example, for non-valuable items, there is a probability that they will be mistaken as valued items. Therefore, it is necessary to remove the category of non-valued items and only keep the category of valued items to avoid misidentification.
  • step 250 the area information of the items belonging to the effective category in the image to be recognized is input into the item recognition model, and the item features corresponding to each area information are extracted based on the item recognition model, and the item features corresponding to each area information are compared with those in the item feature database.
  • the item features are compared to determine the second-level category of each item in the image to be recognized.
  • the second level category can correspond to the specific information of the item. For example, a certain dish is specifically fried green peppers or fried cabbage.
  • the item features corresponding to the area information of the effective category items in the sample image are annotated, the second annotation information is generated, and the item recognition model is trained based on the sample image and the second annotation information. Compare the output result of the item recognition model with the second label information, determine whether the comparison result meets the requirements of the loss function of building the item recognition model, iterate repeatedly, optimize and adjust the parameters of the item recognition model, so that the comparison result finally meets the construction of the item recognition model The requirements of the loss function, save the item recognition model.
  • an image of the dish is first collected, and the image is input to the item detection module, and the item detection model outputs the area information and category of the dish. Then, the dish feature corresponding to the regional information of the dish is annotated, and the image and the annotation information are input to the item recognition model to train the item recognition model.
  • the item recognition model calls the feature library, compares the output dish features with the dish features stored in the feature library, and recognizes the The specific information corresponding to the dish, for example, whether the dish is fried cabbage or fried green pepper.
  • the image is pre-identified first, the unqualified images are removed, and then the categories of the items in the images that meet the requirements are identified.
  • the invalid category is removed, and only the item features corresponding to the area information of the items belonging to the valid category are identified, and specific items can be identified based on the item features, which improves the accuracy of item identification.
  • the effective item features in the item feature database are determined within a predetermined time; the item features corresponding to each area information are compared with the effective item features in the item feature database to determine each The second level category of the item.
  • the item feature database saves the characteristics of each dish in each period, but the vegetables that make up a certain dish may be slightly different in different seasons, or, in certain periods, certain dishes are no longer on sale. Therefore, it is possible to set the features of dishes not currently participating in the sale as invalid features, and the features of dishes participating in the sale as valid features.
  • identifying a dish compare the characteristics of the dish to be identified with the characteristics of the effective dish in the feature library to determine the specific dish.
  • the item features corresponding to each area information are compared with the valid item features in the item feature library to determine the second-level category of each item in the image to be identified, which can reduce the interference in the item identification process, and further Improve the accuracy of recognition.
  • the item feature corresponding to each item in the item feature library is the category corresponding to the item feature closest to the item feature as the second level category of each item; if the minimum distance is greater than the distance threshold, the user is prompted whether to enter the item category and attribute Information; if you need to enter the item category and attribute, add the item category and attribute information, otherwise, the item feature corresponding to each item in the item feature library is the category corresponding to the item feature closest to the item as the second level category of each item.
  • the distance is, for example, Euclidean distance, and the size of the distance represents the size of similarity. The smaller the distance, the more similar the item feature of the item to be identified and the item feature in the item feature library. When the distance exceeds the distance threshold, it means that the item feature database may not contain the feature of the item to be identified. Therefore, the user can be prompted whether to input item category and attribute information. If the user inputs, it means that a new item needs to be registered. If the user does not input, the category corresponding to the item feature closest to each item is taken as the second level category of each item.
  • the attribute information corresponding to the item is matched.
  • the attribute information is, for example, price. For example, after recognizing that a certain dish is stir-fried cabbage, the price corresponding to the dish can be matched. In the settlement, if there are multiple dishes, the multiple dishes can be settled.
  • the attribute information of the item can be more accurately matched.
  • the attribute information is price information, the accuracy of commodity settlement can be improved.
  • the image to be recognized is marked as a training image or a test image, so that the object detection model and the The item recognition model is trained or tested. For example, if it is recognized that a certain dish is stir-fried cabbage, and the price of stir-fried cabbage is matched, but in actual calculation, the user modifies the settlement price, it means that the dish was identified incorrectly. Therefore, the image containing the dish can be used as a training image or a test image, and the image can be used to train or test the object detection model and the object recognition model. Through automatic iteration of the model, the accuracy of the recognition of the model can be improved.
  • the size information of each item is determined based on the item detection model, and the corresponding attribute information is matched according to the category and size of each item.
  • the attribute information is the price.
  • the size boundary of the large and small dishes can be calculated, that is, the average value of the large dishes and the average value of the small dishes.
  • the size of the recognized dish is compared with the size boundary to determine whether the recognized dish is a large portion or a small portion, and then the corresponding price is matched.
  • the attribute information corresponding to the item combination is matched.
  • the set menu information is configured when the restaurant is settled. If you order 15 yuan for a single fried cabbage, 2 yuan for a bowl of rice, and 16 yuan for a fried cabbage and a rice, it is recognized that the image contains fried After cabbage and rice, you need to match the price of 16 yuan.
  • the matching time of item matching attribute information is determined, and the attribute information corresponding to each item is determined according to the matching time. For example, at the time of restaurant settlement, you can configure the discount period and the discount intensity to determine whether the time when the dish matches the price is in the discount period, and if so, you can match the dish with the discount price corresponding to the discount period.
  • FIG. 3 is a schematic structural diagram of some embodiments of the object identification system of the present disclosure.
  • the system includes an image acquisition module 310, a pre-identification module 320, and an item determination module 330.
  • the image acquisition module 310 is configured to acquire one or more images to be identified, where the images to be identified include one or more objects to be identified.
  • the pre-recognition module 320 is configured to use the trained pre-recognition model to determine whether the probability that the image to be recognized is clear and contains a complete item is greater than a threshold.
  • the pre-recognition model may be trained in advance, a certain number of sample images are collected, and the sample images are classified. For example, the image in the sample image is clear and the image containing the complete item is annotated as a positive sample image, the image in the sample image that is not a positive sample image is annotated as a negative sample image, and the pre-recognition model is performed based on the positive sample image and the negative sample image. training.
  • the item determination module 330 is configured to identify the category of each item when the probability is greater than the threshold. That is, in this embodiment, instead of performing item category recognition on all images, it first determines whether the image meets the requirements, and performs item recognition on the images that meet the requirements.
  • each item in the image whose image is clear and the probability of containing a complete item is greater than the threshold is recognized instead of recognizing all the images, which can improve the accuracy and recognition efficiency of the recognition system.
  • the pre-recognition module 320 is further configured to, if the probability of the first image being clear and containing the complete item is greater than the first threshold, and the other images are clear and containing complete If the probability of the item is greater than the second threshold, the first image of the consecutive multiple images is sent to the item determination module 330.
  • the item determination module 330 is configured to perform category recognition on items contained in the first image of the consecutive plurality of images.
  • the second to Nth images are clear and the probability of containing a complete article is greater than 0.1, not processing the second to Nth images can reduce the processing burden of the article recognition system and improve system stability.
  • Fig. 4 is a schematic structural diagram of other embodiments of the object identification system of the present disclosure.
  • the item determination module 330 in the system includes an item detection module 331, an item management module 332, and an item identification module 333.
  • the item detection module 331 is configured to input the image to be recognized into the item detection model, extract the area information and the first-level category corresponding to each item in the image to be recognized based on the item detection model, call the item management module 332, and set the items belonging to the valid category
  • the area information of the item is input to the item identification module 333.
  • the first-level category refers to the category to which the item belongs, such as dishes, fruits, and beverages.
  • the item detection model can be pre-trained, the area information and the first-level category corresponding to the item in the sample image are annotated, the first annotation information is generated, and the item detection model is performed based on the sample image and the first annotation information. training.
  • the item management module 332 is configured to determine the effective category in the first level category.
  • items such as dishes, yogurt, fruits, and beverages in the recognition area
  • non-valuable items such as keys, badges, wallets, mobile phones, chopsticks, spoons, and hands. Therefore, after the first-level category of the article is identified, the invalid category is removed first, and only the valid category is retained.
  • the item recognition module 333 is configured to input the area information of the items belonging to the valid category in the image to be recognized into the item recognition model, extract the item features corresponding to each area information based on the item recognition model, and combine the item features corresponding to each area information with the item features
  • the features of the items in the library are compared to determine the second-level category of each item in the image to be recognized.
  • the second level category can correspond to the specific information of the item. For example, a certain dish is specifically fried green peppers or fried cabbage.
  • the item features corresponding to the area information of the effective category items in the sample image are annotated, the second annotation information is generated, and the item recognition model is trained based on the sample image and the second annotation information.
  • the image is pre-identified first, the unqualified images are removed, and then the categories of the items in the images that meet the requirements are identified.
  • the invalid category is removed, and only the item features corresponding to the area information of the items belonging to the valid category are identified, and specific items can be identified based on the item features, which improves the accuracy of item identification.
  • the item management module 332 is further configured to determine valid item features in the item feature library within a predetermined time.
  • the item identification module 333 is also configured to compare the item features corresponding to each area information with the effective item features in the item feature library to determine the second-level category of each item in the map to be identified.
  • the item feature database saves the characteristics of each dish in each period, but the vegetables that make up a certain dish may be slightly different in different seasons, or, in certain periods, certain dishes are no longer on sale. Therefore, it is possible to set the features of dishes not currently participating in the sale as invalid features, and the features of dishes participating in the sale as valid features.
  • identifying a dish compare the characteristics of the dish to be identified with the characteristics of the effective dish in the feature library to determine the specific dish.
  • the item features corresponding to each area information are compared with the valid item features in the item feature library to determine the second-level category of each item in the image to be identified, which can reduce the interference in the item identification process, and further Improve the accuracy of recognition.
  • the item recognition module 333 is configured to determine the minimum distance between the item feature of each item and the item feature in the item feature library; if the minimum distance is less than or equal to the distance threshold, the item feature library and The item feature corresponding to each item is the category corresponding to the closest item feature as the second-level category of each item; if the minimum distance is greater than the distance threshold, the user is prompted whether to enter the item category and attribute information; if the item category and attribute information need to be entered Attribute, add item category and attribute information, otherwise, the category corresponding to the item feature closest to the item feature corresponding to each item in the item feature library is taken as the second level category of each item.
  • the distance is, for example, Euclidean distance, and the size of the distance represents the size of similarity. The smaller the distance, the more similar the item feature of the item to be identified and the item feature in the item feature library. When the distance exceeds the distance threshold, it means that the item feature database may not contain the feature of the item to be identified. Therefore, the user can be prompted whether to input item category and attribute information. If the user inputs, it means that a new item needs to be registered. If the user does not input, the category corresponding to the item feature closest to each item is taken as the second level category of each item.
  • the system further includes an attribute matching unit 510 configured to match corresponding attribute information according to the category of each item.
  • the attribute information is, for example, price. For example, after recognizing that a certain dish is stir-fried cabbage, the price corresponding to the dish can be matched. In the settlement, if there are multiple dishes, the multiple dishes can be settled.
  • the attribute information of the item can be more accurately matched.
  • the attribute information is price information, the accuracy of commodity settlement can be improved.
  • the item management module 332 is further configured to, after matching the attribute information, in response to the user modifying the attribute information corresponding to the category of the item, mark the image to be recognized as a training image or a test image so as to be based on
  • the image to be recognized trains or tests the item detection model and the item recognition model. For example, if it is recognized that a certain dish is stir-fried cabbage, and the price of stir-fried cabbage is matched, but in actual calculation, the user modifies the settlement price, it means that the dish was identified incorrectly. Therefore, the image containing the dish can be used as a training image or a test image, and the image can be used to train or test the object detection model and the object recognition model. Through automatic iteration of the model, the accuracy of the recognition of the model can be improved.
  • the attribute matching unit 510 is further configured to match the corresponding attribute information according to the category and size of each item, wherein the item detection module 331 is further configured to determine the size of each item based on the item detection model.
  • the attribute information is the price.
  • the size boundary of the large and small dishes can be calculated, that is, the average value of the large dishes and the average value of the small dishes.
  • the size of the recognized dish is compared with the size boundary to determine whether the recognized dish is a large portion or a small portion, and then the corresponding price is matched.
  • the attribute matching unit 510 is further configured to determine whether multiple items in the image to be identified satisfy the item combination, and if the multiple items satisfy the item combination, match the attribute information corresponding to the item combination.
  • the attribute matching unit 510 is further configured to determine whether the attributes corresponding to multiple items in the image to be identified and whether they meet a preset condition, and if the preset condition is met, the attribute matching unit 510 is matched according to the preset condition. And processing.
  • the attribute matching unit 510 is further configured to determine the matching time of item matching attribute information, and determine the attribute information corresponding to each item according to the matching time.
  • this embodiment includes a registration module 610, a pre-identification module 620, an item detection module 630, an item identification module 640, an item management module 650, a search module 660, a feature library 670, and a settlement module 680.
  • the settlement module 680 corresponds to the attribute matching unit 510.
  • the registration module 610 calls the camera to collect images of the settlement area. For accurate subsequent identification, when registering priced commodities such as dishes and beverages, only one commodity is placed in the settlement area. For example, place only one plate of fried cabbage.
  • the registration module 610 inputs the image to the item detection module 630.
  • the item detection module 630 detects the area information of the item and sends the area information to the item identification module 640.
  • the item identification module 640 extracts the feature of the item and then stores the feature in the feature Library 670.
  • the product is brought to the checkout counter.
  • the settlement module 680 calls the camera to take an image
  • the camera sends the image to the pre-recognition module 620 to determine whether the image is available. That is, it is judged whether the probability that the image is clear and contains the complete product is greater than the threshold, and whether the image is the first image among consecutive images greater than the threshold, and if so, the pre-identification module 620 sends the image to the item detection module 630 .
  • the item detection module 630 detects the category of each item included in the image and outputs area information corresponding to each item.
  • the dish registration desk is usually placed inside the back kitchen to facilitate restaurant staff to register dishes.
  • the back kitchen is usually messy, and there may be some irrelevant things appearing near the registration desk. If non-dish items and other non-price items cannot be filtered, then the non-price items may be entered into the feature database, causing misunderstanding.
  • the collected images often include chopsticks, spoons, badges, mobile phones, wallets, hands and other items. These non-price items may be detected as dishes and cause misunderstanding. Calling the item management module 650 can remove non-valuable items and solve the problem of easy interference in commodity detection.
  • the item management module 650 can configure which types are non-priced items. For example, for example, some restaurants have drinks for sale, and some restaurants do not sell drinks, then the restaurant can configure whether drinks participate in the pricing according to the actual situation. For another example, if the restaurant has fruit delivery activities, you can configure the fruit not to participate in the pricing.
  • keys, badges, wallets, mobile phones, chopsticks, spoons, hands, etc. can be configured in the item management module 650 as non-valuable items.
  • the item management module 650 may also set the feature of the commodity not currently participating in the sale as an invalid feature. For example, the item management module 650 records the dishes and their prices sold at each time of day, and triggers menu synchronization through a timer.
  • the item management module 650 can also process daily order data, count the sales of dishes, and customer order information.
  • the item identification module 640 determines the feature information corresponding to the area information of the commodity in the pricing category, and calls the feature library 670 through the search module 660, and finds the feature closest to the feature of the commodity in the feature library 670, so that the item identification module 640 outputs the corresponding feature of the commodity Specific category, and send the product information to the settlement module 680.
  • a settlement system based on dish identification usually requires dish registration before the meal is opened.
  • some dishes such as temporary dishes, are only served after a certain period of time after the meal is opened. These temporary dishes cannot be registered before the meal is opened. Therefore, the temporary dishes cannot be identified at the time of settlement.
  • the user may be prompted, for example, whether the settlement clerk is registered for temporary dishes. If registered, the registration of the temporary dish is completed by entering the dish name and price, and the dish information is sent to the settlement module 680; if not, the dish information is sent to the settlement module 680 according to the current recognition result.
  • This embodiment can solve the problem that temporary dishes cannot be identified.
  • the settlement module 680 performs settlement according to the commodity category and price.
  • Some dishes in the restaurant have different sizes and sizes, and the prices for large and small portions are different.
  • the large portion of eight-treasure porridge is 6 yuan
  • the small portion is 3 yuan.
  • dishes of large and small portions are basically similar in appearance. Therefore, it is necessary to identify the size information of the dishes, and set the price of the large and small dishes in the settlement module 680.
  • the restaurant sells dishes, there may be set meal discount activities.
  • the unit price of clear soup ramen is 9 yuan
  • the unit price of beef slices is 9 yuan
  • the combination of clear soup ramen and beef slices is 16 yuan.
  • the package price needs to be set in the settlement module 680.
  • Some restaurants may discount certain dishes during certain time periods, for example, in the evening. Therefore, it is also necessary to configure the discount time period and discount strength in the settlement module 680. In some restaurants, there will be a full gift event. Therefore, the full gift information can also be set in the settlement module 680.
  • the accuracy of product identification is improved, the accuracy of product settlement can be improved, user experience can be improved, and the cost of product settlement can be reduced.
  • the system further includes an IoT (Internet of Things) platform 6100, an annotation platform 6110, and an algorithm server 6120.
  • IoT Internet of Things
  • the item management module 650 uploads the wrongly recognized image to the IoT platform 6100, the IoT platform 6100 submits the error data of the day to the labeling platform 6110, and the labeling platform 6100 returns the labelled data to the algorithm server 6120 after completing the labeling.
  • the algorithm server 6120 randomly divides the labeled data into a training set and a test set, and performs model training and model testing to improve model iteration efficiency. Before registering the product, the algorithm server 6120 trains each model in the product recognition process.
  • the registration module 610, the pre-identification module 620, and the settlement module 680 may be installed on the client; the item detection module 630, the item identification module 640, the item management module 650, the search module 660, and the feature library 670 may be installed on the server.
  • the module in the client can communicate with the module in the server through the service module 690; the IoT platform 6100, the annotation platform 6110, and the algorithm server 6120 can be set in the cloud.
  • FIG. 7 is a schematic structural diagram of other embodiments of the object identification system of the present disclosure.
  • the system includes a memory 710 and a processor 720, where the memory 710 may be a magnetic disk, flash memory or any other non-volatile storage medium.
  • the memory is used to store instructions in the embodiments corresponding to FIGS. 1 and 2.
  • the processor 720 is coupled to the memory 710 and may be implemented as one or more integrated circuits, such as a microprocessor or a microcontroller.
  • the processor 720 is configured to execute instructions stored in the memory.
  • the system 800 includes a memory 810 and a processor 820.
  • the processor 820 is coupled to the memory 810 through the BUS bus 830.
  • the system 800 can also be connected to an external storage device 850 through the storage interface 840 to call external data, and can also be connected to the network or another computer system (not shown) through the network interface 860, which will not be described in detail here.
  • the data instructions are stored in the memory, and the above instructions are processed by the processor, which can improve the accuracy of item identification.
  • a computer-readable storage medium has computer program instructions stored thereon, and when the instructions are executed by a processor, the steps of the method in the embodiments corresponding to FIGS. 1 and 2 are implemented.
  • the embodiments of the present disclosure may be provided as methods, devices, or computer program products. Therefore, the present disclosure may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware.
  • the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transitory storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. .
  • These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device.
  • the device implements the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.
  • These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment.
  • the instructions provide steps for implementing functions specified in a flow or multiple flows in the flowchart and/or a block or multiple blocks in the block diagram.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Analysis (AREA)

Abstract

La présente invention concerne un procédé et un système de reconnaissance d'objets, se rapportant au domaine de la reconnaissance d'images. Le procédé consiste : à obtenir une ou plusieurs images devant subir une reconnaissance, lesdites images comprenant un ou plusieurs objets devant subir une reconnaissance (110) ; à utiliser un modèle de pré-reconnaissance appris, et à déterminer si la probabilité selon laquelle lesdites images sont claires et contiennent un objet complet est supérieure à une valeur seuil (120) ; et si la probabilité est supérieure à la valeur seuil, à identifier le type de chacun desdits objets (130). L'invention permet d'améliorer la précision et l'efficacité de la reconnaissance d'objets.
PCT/CN2020/080767 2019-04-23 2020-03-24 Procédé et système de reconnaissance d'objets WO2020215952A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910325808.XA CN111832590B (zh) 2019-04-23 2019-04-23 物品识别方法和系统
CN201910325808.X 2019-04-23

Publications (1)

Publication Number Publication Date
WO2020215952A1 true WO2020215952A1 (fr) 2020-10-29

Family

ID=72912277

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/080767 WO2020215952A1 (fr) 2019-04-23 2020-03-24 Procédé et système de reconnaissance d'objets

Country Status (2)

Country Link
CN (1) CN111832590B (fr)
WO (1) WO2020215952A1 (fr)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113806508A (zh) * 2021-09-17 2021-12-17 平安普惠企业管理有限公司 基于人工智能的多轮对话方法、装置及存储介质
CN116699166A (zh) * 2023-08-08 2023-09-05 国网浙江省电力有限公司宁波供电公司 一种基于视觉识别的油色谱样品自动定位方法及系统
CN117056547A (zh) * 2023-10-13 2023-11-14 深圳博十强志科技有限公司 一种基于图像识别的大数据分类方法和系统
CN117372787A (zh) * 2023-12-05 2024-01-09 同方赛威讯信息技术有限公司 一种图像多类别识别方法及装置

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113673576A (zh) * 2021-07-26 2021-11-19 浙江大华技术股份有限公司 图像检测方法、终端及其计算机可读存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9141883B1 (en) * 2015-05-11 2015-09-22 StradVision, Inc. Method, hard negative proposer, and classifier for supporting to collect hard negative images using a similarity map
CN106096932A (zh) * 2016-06-06 2016-11-09 杭州汇萃智能科技有限公司 基于餐具形状的菜品自动识别系统的计价方法
CN107122730A (zh) * 2017-04-24 2017-09-01 乐金伟 自选餐厅自动计价方法
CN107145879A (zh) * 2017-06-23 2017-09-08 依通（北京）科技有限公司 一种植物种类自动识别方法及系统
CN108256474A (zh) * 2018-01-17 2018-07-06 百度在线网络技术（北京）有限公司 用于识别菜品的方法和装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9141883B1 (en) * 2015-05-11 2015-09-22 StradVision, Inc. Method, hard negative proposer, and classifier for supporting to collect hard negative images using a similarity map
CN106096932A (zh) * 2016-06-06 2016-11-09 杭州汇萃智能科技有限公司 基于餐具形状的菜品自动识别系统的计价方法
CN107122730A (zh) * 2017-04-24 2017-09-01 乐金伟 自选餐厅自动计价方法
CN107145879A (zh) * 2017-06-23 2017-09-08 依通（北京）科技有限公司 一种植物种类自动识别方法及系统
CN108256474A (zh) * 2018-01-17 2018-07-06 百度在线网络技术（北京）有限公司 用于识别菜品的方法和装置

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113806508A (zh) * 2021-09-17 2021-12-17 平安普惠企业管理有限公司 基于人工智能的多轮对话方法、装置及存储介质
CN116699166A (zh) * 2023-08-08 2023-09-05 国网浙江省电力有限公司宁波供电公司 一种基于视觉识别的油色谱样品自动定位方法及系统
CN116699166B (zh) * 2023-08-08 2024-01-02 国网浙江省电力有限公司宁波供电公司 一种基于视觉识别的油色谱样品自动定位方法及系统
CN117056547A (zh) * 2023-10-13 2023-11-14 深圳博十强志科技有限公司 一种基于图像识别的大数据分类方法和系统
CN117056547B (zh) * 2023-10-13 2024-01-26 深圳博十强志科技有限公司 一种基于图像识别的大数据分类方法和系统
CN117372787A (zh) * 2023-12-05 2024-01-09 同方赛威讯信息技术有限公司 一种图像多类别识别方法及装置
CN117372787B (zh) * 2023-12-05 2024-02-20 同方赛威讯信息技术有限公司 一种图像多类别识别方法及装置

Also Published As

Publication number Publication date
CN111832590B (zh) 2024-03-05
CN111832590A (zh) 2020-10-27

Similar Documents

Publication Publication Date Title
WO2020215952A1 (fr) Procédé et système de reconnaissance d'objets
Aguilar et al. Grab, pay, and eat: Semantic food detection for smart restaurants
US11151427B2 (en) Method and apparatus for checkout based on image identification technique of convolutional neural network
RU2739542C1 (ru) Система автоматической регистрации для торговой точки
WO2019165892A1 (fr) Procédé et appareil de distribution automatique, et support de stockage lisible par ordinateur
US10628695B2 (en) Fast item identification for checkout counter
JPWO2015140853A1 (ja) Ｐｏｓ端末装置、ｐｏｓシステム、商品認識方法及びプログラム
WO2019096222A1 (fr) Procédé et dispositif de vente automatique basés sur une identification d'identité et une identification de produit
US11354549B2 (en) Method and system for region proposal based object recognition for estimating planogram compliance
CN109766962B (zh) 一种商品识别方法、存储介质及商品识别系统
CN104978585A (zh) 一种自动计价方法
CN111126990B (zh) 物品自动识别方法、结算方法、装置、终端及存储介质
WO2021179138A1 (fr) Procédé et système permettant d'analyser des produits sur une étagère de supermarché
JP2016024596A (ja) 情報処理装置
US11562338B2 (en) Automated point of sale systems and methods
JP2015194791A (ja) Ｐｏｓ端末装置
US20180211297A1 (en) Automatic Update Of A Shadow Catalog Database Based On Transactions Made Using A Catalog Database
CN110019983B (zh) 标签结构的扩展方法、装置及电子设备
CN112508659B (zh) 商品结算处理方法、装置及计算设备、计算机存储介质
Achakir et al. An automated AI-based solution for out-of-stock detection in retail environments
CN111415328B (zh) 物品分析数据确定方法、装置、电子设备
JP7339630B1 (ja) 情報処理システム
CN110781752A (zh) 一种动态视觉智能柜多品类场景下的物体识别方法
CN111126110A (zh) 一种商品信息识别方法、结算方法、装置及无人零售系统
CN113362543B (zh) 结算方法、装置、电子设备以及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20795192

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 01.03.2022)

122 Ep: pct application non-entry in european phase

Ref document number: 20795192

Country of ref document: EP

Kind code of ref document: A1