CN113469044B - Dining recording system and method - Google Patents
Dining recording system and method Download PDFInfo
- Publication number
- CN113469044B CN113469044B CN202110739335.5A CN202110739335A CN113469044B CN 113469044 B CN113469044 B CN 113469044B CN 202110739335 A CN202110739335 A CN 202110739335A CN 113469044 B CN113469044 B CN 113469044B
- Authority
- CN
- China
- Prior art keywords
- meal
- record
- dining
- dish
- face
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 24
- 235000012054 meals Nutrition 0.000 claims abstract description 220
- 235000003166 Opuntia robusta Nutrition 0.000 claims abstract description 59
- 244000218514 Opuntia robusta Species 0.000 claims abstract description 59
- 238000013527 convolutional neural network Methods 0.000 claims description 37
- 238000012937 correction Methods 0.000 claims description 16
- 238000012216 screening Methods 0.000 claims description 15
- 230000015572 biosynthetic process Effects 0.000 claims description 12
- 238000003786 synthesis reaction Methods 0.000 claims description 12
- 238000000605 extraction Methods 0.000 claims description 11
- 238000001514 detection method Methods 0.000 claims description 6
- 238000009499 grossing Methods 0.000 claims description 6
- 230000002452 interceptive effect Effects 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 6
- 238000012549 training Methods 0.000 claims description 6
- 238000013528 artificial neural network Methods 0.000 claims description 5
- 238000003062 neural network model Methods 0.000 claims 1
- 235000013311 vegetables Nutrition 0.000 abstract 3
- XXQCMVYBAALAJK-UHFFFAOYSA-N ethyl n-[4-[benzyl(2-phenylethyl)amino]-2-(2-phenylethyl)-1h-imidazo[4,5-c]pyridin-6-yl]carbamate Chemical compound N=1C=2C(N(CCC=3C=CC=CC=3)CC=3C=CC=CC=3)=NC(NC(=O)OCC)=CC=2NC=1CCC1=CC=CC=C1 XXQCMVYBAALAJK-UHFFFAOYSA-N 0.000 description 6
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 230000002194 synthesizing effect Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000002354 daily effect Effects 0.000 description 1
- 238000013481 data capture Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 235000013305 food Nutrition 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/08—Payment architectures
- G06Q20/20—Point-of-sale [POS] network systems
- G06Q20/208—Input by product or record sensing, e.g. weighing or scanner processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/38—Payment protocols; Details thereof
- G06Q20/40—Authorisation, e.g. identification of payer or payee, verification of customer or shop credentials; Review and approval of payers, e.g. check credit lines or negative lists
- G06Q20/401—Transaction verification
- G06Q20/4014—Identity check for transactions
- G06Q20/40145—Biometric identity checks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/12—Hotels or restaurants
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
- G07G1/00—Cash registers
- G07G1/0036—Checkout procedures
- G07G1/0045—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
- G07G1/00—Cash registers
- G07G1/12—Cash registers electronically operated
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Accounting & Taxation (AREA)
- Health & Medical Sciences (AREA)
- General Business, Economics & Management (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- General Engineering & Computer Science (AREA)
- Molecular Biology (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- Data Mining & Analysis (AREA)
- Biophysics (AREA)
- Finance (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Primary Health Care (AREA)
- Marketing (AREA)
- Computer Security & Cryptography (AREA)
- Human Resources & Organizations (AREA)
- Economics (AREA)
- Image Analysis (AREA)
- Collating Specific Patterns (AREA)
Abstract
The dining recording system and method of the invention comprises: the face acquisition module acquires a face image at an inlet of the dining queue as a first face image, acquires a face image at an outlet of the dining queue as a second face image, matches the second face image with the first face image, and sends a settlement starting signal and a successfully matched face image when matching is successful; the identity recognition and authentication module compares the successfully matched face image with a face image in a user account, a dining record is created in the user account when the recognition is successful, a new user account is created when the recognition is failed, and a dining record is created in the user account; the meal record establishes the module and receives and scans the dinner plate after the settlement start signal and places the region in order to extract the dinner plate picture, requests the record serial number of having a meal, records the record time of having a meal at present, draws every vegetable picture in the dinner plate picture, discerns the meal vegetable that each vegetable picture corresponds and corresponds the meal quantity, gathers these some data finally and constitutes the record of having a meal.
Description
Technical Field
The invention relates to the technical field of dining recording, in particular to a dining recording system and a dining recording method.
Background
The common means of keeping a food order record are as follows:
1. manually recording the dining record: the most common recording method is to manually record daily dining conditions and arrange a dining condition table by self.
2. Recording by APP: some APPs support the user to upload the photo of having a dinner every day by oneself, form the record of having a dinner, and some APPs provide further function, try to use image recognition to judge the dishes in the photo of having a dinner through artificial intelligence's mode, form the record of having a dinner.
The disadvantages of the prior art solutions are listed below:
1. manually recording the dining record: the manual recording of the dining condition is troublesome in operation and poor in compliance, and meanwhile, as the recorder does not have a standard optional description method, the impression of the recorder on specific matters can be faded even after a long time, so that the reference value of the manually recorded dining record on medical treatment is limited.
2. Recording by APP: APP records require users to understand how to register, how to upload meal photographs, how to ensure that photographs taken meet requirements, etc., which constitute a high threshold for users, especially for most elderly people, the biggest hurdle preventing their use.
Disclosure of Invention
Aiming at the problems and the defects in the prior art, the invention provides a dining recording system and a method.
The invention solves the technical problems through the following technical scheme:
the invention provides a dining recording system which is characterized by comprising a face acquisition module, an identity recognition and authentication module and a dining record creation module;
the face acquisition module is used for continuously acquiring faces at an entrance of a dining queue and extracting face images to be used as first face images, continuously acquiring faces at an exit of the dining queue and extracting face images to be used as second face images, matching the current second face images with all the first face images, sending a settlement starting signal and successfully matched face images to the dining record creation module when matching is successful, sending the successfully matched face images to the identity recognition authentication module, and requesting identity recognition;
the identity recognition and authentication module is used for comparing the received successfully matched face image with face images prestored in all user accounts so as to recognize the identity of the dining user, when the recognition is successful, a dining record is created in the corresponding user account, when the recognition is failed, a new user account is created, and a dining record is created in the user account, wherein the dining record comprises a dining record number, the dining record time, the dining dishes, the number of meals corresponding to each kind of dining dish and a settlement transaction record;
the meal record creating module is used for continuously scanning the meal plate placing area after receiving the settlement starting signal to extract a meal plate picture, requesting a meal record number corresponding to the successfully matched face image from the identity recognition and authentication module, recording the current meal record time, extracting each dish picture in the dinner plate picture through an image processing technology, identifying the dishes corresponding to each dish picture and the meal quantity corresponding to each dish picture by utilizing a neural network algorithm, sending each dish and the corresponding meal quantity to a restaurant settlement system for transaction settlement, summarizing a meal record number, a meal record time, each dish, the meal quantity corresponding to each dish and the settlement transaction record of the restaurant settlement system to form a meal record, and transmitting the meal record to the meal record corresponding to the meal record number in the user account.
Preferably, the face acquisition module comprises a control camera positioned at an entrance of the dining queue, a settlement camera positioned at an exit of the dining queue, a target face temporary library and a face matching submodule;
the control camera is used for continuously scanning the human face in the field of view, extracting human face images, screening the human face images with the strongest identifiability from the human face images, and transmitting the human face images serving as first human face images to a target human face temporary library;
the settlement camera is used for continuously scanning the human face in the field of view, extracting human face images, screening the human face images with the strongest identifiability from the human face images, and transmitting the human face images serving as second human face images to the human face matching submodule;
the face matching submodule is used for carrying out face matching on the current second face image and a first face image in the target face temporary library, sending a matching success signal to the target face temporary library when the matching is successful, simultaneously sending a settlement starting signal and a successfully matched face image to the dining record creating module, sending the successfully matched face image to the identity recognition authentication module and requesting identity recognition;
the target face temporary library is used for receiving a first face image transmitted by the control camera, deleting the successfully matched face image in the target face temporary library after receiving the successfully matched signal, and deleting the temporarily stored first face image after the temporary storage time of the first face image in the target face temporary library reaches the set time.
Preferably, the identity identification authentication module comprises an identity identification submodule, a user management submodule and a registration management submodule;
the identity recognition sub-module is used for comparing the received successfully matched face image with a face image prestored in a user account of the user management sub-module so as to recognize the identity of the dining user, when the recognition is successful, a dining record is created in the corresponding user account, when the recognition is failed, a new user account is created, and a dining record is created in the user account;
the user management submodule is used for creating, maintaining and managing a user account, and the user account comprises a user ID, a prestored face image and a dining record;
the registration management submodule is used for managing the registration level of the user, and the registration level is divided into an unauthorized user level, an authorized user level, a registered user level, an authenticated user level and a paid user level.
Preferably, the meal record creating module comprises a meal plate camera positioned at the exit of the meal queue, a matching control submodule, a meal plate extracting submodule, a dish identifying submodule, a dish quantity identifying submodule, a POS record extracting submodule and a record synthesizing submodule, wherein the meal plate camera is connected with the settlement camera in series and is aligned with the meal plate placing area;
the dinner plate camera is used for continuously scanning the dinner plate placing area, extracting a dinner plate picture and transmitting the dinner plate picture to the matching control sub-module;
the matching control sub-module is used for recording the current meal receiving the settlement starting signalRecording time is set as t0Requesting the identity recognition and authentication module for the meal record number corresponding to the successfully matched face image, and searching whether a time window t is present in the transmitted meal plate picture [ t ═ t [ [ t ]0-τ1,t0-τ2]Pushing the picture internally, adding the meal record number to the found first meal plate picture, pushing the picture to a meal plate extraction submodule, and tau1、τ2Is a time window parameter;
the dinner plate extraction submodule is used for converting a first dinner plate picture into a gray image, executing a boundary detection algorithm on the gray image, taking out the boundary of a dinner plate containing dishes from the gray image, forming a closed polygon on the extracted boundary by using a boundary smoothing method, and extracting the picture of each dish from the first dinner plate picture by using the polygon;
the dish identification submodule is used for being realized by using a CNN convolutional neural network model, the input of the model is extracted dish pictures, and the output of the model is meal dishes corresponding to the dish pictures;
the dish quantity identification submodule is used for being realized by using a CNN convolutional neural network model, the input of the model is dish pictures marked with the dishes for each meal, the dish quantity corresponding to each dish for each meal is output, and each dish for each meal and the corresponding meal quantity are sent to the restaurant settlement system for transaction settlement;
the POS record extraction submodule is used for extracting t-t from a POS machine of the restaurant settlement system0>0, the first settlement transaction record is pushed to a record synthesis submodule after the dining record number is added;
the record synthesis submodule is used for summarizing the dining record number, the dining record time, each dining dish, the corresponding dining quantity of each dining dish and the settlement transaction record to form a dining record, and transmitting the dining record to the dining record corresponding to the dining record number in the user account.
Preferably, the record synthesis submodule is used for receiving correction of the dining record by the user through an interactive applet, and generating a final dining record;
the meal record creating module further comprises an online correction training submodule, wherein the online correction training submodule is used for comparing the output result of the meal volume identification submodule by using the final meal record from the record synthesis submodule, screening out the final meal record inconsistent with the output result to generate model difference data, and the model difference data is used for online correction of a CNN convolutional neural network model of the meal volume identification submodule so as to continuously optimize the identification quality of the model.
The invention also provides a dining recording method, which is characterized by comprising the following steps:
s1, continuously collecting the faces at the entrance of the dining queue and extracting face images to be used as first face images, continuously collecting the faces at the exit of the dining queue and extracting face images to be used as second face images, matching the current second face images with all the first face images, sending a settlement starting signal and successfully matched face images to create a dining record when matching is successful, and sending the successfully matched face images to request identity recognition;
s2, comparing the received successfully matched face image with face images prestored in all user accounts to identify the identity of a dining user, creating a dining record in the corresponding user account when the identification is successful, creating a new user account when the identification is failed, and creating a dining record in the user account, wherein the dining record comprises a dining record number, dining record time, dining dishes, the number of meals corresponding to each kind of dining dish and a settlement transaction record;
s3, continuously scanning a dinner plate placing area after receiving a settlement starting signal to extract dinner plate pictures, requesting meal record numbers corresponding to successfully matched face images, recording current meal record time, extracting each dish picture in the dinner plate pictures through an image processing technology, identifying meal dishes corresponding to each dish picture and meal quantity corresponding to each meal dish by utilizing a neural network algorithm, sending each meal dish and corresponding meal quantity to a restaurant settlement system for transaction settlement, summarizing the meal record numbers, the meal record time, each meal dish, the meal quantity corresponding to each meal dish and the settlement transaction records of the restaurant settlement system to form meal records, and transmitting the meal records to the meal records corresponding to the meal record numbers in a user account.
Preferably, step S1 includes:
s11, continuously scanning the faces in the field of view by using a control camera positioned at the entrance of the dining queue, extracting face images, screening the face images with the strongest identifiability from the face images, and transmitting the face images serving as first face images to a target face temporary library;
s12, continuously scanning the faces in the field of view by using a settlement camera positioned at the exit of the dining queue, extracting face images, screening the face images with the strongest identifiability from the face images, and taking the face images as second face images;
s13, carrying out face matching on the current second face image and the first face image in the target face temporary library, sending a matching success signal to the target face temporary library when matching is successful, sending a settlement starting signal and the successfully matched face image to create a dining record, and sending the successfully matched face image to request identity recognition;
and S14, the target face temporary library receives the first face image transmitted by the control camera, deletes the successfully matched face image in the target face temporary library after receiving the successfully matched signal, and deletes the temporarily stored first face image after the temporary storage time of the first face image in the target face temporary library reaches the set time.
Preferably, step S2 includes:
comparing the received successfully matched face image with a face image prestored in a user account to identify the identity of a dining user, creating a dining record in the corresponding user account when the identification is successful, creating a new user account when the identification is failed, and creating a dining record in the user account;
creating, maintaining and managing a user account, wherein the user account comprises a user ID, a pre-stored face image and a dining record;
and managing the registration levels of the users, wherein the registration levels are divided into an unauthorized user level, an authorized user level, a registered user level, an authenticated user level and a paid user level.
Preferably, step S3 includes:
s31, continuously scanning a dinner plate placing area by using a dinner plate camera positioned at the exit of the dining queue to extract a dinner plate picture;
s32, setting the current meal recording time when the settlement starting signal is received as t0Requesting the meal record number corresponding to the successfully matched face image, and searching whether a time window t is present in the transmitted meal plate picture [ t ═ t [ [ t ]0-τ1,t0-τ2]The picture is pushed in, the found first dinner plate picture is pushed after being added with the dinner record number, and tau1、τ2Is a time window parameter;
s33, converting the first dinner plate picture into a gray image, executing a boundary detection algorithm on the gray image, taking out the boundary of the dinner plate containing dishes, forming a closed polygon on the extracted boundary by using a boundary smoothing method, and extracting the picture of each dish from the first dinner plate picture by using the polygon;
s34, dish identification is achieved through the CNN convolutional neural network model, the model is input into each extracted dish picture, and the model is output into a meal dish corresponding to each dish picture;
s35, dish quantity recognition is achieved by using a CNN convolutional neural network model, the input of the model is dish pictures marked with meal dishes, the output is meal quantity corresponding to each meal dish, and each meal dish and the corresponding meal quantity are sent to a restaurant settlement system for transaction settlement;
s36, extracting t-t from POS machine of restaurant settlement system0>0, the first settlement transaction record;
s37, summarizing the dining record number, the dining record time, each dining item, the corresponding dining quantity of each dining item and the settlement transaction record to form a dining record, and transmitting the dining record to the dining record corresponding to the dining record number in the user account.
Preferably, the step S3 further includes: receiving the correction of a user on the dining record through an interactive applet, and generating a final dining record;
and comparing the final meal record with the dish quantity recognition output result, and screening out the final meal record inconsistent with the output result to generate model difference data, wherein the model difference data is used for online correction of the CNN convolutional neural network model for dish quantity recognition so as to continuously optimize the recognition quality of the model.
On the basis of the common knowledge in the field, the above preferred conditions can be combined randomly to obtain the preferred embodiments of the invention.
The positive progress effects of the invention are as follows:
1. the user use threshold is low: compared with the traditional dining recording mode, the dining recording method and the dining recording system greatly reduce the use threshold of the user, and the user can obtain complete, comprehensive and accurate dining records without any operation as long as the user eats in the dining place provided by the invention, thereby ensuring high compliance of the dining records and integrity of collected data.
2. Dish identification and dish quantity identification are accurate: the invention can accurately identify each meal dish in the dinner plate and the meal number corresponding to each dish.
3. Convenient deployment capability: the invention has low requirements on the management system and the management mode of the restaurant and strong self-adaptive capacity, thereby being conveniently deployed in restaurants with various management levels and ensuring the applicability of the scheme.
Drawings
Fig. 1 is a block diagram of a meal recording system according to a preferred embodiment of the present invention.
FIG. 2 is a flowchart of a meal recording method according to a preferred embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
As shown in fig. 1, the present embodiment provides a meal record system, which includes a human face acquisition module 1, an identity recognition and authentication module 2, and a meal record creation module 3.
The face acquisition module 1 is used for continuously acquiring faces at an entrance of a dining queue and extracting face images to serve as first face images, continuously acquiring faces at an exit of the dining queue and extracting face images to serve as second face images, matching the current second face images with all the first face images, sending a settlement starting signal and successfully matched face images to the dining record creation module 3 when matching is successful, sending the successfully matched face images to the identity recognition authentication module 2, and requesting identity recognition.
Specifically, the face acquisition module 1 includes a control camera 11 located at an entrance of a dining queue, a settlement camera 12 located at an exit of the dining queue, a target face temporary library 13, and a face matching sub-module 14.
The control camera 11 is configured to continuously scan faces in the field of view, extract face images, screen out a face image with the strongest identifiability from the face images, and transmit the face image as a first face image to the target face temporary library 13.
The settlement camera 12 is configured to continuously scan faces in the field of view, extract face images, screen out a face image with the strongest identifiability from the face images, and transmit the face image as a second face image to the face matching sub-module 14.
The face matching sub-module 14 is configured to perform face matching on the current second face image and the first face image in the target face temporary library 13, send a matching success signal to the target face temporary library 13 when matching is successful (i.e., the second face image of the settlement camera is matched with the first face image of the control camera), send a settlement starting signal and the successfully matched face image to the meal record creating module 3, send the successfully matched face image to the identity recognition and authentication module 2, and request identity recognition.
The target face temporary library 13 is configured to receive the first face image transmitted by the control camera 11, delete the successfully matched face image therein after receiving the matching success signal, and delete the temporarily stored first face image when the temporary storage time of the first face image in the target face temporary library 13 reaches a set time (for example, 1 hour).
The identity recognition and authentication module 2 is configured to compare the received successfully-matched face image with face images prestored in all user accounts to recognize identities of dining users, create a dining record in the corresponding user account when recognition is successful, create a new user account when recognition is failed, and create a dining record in the user account, where the dining record includes a dining record number, a dining record time, dining dishes, the number of meals corresponding to each kind of dining dish, and a settlement transaction record.
Specifically, the identity authentication module 2 includes an identity sub-module 21, a user management sub-module 22 and a registration management sub-module 23.
The identity recognition submodule 21 is configured to compare the received successfully matched face image with a face image prestored in a user account of the user management submodule 22 to recognize an identity of the dining user, create a dining record in the corresponding user account when the recognition is successful, create a new user account when the recognition is failed, and create a dining record in the user account.
The user management submodule 22 is configured to create, maintain and manage a user account, where the user account includes a user ID, a pre-stored face image and a dining record.
The registration management sub-module 23 is configured to manage registration levels of users, where the registration levels are divided into an unauthorized user level, an authorized user level, a registered user level, an authenticated user level, and a paid user level.
The meal record creating module 3 is used for continuously scanning the meal plate placing area after receiving the settlement starting signal to extract a meal plate picture, requesting the identity identification authentication module 2 for a meal record number corresponding to the successfully matched face image, recording the current meal record time, extracting each dish picture in the dinner plate picture through an image processing technology, identifying the dishes corresponding to each dish picture and the meal quantity corresponding to each dish picture by utilizing a neural network algorithm, sending each dish and the corresponding meal quantity to a restaurant settlement system for transaction settlement, summarizing a meal record number, a meal record time, each dish, the meal quantity corresponding to each dish and the settlement transaction record of the restaurant settlement system to form a meal record, and transmitting the meal record to the meal record corresponding to the meal record number in the user account.
Specifically, the meal record creating module 3 includes a meal plate camera 31, a matching control sub-module 32, a meal plate extracting sub-module 33, a dish identification sub-module 34, a dish quantity identification sub-module 35, a POS record extracting sub-module 36, a record synthesizing sub-module 37 and an online correction training sub-module 38, which are located at the exit of the meal queue, and the meal plate camera 31 is connected in series with the settlement camera 12 and aligned with the meal plate placement area.
The dinner plate camera 31 is used for continuously scanning the dinner plate placing area, extracting a dinner plate picture and transmitting the picture to the matching control sub-module 32.
The matching control sub-module 32 is used for setting the current meal recording time when the settlement starting signal is received as t0Requesting the identity recognition sub-module 21 for the meal record number corresponding to the successfully matched face image, and searching whether a time window t is present in the transmitted meal plate picture [ t ═ t [ [ t ]0-τ1,t0-τ2]The pushed pictures are pushed in, the found first dinner plate picture is added with the dinner record number and pushed to the dinner plate extraction submodule 33, tau1、τ2Is a time window parameter.
The dinner plate extraction submodule 33 is configured to convert the first dinner plate picture into a grayscale image, execute a boundary detection algorithm on the grayscale image, extract a boundary of a dinner plate containing dishes from the grayscale image, form a closed polygon on the extracted boundary by using a boundary smoothing method, and extract a picture of each dish from the first dinner plate picture by using the polygon;
the dish identification submodule 34 is configured to be implemented by using a CNN convolutional neural network model, where the input of the model is each extracted dish picture, and the output is a meal corresponding to each dish picture.
The dish quantity identification submodule 35 is configured to be implemented by using a CNN convolutional neural network model, where the input of the model is a dish picture labeled with each meal dish, and the output is a meal quantity corresponding to each meal dish, so as to send each meal dish and the corresponding meal quantity to the restaurant settlement system for transaction settlement.
The POS record extraction submodule 36 is connected to the POS machine of the restaurant settlement system, and is used for extracting t-t from the POS machine of the restaurant settlement system0>The first settlement transaction record of 0 is pushed to the record composition submodule 37 after the meal record number is added.
The record synthesis submodule 37 is configured to collect the meal record number, the meal record time, each meal item, the meal quantity corresponding to each meal item, and the settlement transaction record to form a meal record, and transmit the meal record to the meal record corresponding to the meal record number in the user account of the user management submodule.
In addition, the record synthesis sub-module 37 is configured to receive a correction of the meal record by the user through the interactive applet, and generate a final meal record.
The online modification training sub-module 38 is configured to compare the output result of the meal volume identification sub-module with the final meal record from the record synthesis sub-module, and screen out the final meal record inconsistent with the output result to generate model difference data, where the model difference data is used for online modification of the CNN convolutional neural network model of the meal volume identification sub-module to continuously optimize the identification quality of the model.
As shown in fig. 2, the present embodiment further provides a meal recording method, which includes the following steps:
And 102, continuously scanning the human faces in the field of view by using a settlement camera positioned at the exit of the dining queue, extracting human face images, screening the human face images with the strongest identifiability from the human face images, and taking the human face images as second human face images.
And 103, carrying out face matching on the current second face image and the first face image in the target face temporary library, sending a matching success signal to the target face temporary library when matching is successful, sending a settlement starting signal and the successfully matched face image to create a dining record, and sending the successfully matched face image to request identity recognition.
And 104, receiving the first face image transmitted by the control camera by the target face temporary library, deleting the successfully matched face image in the target face temporary library after receiving the successfully matched signal, and deleting the temporarily stored first face image after the temporary storage time of the first face image in the target face temporary library reaches a set time (such as 1 hour).
And 105, comparing the received successfully matched face image with a face image prestored in a user account to identify the identity of the dining user, creating a dining record in the corresponding user account when the identification is successful, creating a new user account when the identification is failed, and creating a dining record in the user account, wherein the dining record comprises a dining record number, a dining record time, dining dishes, the number of meals corresponding to each kind of dish and a settlement transaction record.
And step 106, continuously scanning the dinner plate placing area by using the dinner plate camera positioned at the exit of the dining queue to extract the dinner plate picture.
And 108, converting the first dinner plate picture into a gray image, executing a boundary detection algorithm on the gray image, taking out the boundaries of the dinner plate containing dishes, forming a closed polygon by using a boundary smoothing method on the extracted boundaries, and extracting a picture of each dish from the first dinner plate picture by using the polygon.
And step 109, identifying dishes by using the CNN convolutional neural network model, inputting each extracted dish picture by the model, and outputting the dish pictures corresponding to each dish picture.
And 110, identifying the dish quantity by using a CNN convolutional neural network model, inputting a dish picture marked with each meal dish by the model, outputting the meal quantity corresponding to each meal dish, and sending each meal dish and the corresponding meal quantity to a restaurant settlement system for transaction settlement.
And step 112, summarizing the meal record number, the meal record time, each meal dish, the meal quantity corresponding to each meal dish and the settlement transaction record to form a meal record, and transmitting the meal record to the meal record corresponding to the meal record number in the user account.
Furthermore, the method further comprises: receiving the correction of a user on the dining record through an interactive applet, and generating a final dining record; and comparing the final meal record with the dish quantity recognition output result, and screening out the final meal record inconsistent with the output result to generate model difference data, wherein the model difference data is used for online correction of the CNN convolutional neural network model for dish quantity recognition so as to continuously optimize the recognition quality of the model.
In summary, in order to solve the problem of data integration from different sources, the present solution creatively designs a data capture mechanism triggered by multi-camera motion, so as to ensure the quality of the integrated data, which is a technical key point.
According to the scheme, the independent picture of each dish is obtained through an image processing technology, so that the difficulty in identifying the dish type and the dish quantity by using a CNN convolutional neural network model subsequently is greatly simplified, the identification speed and the identification accuracy are improved, and the method is a technical key point.
While specific embodiments of the invention have been described above, it will be appreciated by those skilled in the art that these are by way of example only, and that the scope of the invention is defined by the appended claims. Various changes and modifications to these embodiments may be made by those skilled in the art without departing from the spirit and scope of the invention, and these changes and modifications are within the scope of the invention.
Claims (10)
1. A dining record system is characterized by comprising a face acquisition module, an identity identification and authentication module and a dining record creation module;
the face acquisition module is used for continuously acquiring faces at an entrance of a dining queue and extracting face images to be used as first face images, continuously acquiring faces at an exit of the dining queue and extracting face images to be used as second face images, matching the current second face images with all the first face images, sending a settlement starting signal and successfully matched face images to the dining record creation module when matching is successful, sending the successfully matched face images to the identity recognition authentication module, and requesting identity recognition;
the identity recognition and authentication module is used for comparing the received successfully matched face image with face images prestored in all user accounts so as to recognize the identity of the dinning user, when the recognition is successful, a dinning record is created in the corresponding user account, when the recognition is failed, a new user account is created, and a dinning record is created in the user account, wherein the dinning record comprises a dinning record number, a dinning record time entry, a dinning dish entry, a dinning quantity entry corresponding to each dinning dish and a settlement transaction record entry;
the meal record creating module is used for continuously scanning the meal plate placing area after receiving the settlement starting signal to extract a meal plate picture, requesting a meal record number corresponding to the successfully matched face image from the identity recognition and authentication module, recording the current meal record time, extracting each dish picture in the dinner plate picture through an image processing technology, identifying the dishes corresponding to each dish picture and the meal quantity corresponding to each dish picture by utilizing a neural network algorithm, sending each dish and the corresponding meal quantity to a restaurant settlement system for transaction settlement, summarizing a meal record number, a meal record time, each dish, the meal quantity corresponding to each dish and the settlement transaction record of the restaurant settlement system to form a meal record, and transmitting the dining record to each corresponding item in the dining record corresponding to the dining record number in the user account.
2. The meal recording system according to claim 1, wherein the face acquisition module comprises a control camera located at an entrance of the meal queue, a settlement camera located at an exit of the meal queue, a target face temporary library and a face matching submodule;
the control camera is used for continuously scanning the human face in the field of view, extracting human face images, screening the human face images with the strongest identifiability from the human face images, and transmitting the human face images serving as first human face images to a target human face temporary library;
the settlement camera is used for continuously scanning the human face in the field of view, extracting human face images, screening the human face images with the strongest identifiability from the human face images, and transmitting the human face images serving as second human face images to the human face matching submodule;
the face matching submodule is used for carrying out face matching on the current second face image and the first face image in the target face temporary library, sending a matching success signal to the target face temporary library when the matching is successful, simultaneously sending a settlement starting signal and the successfully matched face image to the dining record creating module, sending the successfully matched face image to the identity recognition authentication module and requesting identity recognition;
the target face temporary library is used for receiving a first face image transmitted by the control camera, deleting the successfully matched face image in the target face temporary library after receiving the successfully matched signal, and deleting the temporarily stored first face image after the temporary storage time of the first face image in the target face temporary library reaches the set time.
3. The meal recording system of claim 1, wherein the identification authentication module comprises an identification sub-module, a user management sub-module, and a registration management sub-module;
the identity recognition sub-module is used for comparing the received successfully matched face image with a face image prestored in a user account of the user management sub-module so as to recognize the identity of the dining user, when the recognition is successful, a dining record is created in the corresponding user account, when the recognition is failed, a new user account is created, and a dining record is created in the user account;
the user management submodule is used for creating, maintaining and managing a user account, and the user account comprises a user ID, a prestored face image and a dining record;
the registration management submodule is used for managing the registration level of the user, and the registration level is divided into an unauthorized user level, an authorized user level, a registered user level, an authenticated user level and a paid user level.
4. The meal record system of claim 1, wherein the meal record creation module comprises a meal plate camera at the exit of the meal queue, a matching control sub-module, a meal plate extraction sub-module, a dish identification sub-module, a dish quantity identification sub-module, a POS record extraction sub-module and a record synthesis sub-module, wherein the meal plate camera is connected in series with the settlement camera and is aligned with a meal plate placement area;
the dinner plate camera is used for continuously scanning the dinner plate placing area, extracting a dinner plate picture and transmitting the dinner plate picture to the matching control sub-module;
the matching control submodule is used for setting the current meal recording time of the received settlement starting signal as t0Requesting the identity recognition and authentication module for the meal record number corresponding to the successfully matched face image, and searching whether a time window t is present in the transmitted meal plate picture [ t ═ t [ [ t ]0-τ1,t0-τ2]Pushing the picture internally, adding the meal record number to the found first meal plate picture, pushing the picture to a meal plate extraction submodule, and tau1、τ2Is a time window parameter;
the dinner plate extraction submodule is used for converting a first dinner plate picture into a gray image, executing a boundary detection algorithm on the gray image, taking out the boundary of a dinner plate containing dishes from the gray image, forming a closed polygon on the extracted boundary by using a boundary smoothing method, and extracting the picture of each dish from the first dinner plate picture by using the polygon;
the dish identification submodule is used for being realized by using a CNN convolutional neural network model, the input of the model is extracted dish pictures, and the output of the model is meal dishes corresponding to the dish pictures;
the dish quantity recognition submodule is used for being realized by utilizing a CNN convolution neural network model, the input of the model is each dish picture marked with a meal dish, the meal quantity corresponding to each meal dish is output, and each meal dish and the corresponding meal quantity are sent to the restaurant settlement system for transaction settlement;
the POS record extraction submodule is used for extracting t-t from a POS machine of the restaurant settlement system0>0, the first settlement transaction record is pushed to a record synthesis submodule after the dining record number is added;
the record synthesis submodule is used for summarizing the dining record number, the dining record time, each dining dish, the corresponding dining quantity of each dining dish and the settlement transaction record to form a dining record, and transmitting the dining record to the dining record corresponding to the dining record number in the user account.
5. The meal record system of claim 4, wherein the record synthesis submodule is configured to receive user corrections to the meal records via an interactive applet, and generate a final meal record;
the meal record creating module further comprises an online correction training submodule, wherein the online correction training submodule is used for comparing the output result of the meal volume identification submodule by using the final meal record from the record synthesis submodule, screening out the final meal record inconsistent with the output result to generate model difference data, and the model difference data is used for online correction of a CNN convolutional neural network model of the meal volume identification submodule so as to continuously optimize the identification quality of the model.
6. A method for recording meals, comprising the steps of:
s1, continuously collecting the faces at the entrance of the dining queue and extracting face images to be used as first face images, continuously collecting the faces at the exit of the dining queue and extracting face images to be used as second face images, matching the current second face images with all the first face images, sending a settlement starting signal and successfully matched face images to create a dining record when matching is successful, and sending the successfully matched face images to request identity recognition;
s2, comparing the received successfully matched face image with face images prestored in all user accounts to identify the identity of a dining user, creating a dining record in the corresponding user account when the identification is successful, creating a new user account when the identification is failed, and creating a dining record in the user account, wherein the dining record comprises a dining record number, a dining record time entry, a dining dish entry, a dining quantity entry corresponding to each dining dish and a settlement transaction record entry;
s3, continuously scanning a dinner plate placing area after receiving a settlement starting signal to extract dinner plate pictures, requesting meal record numbers corresponding to successfully matched face images, recording current meal record time, extracting each dish picture in the dinner plate pictures through an image processing technology, identifying meal dishes corresponding to each dish picture and meal quantity corresponding to each meal dish by utilizing a neural network algorithm, sending each meal dish and corresponding meal quantity to a restaurant settlement system for transaction settlement, summarizing the meal record numbers, the meal record time, each meal dish, the meal quantity corresponding to each meal dish and the settlement transaction records of the restaurant settlement system to form meal records, and transmitting the meal records to a user account under each corresponding item in the meal records corresponding to the meal record numbers.
7. The meal recording method according to claim 6, wherein the step S1 includes:
s11, continuously scanning the faces in the field of view by using a control camera positioned at the entrance of the dining queue, extracting face images, screening the face images with the strongest identifiability from the face images, and transmitting the face images serving as first face images to a target face temporary library;
s12, continuously scanning the faces in the field of view by using a settlement camera positioned at the exit of the dining queue, extracting face images, screening the face images with the strongest identifiability from the face images, and taking the face images as second face images;
s13, carrying out face matching on the current second face image and the first face image in the target face temporary library, sending a matching success signal to the target face temporary library when matching is successful, sending a settlement starting signal and the successfully matched face image to create a dining record, and sending the successfully matched face image to request identity recognition;
and S14, the target face temporary library receives the first face image transmitted by the control camera, deletes the successfully matched face image in the target face temporary library after receiving the successfully matched signal, and deletes the temporarily stored first face image after the temporary storage time of the first face image in the target face temporary library reaches the set time.
8. The meal recording method according to claim 6, wherein the step S2 includes:
comparing the received successfully matched face image with a face image prestored in a user account to identify the identity of a dining user, creating a dining record in the corresponding user account when the identification is successful, creating a new user account when the identification is failed, and creating a dining record in the user account;
creating, maintaining and managing a user account, wherein the user account comprises a user ID, a pre-stored face image and a dining record;
and managing the registration levels of the users, wherein the registration levels are divided into an unauthorized user level, an authorized user level, a registered user level, an authenticated user level and a paid user level.
9. The meal recording method according to claim 6, wherein the step S3 includes:
s31, continuously scanning a dinner plate placing area by using a dinner plate camera positioned at the exit of the dining queue to extract a dinner plate picture;
s32, setting the current meal recording time when the settlement starting signal is received as t0Requesting the meal record number corresponding to the successfully matched face image, and searching whether a time window t is present in the transmitted meal plate picture [ t ═ t [ [ t ]0-τ1,t0-τ2]The picture is pushed in, the found first dinner plate picture is pushed after being added with the dinner record number, and tau1、τ2Is a time window parameter;
s33, converting the first dinner plate picture into a gray image, executing a boundary detection algorithm on the gray image, taking out the boundary of the dinner plate containing dishes, forming a closed polygon on the extracted boundary by using a boundary smoothing method, and extracting the picture of each dish from the first dinner plate picture by using the polygon;
s34, dish identification is achieved through the CNN convolutional neural network model, the model is input into each extracted dish picture, and the model is output into a meal dish corresponding to each dish picture;
s35, dish quantity recognition is achieved by using a CNN convolutional neural network model, the input of the model is dish pictures marked with meal dishes, the output is meal quantity corresponding to each meal dish, and each meal dish and the corresponding meal quantity are sent to a restaurant settlement system for transaction settlement;
s36, extracting t-t from POS machine of restaurant settlement system0>A settlement transaction record for the first of 0;
s37, summarizing the dining record number, the dining record time, each dining item, the corresponding dining quantity of each dining item and the settlement transaction record to form a dining record, and transmitting the dining record to the dining record corresponding to the dining record number in the user account.
10. The meal recording method according to claim 9, wherein the step S3 further comprises: receiving the correction of a user on the dining record through an interactive applet, and generating a final dining record;
and comparing the final meal record with the dish quantity recognition output result, and screening out the final meal record inconsistent with the output result to generate model difference data, wherein the model difference data is used for online correction of the CNN convolutional neural network model for dish quantity recognition so as to continuously optimize the recognition quality of the model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110739335.5A CN113469044B (en) | 2021-06-30 | 2021-06-30 | Dining recording system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110739335.5A CN113469044B (en) | 2021-06-30 | 2021-06-30 | Dining recording system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113469044A CN113469044A (en) | 2021-10-01 |
| CN113469044B true CN113469044B (en) | 2022-07-01 |
Family
ID=77876670
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110739335.5A Active CN113469044B (en) | 2021-06-30 | 2021-06-30 | Dining recording system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113469044B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114092810B (en) * | 2021-11-17 | 2024-04-12 | 四川启睿克科技有限公司 | Method for identifying object position by camera based on self-help weighing meal taking system |
| CN114445925B (en) * | 2022-04-11 | 2022-07-22 | 深圳市润璟元信息科技有限公司 | Facial recognition intelligent attendance system capable of being automatically loaded and deleted |
| CN114743153B (en) * | 2022-06-10 | 2022-09-30 | 北京航空航天大学杭州创新研究院 | Non-sensory vegetable taking model establishing and vegetable taking method and device based on video understanding |
| CN117291759B (en) * | 2023-11-27 | 2024-02-27 | 广州蚁窝智能科技有限公司 | Interactive intelligent restaurant experience system based on 3D recognition technology |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109727406A (en) * | 2018-12-05 | 2019-05-07 | 四川壹脸通科技有限公司 | A kind of self-service dining platform intelligently settled accounts |
| CN112233307A (en) * | 2020-08-24 | 2021-01-15 | 云南乐进科技有限公司 | Canteen consumption supervision system based on face recognition |
| CN112699774A (en) * | 2020-12-28 | 2021-04-23 | 深延科技(北京)有限公司 | Method and device for recognizing emotion of person in video, computer equipment and medium |
| CN113010736A (en) * | 2019-12-20 | 2021-06-22 | 北京金山云网络技术有限公司 | Video classification method and device, electronic equipment and storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10586293B1 (en) * | 2016-12-22 | 2020-03-10 | Worldpay, Llc | Systems and methods for personalized dining and individualized ordering by associating electronic device with dining session |
| CN107705469B (en) * | 2017-09-29 | 2020-08-21 | 阿里巴巴集团控股有限公司 | Dining settlement method, intelligent ordering equipment and intelligent restaurant payment system |
| CN108171500A (en) * | 2018-01-08 | 2018-06-15 | 北京京东金融科技控股有限公司 | Unmanned shop method for processing payment information and system, computer-readable program medium |
| CN110473321A (en) * | 2019-08-05 | 2019-11-19 | 广州织点智能科技有限公司 | Unmanned convenience store enters recognition methods, device, electronic equipment and storage medium |
-
2021
- 2021-06-30 CN CN202110739335.5A patent/CN113469044B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109727406A (en) * | 2018-12-05 | 2019-05-07 | 四川壹脸通科技有限公司 | A kind of self-service dining platform intelligently settled accounts |
| CN113010736A (en) * | 2019-12-20 | 2021-06-22 | 北京金山云网络技术有限公司 | Video classification method and device, electronic equipment and storage medium |
| CN112233307A (en) * | 2020-08-24 | 2021-01-15 | 云南乐进科技有限公司 | Canteen consumption supervision system based on face recognition |
| CN112699774A (en) * | 2020-12-28 | 2021-04-23 | 深延科技(北京)有限公司 | Method and device for recognizing emotion of person in video, computer equipment and medium |
Non-Patent Citations (1)
| Title |
|---|
| 基于人脸识别的食堂管理系统设计与应用研究;祁志斌;《科技与信息》;20200630;第175-177页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113469044A (en) | 2021-10-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113469044B (en) | Dining recording system and method | |
| CN108419027B (en) | Intelligent photographing method and server | |
| US11394708B2 (en) | Account information obtaining method, terminal, server and system | |
| AU2007219702B2 (en) | Identity verification and access control | |
| WO2021068744A1 (en) | Insect identification method and system | |
| JP4636190B2 (en) | Face collation device, electronic device, face collation device control method, and face collation device control program | |
| CN109118621A (en) | The face registration system of recognition of face gate inhibition a kind of and application in access control | |
| CN107360119A (en) | A kind of cloud desktop Sign-On authentication method, cloud desktop control system and client | |
| EP3905188A1 (en) | Information processing device, information processing method, and program | |
| CN110543811A (en) | non-cooperation type examination person management method and system based on deep learning | |
| US20140344238A1 (en) | System And Method For Accessing Electronic Data Via An Image Search Engine | |
| CN114511915B (en) | Trusted certificate photo acquisition system and method based on mobile client | |
| CN113343003A (en) | Dining nutrition construction recording system and method | |
| KR100736342B1 (en) | Portable terminal apparatus and schedule administering method thereof | |
| CN109472230B (en) | Automatic athlete shooting recommendation system and method based on pedestrian detection and Internet | |
| US7286722B2 (en) | Memo image managing apparatus, memo image managing system and memo image managing method | |
| WO2005109310A1 (en) | Biometric identification system | |
| CN111640267B (en) | Self-service settlement method and device, storage medium and computer equipment | |
| CN102750519A (en) | Method and device for identifying and filtering images | |
| CN108960459A (en) | It registers login method and device | |
| US20230082150A1 (en) | Information processing method and apparatus, computer device, and storage medium | |
| CN116503922A (en) | Method, system and equipment for accurately positioning personnel residence | |
| US8923574B2 (en) | Method and apparatus for encouraging social networking through employment of facial feature comparison and matching | |
| JP2023026630A (en) | Information processing system, information processing apparatus, information processing method, and program | |
| CN107220680A (en) | Self-help card-distributing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |