CN112668558A

CN112668558A - Cash registering error correction method and device based on human-computer interaction

Info

Publication number: CN112668558A
Application number: CN202110271596.9A
Authority: CN
Inventors: 张飞云; 张鹏
Original assignee: Koubei Shanghai Information Technology Co Ltd; Zhejiang Koubei Network Technology Co Ltd
Current assignee: Koubei Shanghai Information Technology Co Ltd; Zhejiang Koubei Network Technology Co Ltd
Priority date: 2021-03-12
Filing date: 2021-03-12
Publication date: 2021-04-16

Abstract

The embodiment of the invention discloses a cash registering error correction method and device based on human-computer interaction. The method comprises the following steps: acquiring and presenting at least one first commodity identification result obtained by identifying the commodity image; at least one first commodity identification result is obtained by outputting a commodity identification model; and monitoring selection operation and correction operation executed by a user on a human-computer interaction interface aiming at any first commodity identification result, and acquiring user correction data, wherein the user correction data comprises a second commodity identification result, and the second commodity identification result is an error correction result of the selected first commodity identification result. By utilizing the embodiment of the invention, the user can correct and correct the error in time when finding out the commodity identification error, thereby improving the accuracy of commodity settlement.

Description

Cash registering error correction method and device based on human-computer interaction

Technical Field

The embodiment of the invention relates to the technical field of intelligent cash register, in particular to a cash register error correction method, a cash register error correction device, computing equipment and a computer storage medium based on human-computer interaction, and further relates to a training method, a training device, computing equipment and a computer storage medium of a commodity identification model.

Background

The intelligent store is an off-line store which can be bought by a consumer by self and settled by self. Compare in traditional off-line shop, wisdom shop has that the cost of labor is low, the settlement is efficient and characteristics such as management of being convenient for.

The intelligent store automatically identifies the commodities selected by the consumer through the cash register device, and then completes consumption settlement. In the prior art, a machine learning algorithm is usually adopted to train a commodity identification model, and the commodity identification model is used to identify a commodity selected by a consumer.

In the prior art, professional algorithm personnel are often required to regularly screen sample data from mass data, label the sample in a manual labeling mode, and then update the commodity identification model by using the labeled sample data. However, the commodity identification model cannot be updated in time by adopting the mode, so that the identification precision of the commodity identification model is influenced, and the commodity identification accuracy is low; in addition, the prior art relies on the manual screening and labeling of training samples by algorithm personnel, so that the defects of high labor cost and low model updating efficiency exist.

Disclosure of Invention

In view of the above, embodiments of the present invention are proposed in order to provide a method, an apparatus, a computing device and a computer storage medium for cashier correction based on human-computer interaction, which overcome or at least partially solve the above problems.

According to an aspect of the embodiment of the present invention, a cash register correction method based on human-computer interaction is provided, which includes:

acquiring at least one first commodity identification result obtained by identifying the shot commodity image, and presenting the at least one first commodity identification result in a human-computer interaction interface; wherein the at least one first commodity identification result is obtained by outputting a commodity identification model obtained by training;

and monitoring selection operation and correction operation executed by a user on the man-machine interaction interface aiming at any first commodity identification result, and acquiring user correction data, wherein the user correction data comprises a second commodity identification result, and the second commodity identification result is an error correction result of the selected first commodity identification result.

In an optional manner, the method further comprises: and feeding back the second commodity identification result to a server side so that the server side can update and train the commodity identification model according to the second commodity identification result.

In an optional manner, the feeding back the second product identification result to the server, so that the server performs update training on the product identification model according to the second product identification result specifically includes: and feeding back the second commodity identification result to a server, so that the server takes the second commodity identification result as a labeling result of the commodity image, and updating and training the commodity identification model.

In an optional mode, the human-computer interaction interface comprises response areas corresponding to the first commodity identification results, and the response areas corresponding to the first commodity identification results are not overlapped with each other;

the monitoring of the selection operation executed by the user on the human-computer interaction interface aiming at any first commodity identification result further comprises the following steps:

monitoring selection operation executed by a user on the human-computer interaction interface, and positioning a response area corresponding to the selection operation;

and determining a first commodity identification result selected by the user according to the response area corresponding to the selection operation.

In an optional manner, monitoring a correction operation performed by a user on the human-computer interaction interface for any first product identification result, and acquiring user correction data further includes:

presenting at least one candidate commodity identification result in a human-computer interaction interface;

and monitoring the correction operation of selecting any one alternative commodity identification result from the at least one alternative commodity identification result by the user, and acquiring the selected alternative commodity identification result as the second commodity identification result.

In an optional manner, the at least one candidate product recognition result is a product recognition result output by the product recognition model and having a lower confidence than the first product recognition result.

displaying a commodity information input text box in a human-computer interaction interface;

and monitoring the correction operation of inputting the commodity information by the user in the commodity information input frame, and acquiring the input commodity information as a second commodity identification result.

and monitoring the correction operation of marking the first commodity identification result as the non-settlement commodity by the user, and acquiring a second commodity identification result marked as the non-settlement commodity.

In an optional manner, the presenting the at least one first item identification result in the human-computer interaction interface further includes:

presenting the at least one first commodity identification result and the confidence of the at least one first commodity identification result in a human-computer interaction interface.

According to another aspect of the embodiments of the present invention, there is provided a cash register correction device based on human-computer interaction, including:

the first acquisition module is used for acquiring at least one first commodity identification result obtained by identifying the shot commodity image;

the presentation module is used for presenting the at least one first commodity identification result in a human-computer interaction interface; wherein the at least one first commodity identification result is obtained by outputting a commodity identification model obtained by training;

the second acquisition module is used for monitoring selection operation and correction operation executed by a user on the human-computer interaction interface aiming at any first commodity identification result, and acquiring user correction data, wherein the user correction data comprises a second commodity identification result, and the second commodity identification result is an error correction result of the selected first commodity identification result.

In an optional manner, the apparatus further comprises: and the feedback module is used for feeding the second commodity identification result back to the server side so that the server side can update and train the commodity identification model according to the second commodity identification result.

In an optional manner, the feedback module is further configured to:

and feeding back the second commodity identification result to a server, so that the server takes the second commodity identification result as a labeling result of the commodity image, and updating and training the commodity identification model.

the second obtaining module is further configured to: monitoring selection operation executed by a user on the human-computer interaction interface, and positioning a response area corresponding to the selection operation; and determining a first commodity identification result selected by the user according to the response area corresponding to the selection operation.

In an optional manner, the second obtaining module is further configured to:

In an optional manner, the presentation module is further configured to:

According to yet another aspect of an embodiment of the present invention, there is provided a computing device including: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;

the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the cash register correction method based on human-computer interaction.

According to a further aspect of the embodiments of the present invention, there is provided a computer storage medium, where at least one executable instruction is stored, and the executable instruction causes a processor to execute an operation corresponding to the above-mentioned cash register correction method based on human-computer interaction.

According to the cash registering error correction method and device based on human-computer interaction, the first commodity identification result output by the commodity identification model is presented through the human-computer interaction interface, and selection and correction operation aiming at the first commodity identification result are further provided for a user through the human-computer interaction interface, so that the user can correct and correct errors in time when finding out that the commodity identification is wrong, and the commodity settlement accuracy is improved; in addition, the cash register error correction is completed through the human-computer interaction interface, the operation mode is convenient and fast, and the influence on the settlement experience of the user is avoided. Furthermore, the error correction result of the first commodity identification result can be fed back to the server, so that the server updates and trains the commodity identification model according to the error correction result of the first commodity identification result, the identification precision of the commodity identification model is improved, the labor cost required by updating the commodity identification model is reduced, and the updating efficiency of the commodity identification model can be improved.

In view of the above, embodiments of the present invention are also proposed in order to provide a training method, an apparatus, a computing device and a computer storage medium for a commodity recognition model that overcome or at least partially solve the above problems.

According to an aspect of an embodiment of the present invention, there is provided a training method for a commodity recognition model, including:

inputting the shot commodity image into a commodity identification model obtained by pre-training to obtain a first commodity identification result;

receiving a second commodity identification result fed back by the client; wherein the second commodity identification result is an error correction result of the first commodity identification result;

and updating and training the commodity identification model according to the second commodity identification result.

In an optional manner, the updating and training of the commodity recognition model according to the second commodity recognition result specifically includes:

and taking the second commodity identification result as a labeling result of the commodity image, and performing updating training on the commodity identification model.

In an optional manner, the inputting the captured commodity image into a commodity recognition model obtained through pre-training to obtain a first commodity recognition result specifically includes:

inputting the shot commodity image into a commodity identification model obtained by pre-training to obtain a first commodity identification result and at least one alternative commodity identification result;

the method further comprises the following steps: and sending the at least one candidate commodity identification result to the client so that the client presents the at least one candidate commodity identification result in a human-computer interaction interface, monitoring a correction operation of selecting any one candidate commodity identification result from the at least one candidate commodity identification result by the user, and acquiring the selected candidate commodity identification result as the second commodity identification result.

In an optional manner, the at least one candidate product identification result is a product identification result with a lower confidence than the first product identification result.

In an optional manner, the performing update training on the product recognition model according to the second product recognition result further includes:

and if the second commodity identification result is an alternative commodity identification result or commodity information input by a user, taking the second commodity identification result as a positive sample labeling result of the commodity image, and performing updating training on the commodity identification model.

and if the second commodity identification result is marked as a non-settlement commodity, taking the second commodity identification result as a negative sample marking result of the commodity image, and performing updating training on the commodity identification model.

According to another aspect of the embodiments of the present invention, there is provided a training apparatus for a commodity recognition model, including:

the acquisition module is used for inputting the shot commodity image into a commodity recognition model obtained by pre-training to obtain a first commodity recognition result;

the receiving module is used for receiving a second commodity identification result fed back by the client; wherein the second commodity identification result is an error correction result of the first commodity identification result;

and the updating module is used for updating and training the commodity identification model according to the second commodity identification result.

In an optional manner, the update module is further configured to: and taking the second commodity identification result as a labeling result of the commodity image, and performing updating training on the commodity identification model.

In an optional manner, the obtaining module is further configured to:

the device further comprises: and the sending module is used for sending the at least one alternative commodity identification result to the client so that the client can present the at least one alternative commodity identification result in a human-computer interaction interface, monitoring the correction operation of selecting any alternative commodity identification result from the at least one alternative commodity identification result by the user, and acquiring the selected alternative commodity identification result as the second commodity identification result.

In an optional manner, the update module is further configured to:

the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the training method of the commodity identification model.

According to a further aspect of the embodiments of the present invention, there is provided a computer storage medium, in which at least one executable instruction is stored, and the executable instruction causes a processor to perform operations corresponding to the training method for a commodity recognition model.

According to the training method, the training device, the computing equipment and the computer storage medium of the commodity identification model provided by the embodiment of the invention, the shot commodity image is input into the commodity identification model obtained by pre-training to obtain a first commodity identification result; further receiving a second commodity identification result fed back by the client; and finally, updating and training the commodity identification model according to the second commodity identification result. In the embodiment of the invention, the commodity identification model is updated and trained by the error correction result fed back by the client aiming at the first commodity identification result, and the algorithm personnel are not required to manually screen and label the sample data, so that the labor cost is reduced and the model updating efficiency is improved.

The foregoing description is only an overview of the technical solutions of the embodiments of the present invention, and the embodiments of the present invention can be implemented according to the content of the description in order to make the technical means of the embodiments of the present invention more clearly understood, and the detailed description of the embodiments of the present invention is provided below in order to make the foregoing and other objects, features, and advantages of the embodiments of the present invention more clearly understandable.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the embodiments of the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:

FIG. 1 is a flowchart illustrating a method for cash register correction based on human-computer interaction according to an embodiment of the present invention;

FIG. 2 is a flowchart illustrating a cash register correction method based on human-computer interaction according to another embodiment of the present invention;

FIG. 3 is a schematic diagram of a human-computer interaction interface provided by an embodiment of the invention;

FIG. 4 is a schematic diagram of another human-computer interaction interface provided by the embodiment of the invention;

FIG. 5 is a flowchart illustrating a method for cash correction based on human-computer interaction according to another embodiment of the present invention;

FIG. 6 is a flowchart illustrating a method for cash correction based on human-computer interaction according to another embodiment of the present invention;

FIG. 7 is a flow diagram illustrating a method for training a merchandise recognition model according to one embodiment of the invention;

fig. 8 is a schematic structural diagram of a cash register correction device based on human-computer interaction according to an embodiment of the present invention;

FIG. 9 is a schematic structural diagram of a training apparatus for a commodity recognition model according to an embodiment of the present invention;

FIG. 10 illustrates a schematic structural diagram of a computing device provided in accordance with an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

Fig. 1 shows a flowchart of a cash register correction method based on human-computer interaction according to an embodiment of the present invention. The cash registering error correction method based on human-computer interaction provided by the embodiment can be applied to cash registering clients of intelligent stores and the like.

As shown in fig. 1, the method comprises the steps of:

step 110: acquiring at least one first commodity identification result obtained by identifying the shot commodity image, and presenting the at least one first commodity identification result in a human-computer interaction interface; and outputting at least one first commodity identification result by the trained commodity identification model.

When a user settles commodities in an intelligent store, at least one commodity to be settled can be placed in a commodity image acquisition area corresponding to a client, and the client shoots through image acquisition equipment to obtain a commodity image of the at least one commodity to be settled.

The captured commodity image is further input to the trained commodity recognition model. And after the commodity identification model identifies the commodity image, outputting at least one first commodity identification result aiming at the commodity image. The client side obtains the at least one first commodity identification result and presents the at least one first commodity identification result in the man-machine interaction interface.

Step 120: and monitoring selection operation and correction operation executed by a user on a human-computer interaction interface aiming at any first commodity identification result, and acquiring user correction data, wherein the user correction data comprises a second commodity identification result, and the second commodity identification result is an error correction result of the selected first commodity identification result.

According to at least one first commodity identification result displayed on the man-machine interaction interface, a user can intuitively determine whether each first commodity identification result is accurate, so that corresponding interaction operation can be conveniently and quickly executed in the man-machine interaction interface. Specifically, if the user finds that the at least one first commodity identification result is accurate, the user can perform next interactive operation such as payment or confirmation on the human-computer interaction interface; if the user finds that any one or more first commodity identification results are inaccurate, the user can execute a series of interactive operations such as selection operation, correction operation and the like on the inaccurate first commodity identification results in the man-machine interaction interface.

The client can timely acquire user correction data by monitoring selection operation and correction operation executed by a user on the man-machine interaction interface.

Optionally, the human-computer interaction interface may include response areas (also referred to as hot areas) corresponding to the first product identification results, and the response areas corresponding to the first product identification results are not overlapped with each other. For example, a circumscribed rectangle frame of the first product identification result is selected as a response area, and if there is an overlapping area between any two circumscribed rectangles frames of the first product identification result, the response areas of the two first product identification results are appropriately narrowed.

Monitoring selection operation executed by a user on a human-computer interaction interface, and positioning a response area corresponding to the selection operation; and determining a first commodity identification result selected by the user according to the response area corresponding to the selection operation. If the user finds that the inaccurate first commodity identification result exists, the selection operation can be executed in the response area of the first commodity identification result, and the client locates the response area corresponding to the selection operation by capturing the operation position of the selection operation, so as to determine which first commodity identification result is selected by the user.

Then, the correction operation executed by the user on the man-machine interaction interface is monitored, and user correction data is obtained, wherein the user correction data comprises an error correction result of the first commodity identification result, namely the user correction data comprises a second commodity identification result.

Step 130: and feeding back the second commodity identification result to the server side so that the server side can update and train the commodity identification model according to the second commodity identification result.

The client feeds back the obtained second commodity identification result to the server in time, and the server updates and trains the commodity identification model by using the second commodity identification result, so that the subsequent identification precision of the commodity identification model is improved. Specifically, the second commodity identification result can be fed back to the server, so that the server takes the second commodity identification result as a labeling result of the commodity image to update and train the commodity identification model.

By adopting the cash registering error correction method based on human-computer interaction provided by the embodiment of the invention, the first commodity identification result output by the commodity identification model is presented in the human-computer interaction interface of the cash registering client, so that a user can visually know the identification result of the commodity to be settled; in addition, in the embodiment of the invention, when the user determines that the first commodity identification result is inaccurate, the user can execute selection operation and correction operation in the human-computer interaction interface, so that on one hand, the error correction result of the user aiming at the first commodity identification result can be obtained, and on the other hand, a feedback channel is provided for the user, which is beneficial to improving the user experience; moreover, the commodity identification model is updated and trained by the user according to the error correction result of the first commodity identification result, and algorithm personnel are not required to manually screen and label the sample data, so that the labor cost required by model updating is reduced, and the model updating efficiency is improved; in addition, the error correction result of the user aiming at the first commodity identification result can be timely fed back to the server, so that the server can timely update and train the model by using the error correction result, and the identification precision of the subsequent commodity identification model is guaranteed.

Fig. 2 is a flowchart illustrating a cash register correction method based on human-computer interaction according to another embodiment of the present invention.

As shown in fig. 2, the method comprises the steps of:

step 210: acquiring at least one first commodity identification result obtained by identifying the shot commodity image, and presenting the at least one first commodity identification result in a human-computer interaction interface; and outputting at least one first commodity identification result by the trained commodity identification model.

After the cash register client transmits the shot commodity image to the server, the server inputs the commodity image into the trained commodity identification model, and then the commodity identification model outputs an identification result aiming at the commodity image.

In an actual implementation process, for any one product in the product image, the product identification model usually predicts a plurality of product identification results for the product, and each product identification result has a corresponding confidence. The higher the confidence of the commodity identification result is, the higher the probability that the commodity identification result is close to the real commodity is. Therefore, the commodity identification result with the highest confidence coefficient output by the commodity identification model can be used as the first commodity identification result, and the first commodity identification result is fed back to the client. If the commodity image contains a plurality of commodities which need to be settled together, the commodity identification model respectively predicts the commodity identification results of the commodities, and the first commodity identification result with the highest confidence coefficient of each commodity is selected and fed back to the client.

And the client presents the received first commodity identification result in a human-computer interaction interface. Optionally, in order to facilitate a user to intuitively know the recognition accuracy of the first product recognition result, in addition to the first product recognition result, the human-computer interaction interface in the embodiment of the present invention further presents a confidence level of the first product recognition result. As shown in fig. 3, the first article recognition result "article a" and the confidence "0.836" of the first article recognition result are presented in the human-machine interaction interface.

Step 220: monitoring selection operation executed by a user on a human-computer interaction interface, and positioning a response area corresponding to the selection operation; and determining a first commodity identification result selected by the user according to the response area corresponding to the selection operation.

In a self-service settlement scenario without an administrator, the user in this embodiment is specifically a consumer user. Referring to fig. 3, if the consumer user does not disagree with the first product identification result, the payment request may be triggered by clicking the "payment" control; if the consumer user determines that the first commodity identification result has errors, the placing position and the angle of the commodity can be adjusted, and a new identification request is triggered by clicking the re-identification control. If the first product identification result re-identified still has errors, the consumer user may perform a selection operation on the human-computer interaction interface, for example, click or double click on a hot area corresponding to the first product identification result, and the cashier client determines the first product identification result selected by the user in response to the selection operation of the user.

In a settlement scenario with an administrator, in order to facilitate management of the checkout settlement, the checkout client in the smart store may correspond to a plurality of display devices including a primary display device facing the consumer user and a secondary display device facing the administrator user. And a man-machine interaction interface is displayed in each display device, and the display content in the main display device can be synchronized to the auxiliary display device. When the consumer user determines that the first commodity identification result presented by the human-computer interaction interface in the main display device is wrong, the administrator can be notified. Because the content of the main display device is synchronously displayed in the auxiliary display device, an administrator can obtain the first commodity identification result according to the display content in the auxiliary display device. When the administrator determines that the first commodity identification result really has an error, the administrator user can trigger selection operation and correction operation on the first commodity identification result in the man-machine interaction interface.

As another alternative, to save hardware resources, the cashier client corresponds to only one display device. When the consumer user determines that the first commodity identification result presented by the human-computer interaction interface is wrong, the administrator can be notified. The administrator user can trigger an administrator identity authentication request through an administrator login entrance in a human-computer interaction interface presented by the display device, and after the administrator identity authentication is passed, the administrator user can trigger selection operation and correction operation on the first commodity identification result in the human-computer interaction interface.

Step 230: and responding to the selection operation, and presenting at least one candidate commodity identification result in the human-computer interaction interface.

And responding to the selection operation, and acquiring at least one candidate commodity identification result. The at least one candidate commodity identification result is a commodity identification result, wherein the confidence degree obtained by the commodity identification model prediction is lower than that of the first commodity identification result. And further displaying the at least one candidate commodity identification result in a human-computer interaction interface.

For example, the user, after double-clicking the "article a" hot zone in fig. 3, presents the human-machine interface shown in fig. 4. As shown in fig. 4, the human-computer interaction interface presents two candidate product recognition results, "product B" and "product C", and also presents a confidence "0.732" of "product B" and a confidence "0.543" of "product C". As can be seen from this, the confidence degrees of both the candidate product recognition result "product B" and the candidate product recognition result "product C" are lower than the confidence degree "0.836" of the first product recognition result "product a".

Step 240: and monitoring the correction operation of selecting any one candidate commodity identification result from at least one candidate commodity identification result by the user, and acquiring the selected candidate commodity identification result as a second commodity identification result.

The user can select a correct commodity identification result from the presented candidate commodity identification results as a second commodity identification result.

Optionally, if the user cannot find the correct product identification result in the at least one candidate product identification result, the user may call out the text box through a text box call-out operation, input the correct product identification result in the text box, and use the input product identification result as the second product identification result.

Step 250: and feeding back the second commodity identification result to the server side so that the server side can update and train the commodity identification model according to the second commodity identification result.

By adopting the cash registering and error correcting method based on human-computer interaction provided by the embodiment of the invention, the selection operation and the correction operation triggered by the user on the first commodity identification result in the human-computer interaction interface are monitored, at least one alternative commodity identification result is obtained and presented, and the alternative commodity identification result selected by the user from the at least one alternative commodity identification result is received and taken as the second commodity identification result. According to the embodiment of the invention, after the user triggers the selection operation, the candidate commodity identification result is presented, and the user selects the correct commodity identification result from the candidate commodity identification result, so that the acquisition efficiency of the second commodity identification result is improved, the commodity identification result does not need to be manually input by the user, and the improvement of user experience is facilitated; in addition, the commodity identification model has higher similarity between a plurality of commodity identification results output aiming at the same commodity image, so that the similarity between the first commodity identification result and the alternative commodity identification result is higher, and the embodiment of the invention is particularly suitable for scenes that the commodities to be settled have more similar commodities.

Fig. 5 shows a flowchart of a cash register correction method based on human-computer interaction according to another embodiment of the present invention.

As shown in fig. 5, the method comprises the steps of:

step 510: acquiring at least one first commodity identification result obtained by identifying the shot commodity image, and presenting the at least one first commodity identification result in a human-computer interaction interface; and outputting at least one first commodity identification result by the trained commodity identification model.

Step 520: monitoring selection operation executed by a user on a human-computer interaction interface, and positioning a response area corresponding to the selection operation; and determining a first commodity identification result selected by the user according to the response area corresponding to the selection operation.

The specific implementation process of

steps

510 and 520 may refer to the description in

steps

210 and 220, and this step is not described herein again.

Step 530: and responding to the selection operation, and presenting a commodity information input text box in the human-computer interaction interface.

In the prior art, if a smart store releases a new product, a store manager is often required to shoot a new product image specially, new product information is input through a new product input inlet, the shot new product image and the input new product information are uploaded to a server through a new product uploading inlet, and the server updates a model by using the new product image and the new product information. However, the method needs the manager to additionally shoot the new image, so that the operation is complicated, the efficiency is low, and the system storage resource is easily wasted; in addition, the adoption of the mode needs to additionally configure a new product entry and a new product uploading entry for store managers, further increases the logic complexity of the system, and is easy to cause misoperation.

In the embodiment of the invention, the new product entry can be realized by utilizing the interactive operation of the user. Specifically, if the commodity to be settled is a new commodity, the first commodity identification result has an error, so that the user triggers the selection operation, the embodiment of the invention responds to the selection operation to present a commodity information input text box, and the new commodity information is input through the commodity information input text box.

Optionally, the user triggers a selection operation, may present at least one candidate product recognition result first, and may call out the product information and input the text box through a text box call-out operation if the user cannot find a correct product recognition result in the at least one candidate product recognition result.

Step 540: and monitoring the correction operation of inputting the commodity information by the user in the commodity information input box, and acquiring the input commodity information as a second commodity identification result.

The embodiment of the present invention does not limit the manner in which the commodity information is input in the commodity information input box. For example, a user may input commodity information in a commodity information input box through a virtual or physical keyboard; the commodity information can also be input in a voice input mode. Optionally, in the process of inputting the commodity information in a voice input manner, the voice information may be converted into text characters and then displayed in the commodity information input box, so that the user can check whether the input commodity information is accurate.

Step 550: and feeding back the second commodity identification result to the server side so that the server side can update and train the commodity identification model according to the second commodity identification result.

By adopting the cash registering and error correcting method based on human-computer interaction provided by the embodiment of the invention, the selection operation triggered by the user on the first commodity identification result in the human-computer interaction interface is monitored, the commodity information input text box is presented in response to the selection operation, and the commodity information input by the user in the commodity information input box is received and used as the second commodity identification result. In the scene of releasing the new product in the intelligent store, the embodiment of the invention can realize the update of the commodity identification model without additionally shooting the new product image by store management personnel, thereby simplifying the operation flow, improving the update efficiency of the new product, saving system resources and simplifying system logic.

Fig. 6 is a flowchart illustrating a cash register correction method based on human-computer interaction according to still another embodiment of the present invention.

As shown in fig. 6, the method comprises the steps of:

step 610: acquiring at least one first commodity identification result obtained by identifying the shot commodity image, and presenting the at least one first commodity identification result in a human-computer interaction interface; and outputting at least one first commodity identification result by the trained commodity identification model.

Step 620: monitoring selection operation executed by a user on a human-computer interaction interface, and positioning a response area corresponding to the selection operation; and determining a first commodity identification result selected by the user according to the response area corresponding to the selection operation.

The specific implementation of

steps

610 and 620 may refer to the descriptions in

steps

210 and 220, and this step is not described herein again.

Step 630: and monitoring the correction operation of marking the first commodity identification result as the non-settlement commodity by the user, and acquiring a second commodity identification result marked as the non-settlement commodity.

If the at least one first commodity identification result comprises non-settlement commodities, the user can trigger a correction operation for marking the non-settlement commodities in the human-computer interaction interface. For example, the commodity selected by the consumer user is commodity a, and the commodity image includes the commodity a and the mobile phone of the consumer user. The first product identification result includes "product a" and "mobile phone P", where "mobile phone P" is a non-settlement product. The user can trigger the selection operation and the correction operation to correct the first commodity identification result 'mobile phone P' into a non-settlement commodity.

In an alternative embodiment, the selecting operation in step 620 may be a long press operation in a hot area corresponding to the first product identification result, after the long press operation of the user is monitored, an option of "add to black list" is popped up, the user clicks the option (triggers a correction operation), the corresponding first product identification result is added to the background black list, and the first product identification result added to the black list is marked as a non-settlement product.

Step 640: and feeding back the second commodity identification result to the server side so that the server side can update and train the commodity identification model according to the second commodity identification result.

Continuing with the above example, if the first product recognition result "cell phone P" is added to the blacklist, the server performs update training on the product recognition model according to the second product recognition result, and the "cell phone P" is no longer used as an output result when the updated and trained product recognition model recognizes a subsequent product image.

By adopting the cash registering and error correcting method based on human-computer interaction provided by the embodiment of the invention, the selection operation and the correction operation triggered by the user on the first commodity identification result in the human-computer interaction interface are monitored, and the second commodity identification result marked as a non-settlement commodity is obtained in response to the corresponding operation. According to the embodiment of the invention, the user can add the non-settlement commodity into the blacklist and update and train the commodity identification model by using the second commodity identification result marked as the non-settlement commodity, so that the phenomenon that the commodity identification model outputs the non-settlement commodity identification result is reduced.

FIG. 7 is a flowchart illustrating a training method of a product recognition model according to an embodiment of the present invention. The training method of the product identification model provided by the embodiment of the present invention may be applied to a server corresponding to an intelligent store, and may also be applied to a client of the intelligent store, which is not limited in the present invention. In the following embodiments, the application to the server is taken as an example for explanation.

As shown in fig. 7, the method comprises the steps of:

step 710: and inputting the shot commodity image into a commodity recognition model obtained by training in advance to obtain a first commodity recognition result.

The cashier client sends the commodity image to the server side after the commodity image is shot by the image acquisition device. The server side comprises a commodity identification model obtained through pre-training. The commodity identification model is constructed based on a machine learning algorithm, and the embodiment of the invention does not limit the specific structure of the commodity identification model. The server inputs the commodity image into a commodity identification model obtained through pre-training, and the commodity identification model identifies the commodity image and then outputs a first commodity identification result aiming at the commodity image.

As an alternative, for the intelligent stores such as restaurants, the commodities provided by different intelligent stores are greatly different. If the uniform commodity identification model is adopted to identify the commodity images in different intelligent stores under the category, the defect of poor commodity image identification precision is easy to occur. Thus, in order to improve the recognition accuracy of the product image, a corresponding exclusive product recognition model can be allocated for each of the intelligent stores or the chain-type intelligent stores. The exclusive commodity identification model only utilizes commodity data in the corresponding intelligent stores to carry out model training, and after the model training is finished, the commodity identification model obtained by the training is utilized to identify commodity images uploaded by the corresponding intelligent stores. Therefore, in the step, the client side can send the shot commodity image to the server side, and the server side inputs the commodity image into the exclusive commodity identification model corresponding to the intelligent store and obtains a first commodity identification result output by the exclusive commodity identification model.

As an optional implementation manner, for intelligent stores of the categories such as convenience stores and supermarkets, the difference of the commodities provided by different intelligent stores is small, so that in order to simplify the processing logic and save system resources, a unified commodity identification model can be adopted to identify the commodity images in different intelligent stores of the category. Therefore, in the step, the client side can send the shot commodity image to the server side, and the server side inputs the commodity image into the unified commodity identification model corresponding to the intelligent store and obtains a first commodity identification result output by the unified commodity identification model.

Step 720: receiving a second commodity identification result fed back by the client; and the second commodity identification result is an error correction result of the first commodity identification result.

After the server side sends the first commodity identification result to the client side, the client side displays the first commodity identification result in a human-computer interaction interface of the client side, and obtains user correction data by monitoring interaction operation executed by a user on the human-computer interaction interface, wherein the user correction data comprises a second commodity identification result, and the second commodity identification result is an error correction result of the first commodity identification result. The client further feeds back the second commodity identification result to the server.

Step 730: and updating and training the commodity identification model according to the second commodity identification result.

And the server updates and trains the commodity identification model according to the second commodity identification result fed back by the client so as to improve the identification precision of the subsequent commodity identification model.

By adopting the training method of the commodity identification model provided by the embodiment of the invention, the shot commodity image is input into the commodity identification model obtained by pre-training to obtain a first commodity identification result. Further receiving a second commodity identification result fed back by the client; and finally, updating and training the commodity identification model according to the second commodity identification result. In the embodiment of the invention, the commodity identification model is updated and trained by the error correction result fed back by the client aiming at the first commodity identification result, and the algorithm personnel are not required to manually screen and label the sample data, so that the labor cost is reduced and the model updating efficiency is improved.

As an optional implementation manner, after the commodity image is input into the commodity recognition model obtained by pre-training, for any commodity in the commodity image, the commodity recognition model outputs a plurality of commodity recognition results for the commodity, and each commodity recognition result has a corresponding confidence. The recognition result with the highest confidence coefficient output by the commodity recognition model can be used as the first commodity recognition result of the commodity, and other commodity recognition results with confidence coefficients lower than that of the first commodity recognition result can be used as the candidate commodity recognition results of the commodity. That is, the first product recognition result and the at least one candidate product recognition result for each product can be obtained by inputting the captured product image into the product recognition model trained in advance. The server side can firstly send the first commodity identification result of each commodity to the client side, and the client side can respond to the selection operation of the user for the specific commodity and send at least one alternative commodity obtaining request of the specific commodity to the server side. The server further sends the at least one alternative commodity identification result of the specific commodity to the client, so that the client presents the at least one alternative commodity identification result in the man-machine interaction interface, the correction operation that the user selects any alternative commodity identification result from the at least one alternative commodity identification result is monitored, and the selected alternative commodity identification result is obtained and serves as a second commodity identification result of the specific commodity. Optionally, the server may also send the first product identification result of each product and at least one candidate product identification result to the client, and the client responds to a further selection operation of the user for the specific product, and presents the at least one candidate product identification result of the specific product in the human-computer interaction interface, so that the user performs a correction operation to select the candidate product identification result from the at least one candidate product identification result, which is used as the second product identification result of the specific product.

As another optional implementation, after the server sends the first commodity identification result to the client, the client presents a commodity information input text box in response to a selection operation of the user, and the client receives the commodity information input by the user in the commodity information input box and takes the input commodity information as a second commodity identification result.

As another alternative, if the first product identification result sent by the server to the client includes a non-settlement product, the user may label the non-settlement product at the client, and the client may feed back the second product identification result labeled as the non-settlement product to the server.

Further, in step 730, the second product recognition result is used as a labeling result of the product image, and the product recognition model is updated and trained. In the embodiment of the invention, in the process of updating and training the commodity identification model by using the second commodity identification result, the second commodity identification result is specifically used as the labeling result of the commodity image, and then the commodity identification model is updated and trained.

Specifically, the labeling result of the commodity image is determined according to the type of the second commodity identification result. If the second commodity identification result is an alternative commodity identification result or commodity information input by a user, taking the second commodity identification result as a positive sample labeling result of the commodity image, and performing updating training on the commodity identification model; and if the second commodity identification result is marked as a non-settlement commodity, taking the second commodity identification result as a negative sample marking result of the commodity image, and performing updating training on the commodity identification model.

By adopting the training method of the commodity identification model provided by the embodiment of the invention, the second commodity identification result is taken as the labeling result of the commodity image, and the commodity identification model is updated and trained, so that the labeling efficiency is improved, and the labor cost is reduced; determining the labeling result of the commodity image according to the type of the second commodity identification result, and if the second commodity identification result is the candidate commodity identification result or the commodity information input by the user, taking the second commodity identification result as the positive sample labeling result of the commodity image; and if the second commodity identification result is marked as a non-settlement commodity, taking the second commodity identification result as a negative sample marking result of the commodity image, so that the marking precision is further improved, and the identification precision of the identification model is improved.

Fig. 8 is a schematic structural diagram of a cash register correction device based on human-computer interaction according to an embodiment of the present invention. As shown in fig. 8, the cashier correction device 800 includes: a first obtaining module 810, a presenting module 820, a second obtaining module 830, and a feedback module 840.

A first obtaining module 810, configured to obtain at least one first product identification result obtained by identifying a captured product image;

a presentation module 820, configured to present the at least one first product identification result in a human-computer interaction interface; wherein the at least one first commodity identification result is obtained by outputting a commodity identification model obtained by training;

a second obtaining module 830, configured to monitor a selection operation and a correction operation performed by a user on the human-computer interaction interface for any first commodity identification result, and obtain user correction data, where the user correction data includes a second commodity identification result, and the second commodity identification result is an error correction result of the selected first commodity identification result;

the feedback module 840 is configured to feed the second commodity identification result back to the server, so that the server updates and trains the commodity identification model according to the second commodity identification result.

In an optional manner, the feedback module 840 is further configured to:

In an optional mode, the human-computer interaction interface comprises response areas corresponding to the first commodity identification results, and the response areas corresponding to the first commodity identification results are not overlapped with each other; the second obtaining module 830 is further configured to: monitoring selection operation executed by a user on the human-computer interaction interface, and positioning a response area corresponding to the selection operation; and determining a first commodity identification result selected by the user according to the response area corresponding to the selection operation.

In an optional manner, the second obtaining module 830 is further configured to:

In an optional manner, the presentation module 820 is further configured to:

By adopting the cash register error correction device based on human-computer interaction provided by the embodiment of the invention, the first commodity identification result output by the commodity identification model is presented in the human-computer interaction interface of the cash register client, so that a user can visually know the identification result of the commodity to be settled; in addition, in the embodiment of the invention, when the user determines that the first commodity identification result is inaccurate, the user can perform interactive operations such as selection, correction and the like in the human-computer interaction interface, so that on one hand, the error correction result of the user aiming at the first commodity identification result can be obtained, on the other hand, a feedback channel is provided for the user, and the user experience is favorably improved; moreover, the commodity identification model is updated and trained by the user according to the error correction result of the first commodity identification result, and algorithm personnel are not required to manually screen and label sample data, so that the labor cost is reduced, and the model updating efficiency is improved; in addition, the error correction result of the user aiming at the first commodity identification result can be timely fed back to the server, so that the model can be updated and trained in time by using the error correction result, and the identification precision of the subsequent commodity identification model is guaranteed.

Fig. 9 is a schematic structural diagram illustrating a training apparatus for a product recognition model according to an embodiment of the present invention. As shown in fig. 9, the training device 900 for a product recognition model includes: an obtaining module 910, a receiving module 920, and an updating module 930.

The acquisition module 910 is configured to input a captured commodity image into a commodity identification model obtained through pre-training to obtain a first commodity identification result;

the receiving module 920 is configured to receive a second product identification result fed back by the client; wherein the second commodity identification result is an error correction result of the first commodity identification result;

an updating module 930, configured to perform update training on the commodity identification model according to the second commodity identification result.

In an optional manner, the update module 930 is further configured to: and taking the second commodity identification result as a labeling result of the commodity image, and performing updating training on the commodity identification model.

In an optional manner, the obtaining module 910 is further configured to:

the apparatus 900 further comprises: a sending module (not shown in the figure), configured to send the at least one candidate commodity identification result to the client, so that the client presents the at least one candidate commodity identification result in a human-computer interaction interface, monitors a correction operation of a user selecting any one of the candidate commodity identification results from the at least one candidate commodity identification result, and obtains the selected candidate commodity identification result as the second commodity identification result.

In an optional manner, the update module 930 is further configured to:

By adopting the training device of the commodity identification model provided by the embodiment of the invention, the shot commodity image is input into the commodity identification model obtained by pre-training, and the first commodity identification result is obtained. Further receiving a second commodity identification result fed back by the client; and finally, updating and training the commodity identification model according to the second commodity identification result. In the embodiment of the invention, the commodity identification model is updated and trained by the error correction result fed back by the client aiming at the first commodity identification result, and the algorithm personnel are not required to manually screen and label the sample data, so that the labor cost is reduced and the model updating efficiency is improved.

The embodiment of the invention also provides a nonvolatile computer storage medium, wherein the computer storage medium stores at least one executable instruction, and the executable instruction can execute the cash register error correction method based on human-computer interaction in any method embodiment.

The embodiment of the invention also provides a nonvolatile computer storage medium, wherein the computer storage medium stores at least one executable instruction, and the executable instruction can execute the training method of the commodity identification model in any method embodiment.

FIG. 10 illustrates a schematic structural diagram of a computing device provided in accordance with an embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computing device.

As shown in fig. 10, the computing device may include: a processor (processor)1002, a Communications Interface 1004, a memory 1006, and a Communications bus 1008.

Wherein:

the processor 1002, communication interface 1004, and memory 1006 communicate with each other via a communication bus 1008.

A communication interface 1004 for communicating with network elements of other devices, such as clients or other servers.

The processor 1002 is configured to execute the program 1010, and may specifically execute relevant steps in the foregoing cash register correction method based on human-computer interaction.

In particular, the program 1010 may include program code that includes computer operating instructions.

The processor 1002 may be a central processing unit CPU, or an application Specific Integrated circuit asic, or one or more Integrated circuits configured to implement an embodiment of the present invention. The computing device includes one or more processors, which may be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.

The memory 1006 is used for storing the program 1010. The memory 1006 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.

The program 1010 may be specifically configured to enable the processor 1002 to execute a cash register correction method based on human-computer interaction in any of the method embodiments described above. For specific implementation of each step in the program 1010, reference may be made to corresponding steps and corresponding descriptions in units in the foregoing cash register correction method embodiment based on human-computer interaction, which are not described herein again. It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described devices and modules may refer to the corresponding process descriptions in the foregoing method embodiments, and are not described herein again.

The present invention also provides a computing device comprising: the processor, the memory and the communication interface complete mutual communication through the communication bus; the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the training method of the commodity identification model. The schematic structure of the computing device is the same as the schematic structure of the computing device shown in fig. 10, and is not described here again.

The algorithms or displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. In addition, embodiments of the present invention are not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of embodiments of the present invention as described herein, and any descriptions of specific languages are provided above to disclose preferred embodiments of the invention.

In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that is, the claimed embodiments of the invention require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functionality of some or all of the components according to embodiments of the present invention. Embodiments of the invention may also be implemented as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing embodiments of the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. Embodiments of the invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names. The steps in the above embodiments should not be construed as limiting the order of execution unless specified otherwise.

Claims

1. A cash register error correction method based on human-computer interaction is characterized by comprising the following steps:

2. The method of claim 1, wherein after said obtaining user correction data, the method further comprises: and feeding back the second commodity identification result to a server side so that the server side can update and train the commodity identification model according to the second commodity identification result.

3. The method according to claim 2, wherein the feeding back the second product identification result to the server for the server to perform the update training on the product identification model according to the second product identification result specifically comprises:

4. The method according to any one of claims 1-3, wherein the human-computer interaction interface comprises response areas corresponding to the first commodity identification results, and the response areas corresponding to the first commodity identification results are not overlapped with each other;

5. The method according to any one of claims 1-3, wherein the monitoring of the correction operation performed by the user on the human-computer interface for any first product identification result further comprises:

6. The method of claim 5, wherein the at least one candidate item identification result is specifically an item identification result that has a lower confidence level from the item identification model output than the first item identification result.

7. The method according to any one of claims 1-3, wherein the monitoring of the correction operation performed by the user on the human-computer interface for any first product identification result further comprises:

8. The method according to any one of claims 1-3, wherein the monitoring of the correction operation performed by the user on the human-computer interface for any first product identification result further comprises:

9. The method of any of claims 1-3, wherein presenting the at least one first item identification result in a human-machine interface further comprises:

10. A training method of a commodity identification model is characterized by comprising the following steps:

11. The method according to claim 10, wherein the updating and training of the product recognition model according to the second product recognition result specifically comprises:

12. The method according to claim 10 or 11, wherein the step of inputting the shot commodity image into a commodity recognition model obtained by pre-training to obtain a first commodity recognition result specifically comprises:

13. The method according to claim 12, wherein the at least one candidate item identification result is in particular an item identification result with a lower confidence than the first item identification result.

14. The method of claim 11, wherein the training for updating the product recognition model according to the second product recognition result further comprises:

15. The method of claim 11, wherein the training for updating the product recognition model according to the second product recognition result further comprises:

16. The utility model provides a receive silver-colored error correction device based on human-computer interaction which characterized in that includes:

17. A training device for a commodity recognition model, comprising:

18. A computing device, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;

the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the operation corresponding to the cash register correction method based on human-computer interaction according to any one of claims 1 to 9, or execute the operation corresponding to the training method of the commodity identification model according to any one of claims 10 to 15.

19. A computer storage medium, wherein the storage medium stores at least one executable instruction, and the executable instruction causes a processor to execute an operation corresponding to the cash correction method based on human-computer interaction according to any one of claims 1 to 9, or execute an operation corresponding to the training method of the commodity identification model according to any one of claims 10 to 15.