CN111222388A - Settlement method and system based on visual identification - Google Patents
Settlement method and system based on visual identification Download PDFInfo
- Publication number
- CN111222388A CN111222388A CN201811509823.1A CN201811509823A CN111222388A CN 111222388 A CN111222388 A CN 111222388A CN 201811509823 A CN201811509823 A CN 201811509823A CN 111222388 A CN111222388 A CN 111222388A
- Authority
- CN
- China
- Prior art keywords
- commodities
- settlement
- primary
- trained
- shooting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
- G07G1/00—Cash registers
- G07G1/0036—Checkout procedures
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
- G07G1/00—Cash registers
- G07G1/0036—Checkout procedures
- G07G1/0045—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader
- G07G1/0054—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader with control of supplementary check-parameters, e.g. weight or number of articles
- G07G1/0072—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader with control of supplementary check-parameters, e.g. weight or number of articles with means for detecting the weight of the article of which the code is read, for the verification of the registration
Abstract
The invention belongs to the technical field of image recognition and discloses a settlement method and system based on visual recognition. The settlement method comprises the following steps: acquiring a plurality of images of all commodities on a table top of a checkout counter, wherein the plurality of images correspond to a plurality of shooting angles for shooting the commodities downwards one by one, the plurality of shooting angles comprise at least one first shooting angle, and a shooting point of the first shooting angle is positioned right above the table top of the checkout counter; acquiring a plurality of primary classification results corresponding to each commodity according to the plurality of images and a pre-trained primary classification model, and acquiring a primary classification result corresponding to each commodity according to the plurality of primary classification results corresponding to each commodity and the pre-trained primary linear regression model; and settling accounts according to the primary classification result corresponding to each commodity. The settlement system comprises an image acquisition device, a primary classification device and a settlement device. The invention can accurately identify the commodity and is convenient for self-service settlement through the technical scheme.
Description
Technical Field
The invention belongs to the technical field of image recognition, and particularly relates to a settlement method and system based on visual recognition.
Background
When a customer sees a favorite or required commodity in a shopping place such as a supermarket or a restaurant, the commodity can be obtained only by settling at a settlement table.
In the prior art, two common settlement methods are used: the first is a bar code-based settlement method, in which a commodity is identified by scanning a bar code on the commodity, and then the identified commodity is settled, and the scanning operation is performed by a cashier or by a customer. This method has the following drawbacks: scanning is troublesome, labor-consuming, has certain requirements on operation, can only scan one commodity at a time, cannot scan a plurality of commodities at the same time, and has low efficiency. The second is a settlement method based on RFID, which is to stick a small radio frequency module without battery on the commodity, when the commodity passes through the settlement station, the settlement station will transmit wireless signal to the commodity, the small radio frequency module will feed back a signal to the settlement station after receiving the signal, the feedback signal contains ID information of the commodity, and then settlement is performed based on the ID information. This method has the following drawbacks: the small radio frequency module is required to be attached to each commodity, the commodity is troublesome, and if the small radio frequency module falls off from the commodity, the loss is caused to merchants no matter the module falls naturally or is torn artificially. In addition, when the goods are metal goods, there is a problem that signals are shielded by attaching RFID thereon.
Disclosure of Invention
In order to solve the above problems, an aspect of the present invention provides a settlement method based on visual recognition, including: acquiring a plurality of images of all commodities on a table top of a checkout counter, wherein the plurality of images correspond to a plurality of shooting angles for shooting the commodities downwards one by one, the plurality of shooting angles comprise at least one first shooting angle, and a shooting point of the first shooting angle is positioned right above the table top of the checkout counter; acquiring a plurality of primary classification results corresponding to the commodities according to the images and a pre-trained primary classification model, wherein the primary classification model is an image recognition technical framework based on a convolutional neural network and is a model trained by all commodities in a shopping place, and acquiring a primary classification result corresponding to the commodities according to the primary classification results corresponding to the commodities and a pre-trained primary linear regression model; and settling accounts according to the primary classification result corresponding to each commodity.
In the settlement method as described above, preferably, the number of the shooting angles is two, a shooting point at which one of the shooting angles is located directly above the center of the top board of the settlement table, and a shooting point at which the other of the shooting angles is located obliquely above the top board of the settlement table.
In the settlement method as described above, preferably, the obtaining a primary classification result corresponding to each of the commodities according to the plurality of images and a pre-trained primary classification model specifically includes: determining a first target area image according to the image acquired at the first shooting angle, and respectively performing target detection on each image at the remaining shooting angles according to the number of the first target area images to obtain a second target area image, wherein the remaining shooting angles are shooting angles except the first shooting angle in the plurality of shooting angles; and acquiring a plurality of primary classification results corresponding to the commodities according to the first target area image, the second target area image and a pre-trained primary classification model.
In the settlement method as described above, preferably, after the obtaining of the primary classification result corresponding to each of the commodities based on the plurality of primary classification results corresponding to each of the commodities and a pre-trained primary linear regression model, the settlement method further includes: judging whether the primary classification result is a similar commodity or not; if the similar commodities are judged, acquiring a plurality of secondary classification results corresponding to the similar commodities according to the plurality of images and a pre-trained secondary classification model, acquiring a secondary classification result corresponding to the similar commodities according to the plurality of secondary classification results and a pre-trained secondary linear regression model, and settling accounts according to the secondary classification result, wherein the secondary classification model is a model which is based on an image recognition technology framework of a convolutional neural network in advance and is trained by commodities in a similar commodity group in a shopping place; if not, the step is skipped to, and the settlement is carried out according to the primary classification result corresponding to the commodity.
In the settlement method as described above, preferably, before the settlement is performed according to the primary classification result corresponding to each of the products, the settlement method further includes: and obtaining a third-level classification result according to the plurality of images and a pre-trained support vector machine model corresponding to the first-level classification result, judging whether the third-level classification result is consistent with the first-level classification result, if so, skipping to the step for settlement according to the first-level classification result corresponding to each commodity, otherwise, reminding a consumer that the settlement cannot be carried out.
In another aspect, the present invention provides a settlement system based on visual recognition, which includes: the system comprises an image acquisition device, a counting platform and a counting platform, wherein the image acquisition device is used for acquiring a plurality of images of all commodities on the platform surface of the counting platform, the plurality of images correspond to a plurality of shooting angles for shooting the commodities downwards one by one, the plurality of shooting angles comprise at least one first shooting angle, and a shooting point of the first shooting angle is positioned right above the platform surface of the counting platform; the primary classification device is used for acquiring a plurality of primary classification results corresponding to the commodities according to the images and a pre-trained primary classification model, the primary classification model is a model which is based on an image recognition technology framework of a convolutional neural network and is trained by all commodities in a shopping place, and the primary classification results corresponding to the commodities are acquired according to the primary classification results corresponding to the commodities and a pre-trained primary linear regression model; and the settlement device is used for performing settlement according to the primary classification result corresponding to each commodity.
In the settlement system as described above, preferably, the primary classification device is specifically configured to: determining a first target area image according to the image acquired at the first shooting angle, and respectively performing target detection on each image at the rest shooting angles according to the number of the first target area images to obtain a second target area image, wherein the rest shooting angles are shooting angles except the first shooting angle in the plurality of shooting angles; and acquiring a plurality of primary classification results corresponding to the commodities according to the first target area image, the second target area image and a pre-trained primary classification model.
In the settlement system as described above, preferably, the settlement system further includes: the first judgment device is used for judging whether the primary classification result is a similar commodity; the second-stage classification device is used for acquiring a plurality of second-stage classification results corresponding to the similar commodities according to the plurality of images and a pre-trained second-stage classification model if the similar commodities are judged, acquiring second-stage classification results corresponding to the similar commodities according to the plurality of second-stage classification results and a pre-trained second-stage linear regression model, and settling accounts according to the second-stage classification results, wherein the second-stage classification model is a model which is based on an image recognition technical framework of a convolutional neural network in advance and is trained by commodities in a similar commodity group in a shopping place; and the first selection device is used for jumping to the settlement device if the judgment result is negative.
In the settlement system as described above, preferably, the settlement system further includes: the three-level classification device is used for obtaining a three-level classification result according to the plurality of images and a pre-trained support vector machine model corresponding to the first-level classification result; the second judgment device is used for judging whether the tertiary classification result is consistent with the primary classification result or not; and the second selection device is used for jumping to the settlement device if the judgment result is that the two devices are consistent, otherwise, reminding the consumer that the settlement cannot be carried out.
Still another aspect of the present invention provides a settlement system based on visual recognition, which includes: the system comprises a plurality of cameras, a plurality of storage devices and a plurality of display devices, wherein the plurality of cameras are used for collecting a plurality of images of all commodities placed on a table top of a settlement table, the plurality of images correspond to shooting angles of the plurality of cameras one by one, the plurality of shooting angles comprise at least one first shooting angle, and the camera corresponding to the first shooting angle is positioned right above the table top of the settlement table; a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: and acquiring a plurality of primary classification results corresponding to the commodities according to the images and a pre-trained primary classification model, wherein the primary classification model is an image recognition technical framework based on a convolutional neural network and is a model trained by all commodities in a shopping place, acquiring a primary classification result corresponding to the commodities according to the primary classification results corresponding to the commodities and a pre-trained primary linear regression model, and settling accounts according to the primary classification result corresponding to the commodities.
The technical scheme provided by the embodiment of the invention has the following beneficial effects:
can accurately identify commodities and is convenient for self-service settlement.
Drawings
Fig. 1 is a schematic flow chart illustrating a settlement method based on visual recognition according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of a settlement method based on visual recognition according to another embodiment of the present invention;
fig. 3 is a schematic flow chart of a settlement method based on visual recognition according to another embodiment of the present invention;
fig. 4 is a schematic structural diagram of a settlement apparatus based on visual recognition according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
The present invention will be described in detail below with reference to the embodiments with reference to the attached drawings. The various examples are provided by way of explanation of the invention, and not limitation of the invention. In fact, it will be apparent to those skilled in the art that modifications and variations can be made in the present invention without departing from the scope or spirit thereof. For instance, features illustrated or described as part of one embodiment, can be used with another embodiment to yield a still further embodiment. It is therefore intended that the present invention encompass such modifications and variations as fall within the scope of the appended claims and equivalents thereof.
Referring to fig. 1, an embodiment of the present invention provides a settlement method based on visual recognition, including:
102, acquiring a plurality of primary classification results corresponding to each commodity according to the plurality of images and a pre-trained primary classification model, wherein the primary classification model is an image recognition technical framework based on a convolutional neural network and is a model trained by all commodities in a shopping place, and acquiring the primary classification results corresponding to each commodity according to the plurality of primary classification results corresponding to each commodity and the pre-trained primary linear regression model.
And 103, settling accounts according to the primary classification result corresponding to each commodity.
The method comprises the steps of shooting all commodities on a counter table under a plurality of shooting angles to obtain a plurality of images, then obtaining a plurality of primary classification results corresponding to the commodities according to the images and a pre-trained primary classification model, wherein the primary classification model is a model which is based on an image recognition technical framework of a convolutional neural network and trained by all the commodities in a shopping place, obtaining the primary classification results corresponding to the commodities according to the primary classification results corresponding to the commodities and a pre-trained primary linear regression model, and then settling accounts according to the primary classification results of the commodities, so that the settling efficiency is improved, and the accuracy of commodity recognition is also improved.
Referring to fig. 2, another embodiment of the present invention provides a settlement method based on visual recognition, which, in combination with the content of the first embodiment, includes the following steps:
Specifically, an image can be acquired by photographing a commodity placed on the top of the checkout counter from a certain photographing angle. Because one shooting angle corresponds to one image, different shooting angles are changed, and images corresponding to different shooting angles can be collected. The manner of realizing the plurality of shooting angles may be to arrange a plurality of cameras, which correspond to the plurality of shooting angles one to one; the method may further include arranging N cameras, and implementing a plurality of shooting angles by changing positions and/or orientations of the cameras, where N is smaller than the number of shooting angles, and the implementation manner of the plurality of shooting angles is not limited in this embodiment. The shooting angles of the embodiment are all located above the table top of the settlement table, namely, the shooting directions are all downward when the commodity is shot, the table top of the settlement table is taken as a reference, at least one shooting point where the shooting angle is located right above the table top of the settlement table, the right above is located in a columnar area defined by a cylindrical surface and the table top of the settlement table, and the cylindrical surface is a curved surface formed by parallel movement of a straight line perpendicular to the table top of the settlement table along the edge of the table top of the settlement table. Obliquely above means outside the columnar area.
The number of the cameras is M, the number of the M can be 2, for example, 1 shooting point is positioned right above the table top of the settlement table, and the other 1 shooting point is positioned obliquely above the table top of the settlement table; 3 shooting points can be arranged, wherein 1 shooting point is positioned right above the table top of the settlement table, and the other 2 shooting points are positioned obliquely above one side of the table top of the settlement table; or 4, 2 shooting points are all positioned right above the table board of the settlement table, and the other 2 shooting points are all positioned obliquely above the table board of the settlement table. When the number of the cameras positioned right above the table board of the checkout counter is more than 1, preferably, one camera is positioned right above the center of the table board of the checkout counter, the camera can be called as a center camera, and other cameras positioned right above are positioned at the periphery of the center camera, so that the number of the commodities placed on the table board of the checkout counter can be accurately acquired through images. It should be noted that: the number and arrangement of the cameras are not limited in this embodiment. Generally, the more the number of the cameras is, the more the number of the acquired images is, the more the information of the commodities contained in all the images is, which is beneficial to the classification of the commodities, but the complexity of hardware is increased, the calculation amount is increased, and therefore the number of the cameras can be selected according to the actual situation. It should be noted that, the shooting angle in this document may refer to one factor of the shooting direction, may also refer to two factors of the shooting direction and the shooting distance, and may also refer to other factors or other number of factors, which is not limited in this embodiment.
The collecting action (or shooting action) can be triggered by a scale arranged on the settlement table, for example, the scale is a scale with a pressure sensor, and whether shooting is triggered or not is determined according to the change of the weight sensed by the scale. When the weight sensed by the scale changes and the change is stabilized, if the weight does not change within a preset time (namely, a time period taking the weight change time as an initial time and taking the initial time plus the preset time as an end time), the camera starts shooting, so that an image meeting the requirements can be ensured to be shot, namely, the image at the moment is shot after a customer puts a commodity. In other embodiments, the triggering of the camera photographing action may adopt a technical means of pattern recognition and computer vision, the camera first continuously observes and photographs an area where a commodity is placed, for example, when a hand of a customer is found to be extended, a commodity is put down, and then the hand is taken away, when such an action is captured from a video, that is, the placing action of the commodity, an initial time is recorded, and when such an action is not captured from the video within a preset time (that is, a time period in which the action moment is captured as the initial time and the initial time + the preset time is taken as the end time), a photographing instruction is taken, that is, the camera is triggered to photograph.
Specifically, the target detection is performed on the image acquired at the first shooting angle, and when the target is detected, the target areas corresponding to the commodities are pulled out from the image, each target area contains one commodity, the number of the target areas is equal to the number of the commodities, and the image corresponding to the target area is referred to as a first target area image, which is an image used for classifying the commodities, and is output to step 203. The number of first target area images is the same as the number of target areas. When the number of the commodities on the table surface of the settlement table is multiple, the number of the first target area images is also multiple; when the number of the commodities on the counter top is 1, the number of the first target area images is 1. The shape of the target area may be rectangular or circular. Because there is no blocking, the number of commodities to be classified placed on the top of the checkout counter can be accurately determined in the image obtained by shooting from the top of the commodity to the bottom, and according to the number, target detection is performed on each image obtained at other shooting angles (or called residual shooting angles), when target detection is performed, the target areas with the same number as that of the first target area images are pulled out from each image at the residual shooting angles, each target area also contains a commodity, and the image corresponding to the target area is called a second target area image which is an image for classifying the commodities and is output to step 203.
Specifically, the data collection is performed to establish a data set, and the data collection process includes: 1) a large number of images are taken of all items within a shopping venue from various angles of capture and in various poses. 2) These images are then annotated: and marking the position, size and type of the commodity in the image. The data set includes data referring to the images and the annotations made on the images. The primary classification model is a model of an image recognition technology architecture based on a convolutional neural network, is trained by using data of all commodities in a shopping place, and can be trained in a gradient descending mode.
And the trained primary classification model classifies a first target area image in the image acquired at the first shooting angle and a second target area image in the image acquired at the rest shooting angles. And obtaining a plurality of primary classification results corresponding to each commodity due to the number of the images, wherein the number of the primary classification results is consistent with the number of the images, the primary classification result is an O-dimensional vector, O represents the total number of the commodities in the shopping place, and the meaning of each element in the vector represents the probability that the commodity to be classified belongs to each commodity in the O commodities by the primary classification model.
And 204, acquiring a primary classification result corresponding to each commodity according to the plurality of primary classification results corresponding to each commodity and a pre-trained primary linear regression model.
Specifically, when the primary classification model is trained in step 203, the primary classification result output by the primary classification model is used as the input of the primary linear regression model, and the correct classification of the commodity contained in the image corresponding to the primary classification result is used as the output of the primary linear regression model, so as to train the primary linear regression model. And the trained primary linear regression model performs data fusion on a plurality of primary classification results of the commodities to obtain a primary classification result corresponding to the commodities, and the primary classification result represents which category of the commodities in the shopping place is one of the commodities in a predicted image of the primary linear regression model.
There are a variety of goods in a shopping venue, and there are some goods that are similar in appearance and visually confusable among the various goods, which are referred to as similar goods, such as yellow marshal apple and yellow snow pear. If a single commodity to be classified is a similar commodity, the first-level classification model is difficult to accurately classify the commodity, and if the apple of the yellow marshal is mixed with the yellow snow pear, the apple of the yellow marshal is classified into the yellow snow pear, so that the settlement method further comprises the following steps in order to improve the identification accuracy:
Specifically, a plurality of similar commodities are arranged into a similar commodity table, after a primary classification result is obtained, searching is carried out in a preset similar commodity table, and if similar commodities matched with the primary classification result are searched, the primary classification result is judged to be similar commodities; if the similar commodity matched with the primary classification result is not found, judging that the primary classification result is not the similar commodity.
And step 206, acquiring a plurality of secondary classification results corresponding to the similar commodities according to the plurality of images and a pre-trained secondary classification model, acquiring a secondary classification result corresponding to the similar commodities according to the plurality of secondary classification results and a pre-trained secondary linear regression model, and settling accounts according to the secondary classification result, wherein the secondary classification model is a model which is based on an image recognition technical framework of a convolutional neural network in advance and is trained by commodities in a similar commodity group in a shopping place.
Specifically, the secondary classification model is trained by using the data of similar commodities in the data set established in step 203, and the training may be performed in a gradient descent manner. The second-level classification model is also an image recognition technical framework based on a convolutional neural network, the difference between the second-level classification model and the first-level classification model is that the data used in training is different, the data used by the first-level classification model is the data of all goods in a shopping place, and the data used by the second-level classification model is the data of similar goods in the shopping place.
The trained secondary classification model classifies the commodities in the first target area image and the second target area image corresponding to the similar commodities to obtain a plurality of secondary classification results corresponding to the similar commodities, each secondary classification result is also a p-dimensional vector, the meaning of each element in the vector represents the probability that the commodity to be classified belongs to each commodity in p similar commodities by the secondary classification model, and p is less than or equal to O and represents the total number of the similar commodities in a shopping place.
In practice, there are multiple groups of similar commodities in the shopping place, such as one group of similar commodities including yellow marshal apples and yellow snow pears, and another group of similar commodities including salt in bulk and white sugar in bulk; another group of similar goods includes soda and flour. The method comprises the steps of training a secondary classification model aiming at all groups of similar commodities, training a secondary classification model aiming at each group of similar commodities in order to further improve the accuracy of commodity classification, and calling the secondary classification model corresponding to the primary classification result if the primary classification result is the similar commodity.
And when the secondary classification model is trained, taking a secondary classification result output by the secondary classification model as the input of the secondary linear regression model, and taking the correct classification of the commodities contained in the image corresponding to the secondary classification result as the output of the secondary linear regression model so as to train the secondary linear regression model. And the trained secondary linear regression model performs data fusion on a plurality of secondary classification results corresponding to the similar commodities to obtain a secondary classification result, and the secondary classification result represents which category of the commodities in the shopping place is predicted in the image predicted by the secondary linear regression model.
And then settlement is carried out according to the secondary classification result.
After the second-level classification result is obtained, the commodity price corresponding to the second-level classification result is obtained, and then the commodity price of the commodity placed on the settlement table is obtained, so that the cost required to be paid by the customer for the commodity placed on the settlement table is determined, the commodity name, the commodity price and the payment cost can be displayed through a display on the settlement table, and the commodity name can be prompted to the customer through voice. When the customer pays, the payment can be completed by scanning the two-dimensional code displayed by the display or aligning the two-dimensional code of the account on the mobile terminal to the code scanning terminal on the settlement table, and the automatic settlement can be realized by the pre-bound payment account.
And step 207, performing settlement according to the primary classification result corresponding to each commodity.
After the primary classification result is obtained, the commodity price corresponding to the primary classification result is obtained, then the commodity price of the commodity placed on the settlement table is obtained, the price of all the commodities on the table surface of the settlement table can be obtained by operating all the commodities, the cost required to be paid by a customer for the commodity placed on the settlement table is determined, the commodity name, the commodity price and the payment cost can be displayed through a display on the settlement table, and the commodity name can be prompted to the customer through voice. When the customer pays, the payment can be completed by scanning the two-dimensional code displayed by the display or aligning the two-dimensional code of the account on the mobile terminal to the code scanning terminal on the settlement table, and the automatic settlement can be realized by the pre-bound payment account.
In order to avoid classification errors and improve the accuracy of settlement, referring to fig. 3, before step 103, the settlement method further includes:
and 208, obtaining a third-level classification result according to the plurality of images and the pre-trained support vector machine model corresponding to the first-level classification result, judging whether the third-level classification result is consistent with the first-level classification result, if so, skipping to 207 to settle according to the first-level classification result corresponding to each commodity, otherwise, executing step 209, and reminding the consumer that the settlement cannot be performed.
Specifically, a machine learning model (or support vector machine model) of the support vector machine is constructed for each commodity in the shopping venue, that is, each commodity has a support vector machine model corresponding to the commodity, and the model is trained by using data corresponding to the commodity in the data set established in step 203. When the first-level classification model is constructed, an intermediate calculation result exists, the intermediate calculation result is a vector with the length of 1024, the vector can be regarded as a feature of an image, and accordingly a support vector machine model from the vector to the step of judging whether the commodity belongs to a certain class of commodities is constructed.
After the primary classification result of each commodity is obtained, a support vector machine model corresponding to the primary classification result is used for judging a first target area image in an image obtained at a first shooting angle and a second target area image in images obtained at the rest shooting angles to obtain a plurality of primary judgment results, and the primary judgment results show whether the commodity in each image is consistent with the primary classification result. And if the number of the images is N, the number of the initial judgment results is N. If the number of the N primary judgment results is larger than or equal to a preset threshold value, the commodity in the image is judged to be consistent with the primary classification result, otherwise, the customer is reminded that the commodity is out of stock, namely, the commodity cannot be settled, the commodity can be marked in the image collected at the first shooting angle to indicate that the commodity cannot be settled, the commodity is displayed by a display to remind the customer, and the commodity can be prompted in a voice mode, wherein the voice can contain text contents such as 'no stock', 'no identification' and the like, and can also be an alarm sound such as a dripping.
Referring to fig. 4, an embodiment of the present invention provides a settlement system based on visual recognition, for performing the method provided by the above embodiment, which includes: an image acquisition device 401, a primary classification device 402 and a settlement device 403.
The image acquiring device 401 is configured to acquire a plurality of images of all the commodities on the top of the checkout counter, where the plurality of images correspond to a plurality of downward shooting angles for shooting the commodities one by one, the plurality of shooting angles include at least one first shooting angle, and a shooting point at which the first shooting angle is located is directly above the top of the checkout counter.
The primary classification device 402 is configured to obtain a plurality of primary classification results corresponding to the commodities according to the plurality of images and a pre-trained primary classification model, where the primary classification model is a model based on an image recognition technology architecture of a convolutional neural network and trained by all commodities in a shopping venue, and obtain a primary classification result corresponding to the commodities according to the plurality of primary classification results corresponding to the commodities and the pre-trained primary linear regression model.
The settlement device 403 is used to perform settlement according to the primary classification result corresponding to each product.
Preferably, the primary classification device 402 is specifically configured to: determining a first target area image according to the image acquired at the first shooting angle, and respectively performing target detection on each image at the rest shooting angles according to the number of the first target area images to obtain a second target area image, wherein the rest shooting angles are shooting angles except the first shooting angle in the plurality of shooting angles; and acquiring a plurality of primary classification results corresponding to each commodity according to the first target area image, the second target area image and a pre-trained primary classification model.
Preferably, the settlement system further comprises: the first judgment device is used for judging whether the primary classification result is a similar commodity; the second-stage classification device is used for acquiring a plurality of second-stage classification results corresponding to the similar commodities according to the plurality of images and a pre-trained second-stage classification model if the similar commodities are judged, acquiring a second-stage classification result corresponding to the similar commodities according to the plurality of second-stage classification results and a pre-trained second-stage linear regression model, and settling accounts according to the second-stage classification results, wherein the second-stage classification model is a model which is based on an image recognition technology framework of a convolutional neural network in advance and is trained by commodities in a similar commodity group in a shopping place; and the first selection device is used for jumping to the settlement device if the judgment result is negative.
Preferably, the settlement system further comprises: the three-level classification device is used for obtaining a three-level classification result according to the multiple images and a pre-trained support vector machine model corresponding to the first-level classification result; the second judgment device is used for judging whether the three-level classification result is consistent with the first-level classification result or not; and the second selection device is used for jumping to the settlement device if the judgment result is that the two devices are consistent, otherwise, reminding the consumer that the settlement cannot be carried out.
It should be noted that, for specific descriptions of the image obtaining device 401, the primary classifying device 402, the settling device 403, the first judging device, the secondary classifying device, the first selecting device, the tertiary classifying device, the second judging device, and the second selecting device, reference may be made to the related contents of steps 101 to 103 and 201 to 209 in the foregoing embodiment, and details are not repeated here.
An embodiment of the present invention provides a settlement system based on visual recognition, which includes: a plurality of cameras, a processor, and a memory.
The plurality of cameras are used for collecting a plurality of images of commodities placed on the table top of the settlement table, the plurality of images correspond to shooting angles of the plurality of cameras one to one, the plurality of shooting angles comprise at least one first shooting angle, and the camera corresponding to the first shooting angle is located right above the table top of the settlement table. The memory is used for storing instructions executable by the processor. The processor is configured to: the method comprises the steps of obtaining a plurality of primary classification results corresponding to commodities according to a plurality of images and a pre-trained primary classification model, wherein the primary classification model is a model which is based on an image recognition technology framework of a convolutional neural network and is trained by all commodities in a shopping place, obtaining the primary classification results corresponding to the commodities according to the primary classification results and the pre-trained primary linear regression model, and settling accounts according to the primary classification results corresponding to the commodities. For specific descriptions of the camera and the processor, reference may be made to the related contents of steps 101 to 103 and 201 to 209 in the above embodiments, and details are not repeated here.
In summary, the embodiments of the present invention have the following beneficial effects:
can accurately identify commodities and is convenient for self-service settlement.
It will be appreciated by those skilled in the art that the invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The embodiments disclosed above are therefore to be considered in all respects as illustrative and not restrictive. All changes which come within the scope of or equivalence to the invention are intended to be embraced therein.
Claims (10)
1. A settlement method based on visual recognition, characterized in that the settlement method comprises:
acquiring a plurality of images of all commodities on a table top of a checkout counter, wherein the plurality of images correspond to a plurality of shooting angles for shooting the commodities downwards one by one, the plurality of shooting angles comprise at least one first shooting angle, and a shooting point of the first shooting angle is positioned right above the table top of the checkout counter;
acquiring a plurality of primary classification results corresponding to the commodities according to the images and a pre-trained primary classification model, wherein the primary classification model is an image recognition technical framework based on a convolutional neural network and is a model trained by all commodities in a shopping place, and acquiring a primary classification result corresponding to the commodities according to the primary classification results corresponding to the commodities and a pre-trained primary linear regression model;
and settling accounts according to the primary classification result corresponding to each commodity.
2. The settlement method according to claim 1, wherein the number of the photographing angles is two, one of the photographing angles is located at a photographing point right above the center of the top of the settlement table, and the other photographing angle is located at a photographing point obliquely above the top of the settlement table.
3. The settlement method according to claim 1 or 2, wherein the obtaining of the primary classification result corresponding to each of the commodities based on the plurality of images and a pre-trained primary classification model specifically comprises:
determining a first target area image according to the image acquired at the first shooting angle, and respectively performing target detection on each image at the remaining shooting angles according to the number of the first target area images to obtain a second target area image, wherein the remaining shooting angles are shooting angles except the first shooting angle in the plurality of shooting angles;
and acquiring a plurality of primary classification results corresponding to the commodities according to the first target area image, the second target area image and a pre-trained primary classification model.
4. The settlement method according to claim 1 or 2, wherein after the primary classification results corresponding to each of the commodities are obtained from the plurality of primary classification results corresponding to each of the commodities and a pre-trained primary linear regression model, the settlement method further comprises:
judging whether the primary classification result is a similar commodity or not;
if the similar commodities are judged, acquiring a plurality of secondary classification results corresponding to the similar commodities according to the plurality of images and a pre-trained secondary classification model, acquiring a secondary classification result corresponding to the similar commodities according to the plurality of secondary classification results and a pre-trained secondary linear regression model, and settling accounts according to the secondary classification result, wherein the secondary classification model is a model which is based on an image recognition technology framework of a convolutional neural network in advance and is trained by commodities in a similar commodity group in a shopping place;
if not, the step is skipped to, and the settlement is carried out according to the primary classification result corresponding to the commodity.
5. The settlement method according to claim 1, wherein before the settlement according to the primary classification result corresponding to each of the commodities, the settlement method further comprises:
and obtaining a third-level classification result according to the plurality of images and a pre-trained support vector machine model corresponding to the first-level classification result, judging whether the third-level classification result is consistent with the first-level classification result, if so, skipping to the step for settlement according to the first-level classification result corresponding to each commodity, otherwise, reminding a consumer that the settlement cannot be carried out.
6. A visual recognition-based settlement system, comprising:
the system comprises an image acquisition device, a counting platform and a counting platform, wherein the image acquisition device is used for acquiring a plurality of images of all commodities on the platform surface of the counting platform, the plurality of images correspond to a plurality of shooting angles for shooting the commodities downwards one by one, the plurality of shooting angles comprise at least one first shooting angle, and a shooting point of the first shooting angle is positioned right above the platform surface of the counting platform;
the primary classification device is used for acquiring a plurality of primary classification results corresponding to the commodities according to the images and a pre-trained primary classification model, the primary classification model is a model which is based on an image recognition technology framework of a convolutional neural network and is trained by all commodities in a shopping place, and the primary classification results corresponding to the commodities are acquired according to the primary classification results corresponding to the commodities and a pre-trained primary linear regression model;
and the settlement device is used for performing settlement according to the primary classification result corresponding to each commodity.
7. The settlement system of claim 6, wherein the primary classification means is specifically configured to:
determining a first target area image according to the image acquired at the first shooting angle, and respectively performing target detection on each image at the rest shooting angles according to the number of the first target area images to obtain a second target area image, wherein the rest shooting angles are shooting angles except the first shooting angle in the plurality of shooting angles;
and acquiring a plurality of primary classification results corresponding to the commodities according to the first target area image, the second target area image and a pre-trained primary classification model.
8. The settlement system according to claim 6, further comprising:
the first judgment device is used for judging whether the primary classification result is a similar commodity;
the second-stage classification device is used for acquiring a plurality of second-stage classification results corresponding to the similar commodities according to the plurality of images and a pre-trained second-stage classification model if the similar commodities are judged, acquiring second-stage classification results corresponding to the similar commodities according to the plurality of second-stage classification results and a pre-trained second-stage linear regression model, and settling accounts according to the second-stage classification results, wherein the second-stage classification model is a model which is based on an image recognition technical framework of a convolutional neural network in advance and is trained by commodities in a similar commodity group in a shopping place;
and the first selection device is used for jumping to the settlement device if the judgment result is negative.
9. The settlement system according to claim 6, further comprising:
the three-level classification device is used for obtaining a three-level classification result according to the plurality of images and a pre-trained support vector machine model corresponding to the first-level classification result;
the second judgment device is used for judging whether the tertiary classification result is consistent with the primary classification result or not;
and the second selection device is used for jumping to the settlement device if the judgment result is that the two devices are consistent, otherwise, reminding the consumer that the settlement cannot be carried out.
10. A visual recognition-based settlement system, comprising:
the system comprises a plurality of cameras, a plurality of storage devices and a plurality of display devices, wherein the plurality of cameras are used for collecting a plurality of images of all commodities placed on a table top of a settlement table, the plurality of images correspond to shooting angles of the plurality of cameras one by one, the plurality of shooting angles comprise at least one first shooting angle, and the camera corresponding to the first shooting angle is positioned right above the table top of the settlement table;
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to:
and acquiring a plurality of primary classification results corresponding to the commodities according to the images and a pre-trained primary classification model, wherein the primary classification model is an image recognition technical framework based on a convolutional neural network and is a model trained by all commodities in a shopping place, acquiring a primary classification result corresponding to the commodities according to the primary classification results corresponding to the commodities and a pre-trained primary linear regression model, and settling accounts according to the primary classification result corresponding to the commodities.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811509823.1A CN111222388B (en) | 2018-12-11 | 2018-12-11 | Settlement method and system based on visual recognition |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811509823.1A CN111222388B (en) | 2018-12-11 | 2018-12-11 | Settlement method and system based on visual recognition |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111222388A true CN111222388A (en) | 2020-06-02 |
| CN111222388B CN111222388B (en) | 2023-09-19 |
Family
ID=70827438
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811509823.1A Active CN111222388B (en) | 2018-12-11 | 2018-12-11 | Settlement method and system based on visual recognition |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111222388B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113159089A (en) * | 2021-01-18 | 2021-07-23 | 安徽建筑大学 | Pavement damage identification method, system, computer equipment and storage medium |
| WO2023124172A1 (en) * | 2021-12-30 | 2023-07-06 | 百富计算机技术（深圳）有限公司 | Product checkout processing method, apparatus, terminal device, and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101814147A (en) * | 2010-04-12 | 2010-08-25 | 中国科学院自动化研究所 | Method for realizing classification of scene images |
| CN105335710A (en) * | 2015-10-22 | 2016-02-17 | 合肥工业大学 | Fine vehicle model identification method based on multi-stage classifier |
| CN108320404A (en) * | 2017-09-27 | 2018-07-24 | 缤果可为（北京）科技有限公司 | Commodity recognition method, device, self-service cashier based on neural network |
| US20180225534A1 (en) * | 2017-02-06 | 2018-08-09 | Toshiba Tec Kabushiki Kaisha | Article recognition apparatus and article recognition method |
| CN207993164U (en) * | 2017-07-24 | 2018-10-19 | 图灵通诺（北京）科技有限公司 | The checkout apparatus of image recognition technology based on convolutional neural networks |
| JP2018180717A (en) * | 2017-04-06 | 2018-11-15 | 京セラドキュメントソリューションズ株式会社 | Commodity settlement system and commodity settlement method |
-
2018
- 2018-12-11 CN CN201811509823.1A patent/CN111222388B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101814147A (en) * | 2010-04-12 | 2010-08-25 | 中国科学院自动化研究所 | Method for realizing classification of scene images |
| CN105335710A (en) * | 2015-10-22 | 2016-02-17 | 合肥工业大学 | Fine vehicle model identification method based on multi-stage classifier |
| US20180225534A1 (en) * | 2017-02-06 | 2018-08-09 | Toshiba Tec Kabushiki Kaisha | Article recognition apparatus and article recognition method |
| JP2018180717A (en) * | 2017-04-06 | 2018-11-15 | 京セラドキュメントソリューションズ株式会社 | Commodity settlement system and commodity settlement method |
| CN207993164U (en) * | 2017-07-24 | 2018-10-19 | 图灵通诺（北京）科技有限公司 | The checkout apparatus of image recognition technology based on convolutional neural networks |
| CN109299715A (en) * | 2017-07-24 | 2019-02-01 | 图灵通诺（北京）科技有限公司 | The settlement method and device of image recognition technology based on convolutional neural networks |
| CN108320404A (en) * | 2017-09-27 | 2018-07-24 | 缤果可为（北京）科技有限公司 | Commodity recognition method, device, self-service cashier based on neural network |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113159089A (en) * | 2021-01-18 | 2021-07-23 | 安徽建筑大学 | Pavement damage identification method, system, computer equipment and storage medium |
| WO2023124172A1 (en) * | 2021-12-30 | 2023-07-06 | 百富计算机技术（深圳）有限公司 | Product checkout processing method, apparatus, terminal device, and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111222388B (en) | 2023-09-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109299715B (en) | Settlement method and device based on image recognition technology of convolutional neural network | |
| CN108320404B (en) | Commodity identification method and device based on neural network and self-service cash register | |
| CN109409175B (en) | Settlement method, device and system | |
| CN111222389B (en) | Analysis method and system for commodity on super commodity shelf | |
| CN109508974B (en) | Shopping checkout system and method based on feature fusion | |
| WO2022111059A1 (en) | Intelligent weighing apparatus based on computer vision technology, and intelligent weighing method | |
| US10503961B2 (en) | Object recognition for bottom of basket detection using neural network | |
| CN110866429A (en) | Missed scanning identification method and device, self-service cash register terminal and system | |
| CN109635705B (en) | Commodity identification method and device based on two-dimensional code and deep learning | |
| KR20190096975A (en) | Automatic In-Store Registration System | |
| CN111222870B (en) | Settlement method, device and system | |
| EP3901818A1 (en) | Method and system for analyzing commodities on supermarket shelf | |
| US10372998B2 (en) | Object recognition for bottom of basket detection | |
| EP3901841A1 (en) | Settlement method, apparatus, and system | |
| CN111222388B (en) | Settlement method and system based on visual recognition | |
| CN111428743B (en) | Commodity identification method, commodity processing device and electronic equipment | |
| EP3293685A1 (en) | Information processing apparatus that identifies an item based on a captured image thereof | |
| EP3474181A1 (en) | Device for automatic recognition of products | |
| US11812131B2 (en) | Determination of appropriate image suitable for feature extraction of object from among captured images in which object is detected | |
| CN109741119A (en) | Accounting method, device, system and computer readable storage medium | |
| JP2021103349A (en) | Information process system, information processing device, and information processing method | |
| CN115631585A (en) | Intelligent self-service weighing system for commodity settlement | |
| CN112669026A (en) | Automatic checkout system and implementation method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |