JP2020197758A - Settlement device and settlement method - Google Patents

Abstract

To provide an improved settlement device and settlement method for a user who purchases a commodity to settle the amount payable for the commodity.SOLUTION: A settlement device includes: a first camera that photographs a commodity placed on a table: a commodity image recognition circuit that recognizes a commodity based on an image of the commodity; a display device that displays the commodity recognition result; an input device that accepts input of correction instructions for the commodity recognition result; a microphone that detects voice; a voice recognition circuit that recognizes voice; and a control circuit that extracts a commodity candidate based on a keyword included in the voice recognition result. The display device displays the candidate. The input device accepts the selection of the candidate from a user. The control circuit corrects the commodity recognition result based on the selection.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a settlement device and a settlement method.

An increasing number of stores are introducing point-of-sale (POS, Point Of Sales) systems that manage sales for each product at the time of sale. The register used in the POS system includes a barcode reader that reads a barcode printed on the package of the product purchased by the customer, and the register identifies and registers the product based on the read barcode. A product registration device including a display unit and an operation unit has been proposed for registering a product (for example, vegetables, fruits) that does not have a barcode (Patent Document 1).

Japanese Unexamined Patent Publication No. 2014-6856

In recent years, as it has become difficult to secure store employees due to soaring labor costs, there is an increasing demand for self-checkouts in which customers who purchase products pay for the products without the intervention of the store employees. However, if a product registration device such as the product registration device described in Patent Document 1 that is supposed to be used by a store employee is diverted to a self-checkout and used by a customer, various difficulties occur. For example, the printing position of the barcode on the product package differs depending on the product. Therefore, since the customer searches for the barcode for each product in order to scan the barcode, it takes time to settle the payment, which results in an excessive burden on the customer. In addition, if the number of products handled by the store is large, the customer who does not know the position where the image of the product is displayed on the touch panel display searches for the image of the product for each product, so that it takes time to settle the payment and the customer The result is an excessive burden.

One embodiment of the present disclosure contributes to the provision of an improved settlement device and settlement method for a user who purchases a product to settle the price of the product.

The settlement device according to the present disclosure displays a first camera that photographs a product placed on a table, a product image recognition circuit that recognizes the product based on an image of the product, and a recognition result of the product. Based on the display device, the input device that receives the input of the correction instruction of the recognition result of the product, the microphone that detects the voice, the voice recognition circuit that recognizes the voice, and the keyword included in the result of the voice recognition. A control circuit for extracting a candidate for the product is provided, the display device displays the candidate, the input device accepts the selection of the candidate from the user, and the control circuit is based on the selection. , Correct the recognition result of the product, adopt the structure.

In the settlement method according to the present disclosure, the camera photographs the product placed on the table, the product image recognition circuit recognizes the product based on the image of the product, and the display device recognizes the product. The result is displayed, the input device receives the input of the correction instruction of the recognition result of the product, the microphone detects the voice, the voice recognition circuit recognizes the voice, and the control circuit recognizes the voice. Candidates for the product are extracted based on the keywords included in the result, the display device displays the candidates, the input device accepts the selection of the candidates from the user, and the control circuit is based on the selection. Therefore, a configuration is adopted in which the recognition result of the product is corrected.

It should be noted that these comprehensive or specific embodiments may be realized in a system, device, method, integrated circuit, computer program, or recording medium, and the system, device, method, integrated circuit, computer program, and recording medium. It may be realized by any combination of.

According to one embodiment of the present disclosure, it is possible to provide an improved settlement device and settlement method for a user who purchases a product to settle the price of the product.

Further advantages and effects in one embodiment of the present disclosure will be apparent from the specification and drawings. Such advantages and / or effects are provided by some embodiments and features described in the specification and drawings, respectively, but not all need to be provided in order to obtain one or more identical features. There is no.

The figure which shows an example of the external configuration of the settlement apparatus which concerns on this disclosure. The figure which shows an example of the internal structure of the settlement apparatus which concerns on this disclosure. A flowchart showing an example of the operation of the settlement device according to the present disclosure. A flowchart showing an example of the operation of the settlement device in step S102 of FIG. Diagram showing an example of the settlement screen A flowchart showing an example of the operation of the settlement device in step S108 of FIG. A flowchart showing an example of the operation of the settlement device in steps S304 and S308 of FIG. A flowchart showing an example of the operation of the settlement device in step S404 of FIG. Diagram showing an example of the product correction screen A flowchart showing an example of the operation of the settlement device in step S506 of FIG. Diagram showing an example of the product specification screen Diagram showing an example of the product specification screen A flowchart showing an example of the operation of the settlement device in step S410 of FIG. A flowchart showing an example of the operation of the settlement device in step S414 of FIG. A flowchart showing an example of the operation of the settlement device in step S110 of FIG.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The embodiments described below are examples, and the present disclosure is not limited to the following embodiments.

[Configuration of payment device 10]
FIG. 1 is a diagram showing an example of an external configuration of the settlement device 10 according to the present disclosure. FIG. 2 is a diagram showing an example of the internal configuration of the settlement device 10 according to the present disclosure.

The settlement device 10 is a device used by the user to settle the price of the product to be purchased. The user is, for example, a visitor. The checkout device 10 is installed, for example, between the product display shelf and the exit of the store. The settlement device 10 includes a settlement table (table, shooting area) 110, a first camera 130, a touch panel display 150, a second camera 160, a microphone 170, a voice recognition unit (voice recognition circuit) 210, and the like. Product image recognition unit (product image recognition circuit) 220, face image recognition unit (face image recognition circuit) 230, settlement processing unit (settlement processing circuit) 240, memory 260, storage 270, processor (control circuit) 280 and.

The checkout table 110 is a table on which the product purchased by the user is placed. The checkout table 110 is provided, for example, on the top plate 120 of the checkout device 10 so that the goods can be placed easily and stably. The checkout table 110 preferably exhibits one color such as white, gray, and black so that the product can be easily identified from the image taken by the first camera 130.

The first camera 130 captures a product purchased by the user. The first camera 130 is provided, for example, on the ceiling portion 140 of the settlement device 10 so as to face the settlement table 110. The first camera 130 includes, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

The touch panel display 150 includes a display (display device) 152 and an input device 154. The touch panel display 150 is provided, for example, on the top plate 120 of the settlement device 10. In one example, the touch panel display 150 is provided adjacent to the checkout table 110, the second camera 160, and the microphone 170. The display 152 displays the settlement screen 300 (see FIG. 5), the product correction screen 400 (see FIG. 9), the product designation screen 500 (see FIG. 11), or the product designation screen 600 (see FIG. 12) to the user. .. The display 152 is, for example, a liquid crystal monitor or an OLED (Organic Light Emtting Diode) monitor. The input device 154 detects the touch operation by the user and the position where the touch operation is performed. The input device 154 includes, for example, a touch panel of a resistive film type, a capacitance type, an ultrasonic surface acoustic wave type, an infrared optical imaging type, or an electromagnetic induction type.

The second camera 160 captures the user's face. The second camera 160 is provided on the back side of the touch panel display 150, for example, on the top plate 120 of the checkout device 10 so that the face of the user who operates the touch panel display 150 can be easily photographed. The second camera 160 includes, for example, a CCD sensor or a CMOS sensor.

The microphone 170 detects voice and converts it into an electric signal. The voice includes the user's utterance. The microphone 170 includes, for example, a directional microphone array. The microphone 170 is, for example, the back side of the touch panel display 150 of the top plate 120 of the checkout device 10 so as to easily detect the utterance of the user who operates the touch panel display 150, and is the checkout table 110 and the second. It is provided between the cameras 160.

The voice recognition unit 210 executes voice recognition for the detected voice and generates a character string corresponding to the user's utterance. In one example, the voice recognition unit 210 includes a processor that executes voice recognition. The processor is, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array).

The product image recognition unit 220 recognizes (or identifies) the photographed product from the photographed image of the product. In one example, the product image recognition unit 220 includes a processor. The processor is, for example, a CPU, GPU, ASIC, or FPGA. The processor recognizes (or or recognizes) the captured product based on the captured image and the trained data using a supervised learning technique such as a linear classifier, SVM (Support Vector Machine), or neural network. Identify). Here, the learned data is data obtained by learning in advance the relationship between the photographed image of the product and the information that identifies the product.

The face image recognition unit 230 recognizes (or identifies) the photographed user from the photographed image or moving image of the user's face. In one example, the product image recognition unit 220 includes a processor. The processor is, for example, a CPU, GPU, ASIC, or FPGA. The processor uses a supervised learning technique such as a linear classifier, SVM, or neural network to recognize (or identify) the photographed user based on the photographed image of the face and the trained data. Here, the trained data is data obtained by pre-learning the relationship between the photographed image of the user's face and the information that identifies the user.

The settlement processing unit 240 setstles the price to the recognized person. In one example, the settlement processing unit 240 charges the user's account at the bank or credit card company for the price calculated by the processor 280, for example, the network interface for connecting to the server of the bank or the credit card. It comprises a processor for doing so.

In one example, the checkout device 10 further includes, for example, a product recognition device 250.

The product recognition device 250 identifies an identifier that identifies the product from the input barcode. The product recognition device 250 is, for example, a barcode reader. The barcode is a one-dimensional barcode or a two-dimensional barcode. The one-dimensional barcode is, for example, JAN, EAN, UPC, CODE39, CODE128, NW-7, or ITF. The two-dimensional barcode is, for example, a QR code (registered trademark).

The memory 260 or the storage 270 stores information indicating a product (product information). The product information includes, for example, an image showing a product handled by the store, a keyword related to the product, and a price of the product. Keywords related to products include, for example, product names. Here, the product name is a name that identifies the product, and is, for example, a trademark, an abbreviation for a trademark, a registered trademark, or an abbreviation for a registered trademark.

In one example, the keyword related to the product further includes at least one of the manufacturer (manufacturer) name, the brand name, the genre name, and the series name of the product handled by the store. The manufacturer name, brand name, genre name, and series name may be abbreviated words or broken words (such as shortening iced coffee to Aiko).

Here, the brand name is a name used to identify a specific product group, and is, for example, a trade name, an abbreviation for a trade name, a trademark, an abbreviation for a trademark, a registered trademark, or an abbreviation for a registered trademark. Further, the genre name (or category name) includes a common name or an idiomatic name indicating a product, and the genre (or category) is a range of products indicated by the genre name (or category name). .. The genre names are, for example, "rice balls", "chocolate", and "fried noodles". The series name is a name that comprehensively identifies products having variations. The series name is, for example, the comprehensive name "AAA ramen" of the product having the variations "AAA ramen soy sauce taste" and "AAA ramen miso taste", and the product having the variations "BBB snack salty taste" and "BBB snack curry taste". The comprehensive name is "BBB Snack".

In one example, the memory 260 or storage 270 comprises a database for looking up product information based on keywords or combinations thereof relating to the product. Further, in one example, the memory 260 or the storage 270 includes a database for looking up product information based on an identifier specified by the product recognition device 250.

The processor 280 controls each component of the settlement device 10 and is described later with reference to FIGS. 3, 4, 6, 6, 7, 8, 10, 13, 14, and 15. To execute. The processor 280 is, for example, a CPU, GPU, ASIC, or FPGA.

[Operation of payment device 10]
FIG. 3 is a flowchart showing an example of the operation of the settlement device 10 according to the present disclosure. All of the following buttons and the like can be pressed by voice recognition when the user utters "correction" or the like.

In step S102, the settlement device 10 executes the product image recognition process. The details of the product image recognition process will be described later with reference to FIG.

In step S104, the input device 154 accepts the button press. The button is, for example, a button displayed on the touch panel display 150.

In step S106, processor 280 determines the type of button pressed. The type of button to be determined is, for example, a modify button, an add button, a delete button, or a checkout button. Details of these buttons will be described later with reference to FIG.

In step S108, the processor 280 executes the product correction process. Details of the product correction process will be described later with reference to FIG.

In step S110, the processor 280 executes the settlement process of the product purchased by the user. Details of the settlement process will be described later with reference to FIG. After that, the settlement device 10 ends the flow.

[Product image recognition processing]
FIG. 4 is a flowchart showing an example of the operation of the settlement device 10 in step S102 (product image recognition processing) of FIG.

In step S202, the first camera 130 takes an image of the product. Here, the number of products to be photographed may be any number of one or more. The first camera 130 captures an image of the product, for example, in response to detecting that the user has placed the product on the checkout table 110. The first camera 130 determines, for example, whether or not the image to be captured is a still image, and when the image to be captured is a still image other than the still image of the checkout table 110, the user puts the product on the checkout table 110. Is detected.

In step S204, the product image recognition unit 220 recognizes the product from the captured image. If necessary, the settlement device 10 may repeatedly execute steps S202 and S204.

In step S206, processor 280 generates a purchased merchandise list that includes all recognized merchandise. The processor 280 stores the generated purchase product list in the memory 260 or the storage 270.

In step S208, the processor 280 displays a list of purchased products stored in the memory 260 or the storage 270. After that, the settlement device 10 ends the flow.

FIG. 5 is a diagram showing an example of the settlement screen 300. The settlement screen 300 is displayed on, for example, the display 152 (see FIGS. 1 and 2) of the touch panel display 150.

The settlement screen 300 includes, for example, a purchased product list display 310, an additional button 340, and a settlement button 350.

The purchased product list display 310 is a display of the purchased product list generated in step S206. The purchased product list display 310 includes display (purchased product display) 320a and 320b of products included in the purchased product list, and a scroll bar 330.

The purchased product display 320a and 320b include a product image display 321, a product name display 323, a price display 325, a correction button 327, and a delete button 329, respectively. The processor 280 displays, for example, the product image display 321 and the product name display 323 and the price display 325 on the settlement screen 300 based on the product information read from the memory 260 or the storage 270.

The scroll bar 330 is a user interface for allowing the user to specify the purchased product displays 320a and 320b to be displayed on the display 152 when the number of purchased product displays 320a and 320b included in the purchased product list display 310 is large.

The correction button 327, the add button 340, and the delete button 329 are buttons for the user to instruct the product correction process. Here, the product correction process is a process executed when the user corrects the purchased product list display 310. The product correction process includes a product correction process, a product addition process, and a product deletion process. The product correction process will be described later with reference to FIG. 6 and the like.

The settlement button 350 is a button for the user to instruct the product settlement process. Details of the settlement process will be described later with reference to FIG.

[Product correction process]
When the product actually purchased by the user does not match the purchased product display 320a and 320b included in the purchased product list display 310, the user presses the correction button 327, the add button 340, or the delete button 329. The product correction process executed in response to the pressing will be described below.

FIG. 6 is a flowchart showing an example of the operation of the settlement device 10 in step S108 (product correction process) of FIG.

In step S302, the processor 280 determines the type of button pressed on the checkout screen 300 (see FIG. 5). The type of button to be determined is, for example, a correction button 327, an add button 340, or a delete button 329.

When the type of the pressed button is the add button 340 (step S302: addition), in step S304, the processor 280 executes the product designation process for determining the designated product in order to execute the product addition process. To do. The details of the product designation process will be described later with reference to FIG. 7.

In step S306, the processor 280 adds the determined designated product to the list of purchased products stored in the memory 260 or the storage 270.

When the type of the pressed button is the correction button 327 (step S302: correction), in step S308, the processor 280 executes the product designation process for determining the designated product in order to execute the product correction process. To do. The details of the product designation process will be described later with reference to FIG. 7.

In step S310, the processor 280 replaces the target product of the purchased product list stored in the memory 260 or the storage 270 with the determined designated product. Here, the target product is a product corresponding to the pressed correction button 327 (for example, the product "AAA chocolate" or the product "BBB canned coffee" shown in FIG. 5).

When the type of the pressed button is the delete button 329 (step S302: delete), in step S312, the processor 280 is in the purchased product list stored in the memory 260 or the storage 270 in order to execute the product deletion process. Delete the target product. Here, the target product is a product corresponding to the pressed delete button 329 (for example, the product "AAA chocolate" or the product "BBB canned coffee" shown in FIG. 5).

In step S314, the processor 280 updates the display of the purchased product list on the settlement screen 300. After that, the processor 280 ends the product correction process.

[Product designation processing]
FIG. 7 is a flowchart showing an example of the operation of the settlement device 10 in steps S304 and S308 (product designation processing) of FIG.

In step S402, the processor 280 determines whether the microphone 170 has detected the voice input.

When the microphone 170 detects the voice input (step S402: Yes), in step S404, the settlement device 10 executes the voice recognition product designation process. The details of the voice recognition product designation process will be described later with reference to FIG.

In step S406, the processor 280 determines whether or not the product designation has been completed, that is, in step S404, the designated product has been confirmed. When the product designation is not completed (step S406: No), the settlement device 10 shifts the process to step S402. On the other hand, when the product designation is completed (step S406: Yes), the processor 280 ends the flow.

When the microphone 170 does not detect the voice input (step S402: No), in step S408, the processor 280 determines whether or not the input device 154 has detected the touch panel input.

When the input device 154 does not detect the touch panel input (step S408: Yes), the processor 280 executes the touch panel product designation process in step S410. The details of the touch panel product designation process will be described later with reference to FIG. After that, the processor 280 ends the flow.

When the input device 154 does not detect the touch panel input (step S408: No), in step S412, the processor 280 determines whether or not the product recognition device 250 has detected the bar code input.

When the product recognition device 250 detects the input of the barcode (step S412: Yes), in step S414, the settlement device 10 executes the barcode product designation process. Details of the barcode product designation process will be described later with reference to FIG. After that, the processor 280 ends the flow.

In the embodiment in which the settlement device 10 does not include the product recognition device 250, steps S412 and S414 can be omitted. In this case, if the input device 154 does not detect the touch panel input (step S408: No), the processor 280 shifts the process to step S402.

[Voice recognition product designation processing]
FIG. 8 is a flowchart showing an example of the operation of the settlement device 10 in step S404 (voice recognition product designation process) of FIG.

In step S502, the voice recognition unit 210 executes voice recognition for the detected voice.

In step S504, the processor 280 determines whether or not the recognition target keyword is included in the voice recognition result. For example, the processor 280 searches the voice recognition result from the database in which the keyword to be recognized is registered, which is stored in advance in the storage 270.

When the recognition target keyword is included in the voice recognition result (step S504: Yes), in step S506, the processor 280 executes the product narrowing process. Details of the product narrowing process will be described later with reference to FIG.

In step S508, the display 152 displays a list of products extracted by the product narrowing process. An example of the display will be described later with reference to FIGS. 11 and 12.

In step S510, the input device 154 accepts the designation (selection) of the product. After that, the settlement device 10 ends the flow.

On the other hand, when the recognition target keyword is not included in the voice recognition result (step S504: No), the settlement device 10 ends the flow.

FIG. 9 is a diagram showing an example of the product correction screen 400. The product correction screen 400 is displayed, for example, on the display 152 (see FIGS. 1 and 2) of the touch panel display 150.

The product correction screen 400 includes a guidance display 410 and a genre selection screen 420. The genre selection screen 420 includes genre buttons 422a, 422b, and 422c. The genre name is displayed on the genre buttons 422a, 422b, and 422c, and is a button for the user to specify the genre by pressing the button.

The guidance display 410 is a display indicating to the user that the product or genre can be specified by voice input instead of pressing the genre buttons 422a, 422b, 422c. The guidance display 410 may include a guidance balloon display 412 for showing an example of an utterance of a genre name or a combination of a genre name and a genre name, such as "rice ball", "DDD (DDD maker) chocolate", and "yakisoba". Good. The content of the guidance display 410 is generated based on, for example, the genre name of the genre buttons 422a, 422b, 422c.

FIG. 10 is a flowchart showing an example of the operation of the settlement device 10 in step S506 (product narrowing process) of FIG.

In step S602, the processor 280 determines whether or not the keyword corresponding to the manufacturer name is included in the voice recognition result.

When the keyword corresponding to the manufacturer name is included in the voice recognition result (step S602: Yes), in step S604, the processor 280 uses the keyword corresponding to the genre name (for example, "chocolate (chocolate)", "rice ball", "onigiri", " It is determined whether or not "canned coffee") is included in the voice recognition result.

When the keyword corresponding to the genre name is included in the voice recognition result (step S604: Yes), in step S606, the processor 280 extracts the product corresponding to the keyword. The extracted products are candidates for products that can be included in the purchased product list after the product correction process. After that, the processor 280 ends the flow. If the keyword corresponding to the genre name is not included in the voice recognition result (step S604: No), the processor 280 ends the flow.

On the other hand, when the keyword corresponding to the manufacturer name is not included in the voice recognition result (step S602: No), in step S608, the processor 280 determines whether or not the keyword corresponding to the brand name is included in the voice recognition result. judge.

When the keyword corresponding to the brand name is included in the voice recognition result (step S608: Yes), the processor 280 shifts the process to step S606. On the other hand, when the keyword corresponding to the brand name is not included in the voice recognition result (step S608: No), in step S610, the processor 280 uses the keyword corresponding to the genre name (for example, "chocolate (chocolate)", "rice ball". , "Canned coffee")) is determined whether or not it is included in the voice recognition result.

When the keyword corresponding to the genre name is included (step S610: Yes), the processor 280 shifts the process to step S606. On the other hand, when the keyword corresponding to the genre name is not included in the voice recognition result (step S610: No), in step S612, the processor 280 uses the keyword corresponding to the product name (for example, "EEE rice ball", "FFF rice ball"". ) Is included in the voice recognition result.

When the keyword corresponding to the product name is included (step S612: Yes), the processor 280 shifts the process to step S606. On the other hand, when the keyword corresponding to the product name is not included in the voice recognition result (step S612: No), the processor 280 ends the flow.

FIG. 11 is a diagram showing an example of the product designation screen 500. The product designation screen 500 is displayed, for example, on the display 152 (see FIGS. 1 and 2) of the touch panel display 150. In one example, the product designation screen 500 is displayed when the number of products extracted by the product narrowing process is less than or equal to a predetermined number in step S506 of FIG.

The product designation screen 500 includes displays 510a and 510b of the extracted products. The product display 510a and 510b include a product image display 512, a product name display 514, a price display 516, and a designation button 518, respectively. The processor 280 causes the product designation screen 500 to display the product image display 512, the product name display 514, and the price display 516, for example, based on the product information read from the memory 260 or the storage 270. The designation button 518 is a button for the user to instruct the designation of the product.

FIG. 12 is a diagram showing an example of the product designation screen 600. The product designation screen 600 is displayed, for example, on the display 152 (see FIGS. 1 and 2) of the touch panel display 150. In one example, the product designation screen 500 is displayed when the number of products extracted by the product narrowing process is larger than a predetermined number in step S506 of FIG.

The product designation screen 600 includes a guidance display 610 and a list display screen 620 of the extracted products. The list display screen 620 of the extracted products includes displays 622a and 622b of the extracted products. The product display 622a and 622b include a product image display 632, a product name display 634, a price display 636, and a designation button 638, respectively. The processor 280 displays, for example, the product image display 632, the product name display 634, and the price display on the product designation screen 600 based on the product information read from the memory 260 or the storage 270. 636 and are displayed. The designation button 638 is a button for the user to instruct the designation of the product.

The guidance display 610 is a display indicating to the user that the product can be further narrowed down by voice input instead of pressing the designated button 638. For example, when the user utters "sweets", the guidance display 610 may include an example of an utterance for narrowing down "sweets", for example, a guidance balloon display 612 for indicating "chocolate, gum, candy". Good. The content of the guidance display 610 is generated based on, for example, the product name included in the product list display screen 620.

In one example, the processor 280 changes the order of the extracted product displays 622a, 622b according to the user or the attributes of the user. The user's attributes are, for example, the user's estimated age or the user's estimated gender. For example, the second camera 160 takes a picture of the user, and the face image recognition unit 230 recognizes the user or the user's attributes based on the user's face image or face moving image. The processor 280 changes the order of the extracted product displays 622a and 622b based on the result of recognizing the user or the user's attributes. For example, the memory 260 or storage 270 stores products purchased by the user in the past, and the processor 280 displays the extracted products so that the products purchased by the user in the past take precedence over other products. 622a, 622b Change the order of.

[Touch panel product designation processing]
FIG. 13 is a flowchart showing an example of the operation of the settlement device 10 in step S410 (touch panel product designation process) of FIG.

In step S702, the display 152 displays a list of genres. If the number of genres is too large to display at one time, the display 152 may display, for example, scroll bars, scroll buttons, or tabs along with some of the genres.

In step S704, the input device 154 accepts a genre designation from the user.

In step S706, the display 152 displays a list of products belonging to the genre designated by the user. When the number of products is large and cannot be displayed at one time, the display 152 may display, for example, a scroll bar, a scroll button, or a tab together with a part of the products.

In step S708, the input device 154 receives the designation of the product from the user. After that, the settlement device 10 ends the flow.

[Barcode product specification processing]
FIG. 14 is a flowchart showing an example of the operation of the settlement device 10 in step S414 (bar code product designation processing) of FIG.

In step S802, the product recognition device 250 identifies the product identifier from the input barcode.

In step S804, processor 280 specifies the goods specified by the identifier. After that, the settlement device 10 ends the flow.

[Settlement process]
FIG. 15 is a flowchart showing an example of the operation of the settlement device 10 in step S110 (settlement processing) of FIG.

In step S902, the second camera 160 captures the user's face image or face moving image.

In step S904, the user is recognized from the face image or the face moving image. Step S902 and step S904 may be repeated until the user can be recognized.

In step S906, the settlement processing unit 240 charges the recognized user for the price of the product. After that, the settlement device 10 ends the flow.

According to the present disclosure, the user can register the product in the settlement device 10 and settle the purchase price without using a barcode reader. Therefore, the customer can eliminate or reduce the burden of using the barcode reader by using the settlement device 10, and can shorten the time required for settlement.

Further, according to the present disclosure, when there are many products displayed on the touch panel display 150, the user (customer) can narrow down the products by utterance. Therefore, by using the settlement device 10, the customer can reduce the burden of selecting the image of the product using the touch panel display, and can shorten the time required for settlement. For example, even if the store handles a large number of products and there are hundreds of product candidates, a list of all products or all product categories is displayed on the display, and customers can select the desired product from among them. It is possible to avoid situations where the operation becomes complicated, such as selecting. In addition, the customer can shorten the time and effort required to select the target product by using the payment device 10.

Further, according to the present disclosure, the customer does not have to complete the registration of the purchased product based only on the utterance, and the number of steps for narrowing down the product is often small. Therefore, the customer can quickly complete the settlement process by using the settlement device 10.

Further, according to the present disclosure, in the product narrowing process, the products are narrowed down based on the combination of keywords related to the products, and the user (customer) specifies the products from the narrowed down products via the touch panel display 150. (Selection) Yes. Therefore, since it is difficult to uniquely understand and speak the exact name of the product name, the settlement device 10 narrows down the products based on the utterance even when the customer speaks a keyword different from the product name. This makes it easier for customers to specify (select) products. Therefore, even if the utterance content of the customer is ambiguous, it is often allowed, and even if there are no store employees, the customer can easily operate the checkout device 10.

Further, according to the present disclosure, the product recognition device 250 (for example, a barcode reader) is not an essential component. Therefore, by omitting the product recognition device 250, it is possible to save the installation space of the payment device 10.

(Other embodiments)
In the present disclosure, for example, the settlement screen 300 shown in FIG. 5 includes a correction button 327, a delete button 329, an add button 340, and a settlement button 350. Further, for example, the product designation screen 500 shown in FIG. 11 and the product designation screen 600 shown in FIG. 12 include a designation button 638. However, on the settlement screen 300, the product correction screen 400 shown in FIG. 9, the product designation screen 500, and the product designation screen 600, the correction button 327, the delete button 329, the addition button 340, the settlement button 350, and the designation button 638 Buttons different from either may be displayed.

In one example, another button for instructing a change in the control flow of the checkout device 10 may be displayed. For example, the settlement screen 300 may display a voice recognition redo button instructing the settlement device 10 to re-execute the voice recognition product designation process in steps S404 and S406 shown in FIG. Further, for example, on the product correction screen 400, the product designation screen 500, or the product designation screen 600, the settlement device 10 is instructed to re-execute the product narrowing process of step S506 shown in FIG. Buttons may be displayed. For example, the memory 260 or the storage 270 stores the history of the operation state of the settlement device 10 such as the purchased product list in each step and the result of product narrowing down, and the voice recognition redo button or the product narrowing redo button is pressed in response to the pressing. , The operation state may be updated retroactively.

In the present disclosure, for example, in the flow chart of the product narrowing process shown in FIG. 10, a manufacturer name, a brand name, a genre name, and a product name are described as keywords related to the product. However, an embodiment using a series name as a keyword related to a product is also conceivable. For example, in FIG. 10, a step for determining whether or not the processor 280 includes a keyword corresponding to the series name may be inserted as step S611 between steps S610 and S612.

In the present disclosure, as shown in FIG. 3, the settlement process of step S110 is executed after the settlement button is pressed. However, among the settlement processes of step S110, steps S902 and S904 shown in FIG. 15 may be executed before the settlement button is pressed.

In the present disclosure, after recognizing the image of the product in step S204 as shown in FIG. 4, the purchased product list is generated in step S206. Alternatively, an embodiment in which steps S202 and S204 are omitted and the processor 280 generates a purchased product list including all products in the store in step S206 can be considered. In this case, the first camera 130 and the product image recognition unit 220 may be omitted from the configuration of the settlement device 10.

In the present disclosure, as shown in FIG. 1, the microphone 170 is provided on the back side of the touch panel display 150 of the top plate 120 of the checkout device 10 and between the checkout table 110 and the second camera 160. Be done. Instead of this, an embodiment in which the microphone 170 is provided on the handset connected to the checkout device 10 can be considered. By providing the microphone 170 in the handset, it becomes easier for the customer to speak so that the content of his / her utterance cannot be heard by others.

In the above-described embodiment, the notation "... part" used for each component is "... circuitry", "... device", "... unit", or "... unit". It may be replaced with another notation such as "... module".

Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can come up with various modifications or modifications within the scope of the claims, which naturally belong to the technical scope of the present disclosure. Understood. In addition, each component in the above embodiment may be arbitrarily combined as long as the purpose of disclosure is not deviated.

The present disclosure can be realized by software, hardware, or software linked with hardware. Each functional block used in the description of the above embodiment is partially or wholly realized as an LSI which is an integrated circuit, and each process described in the above embodiment is partially or wholly. It may be controlled by one LSI or a combination of LSIs. The LSI may be composed of individual chips, or may be composed of one chip so as to include a part or all of functional blocks. The LSI may include data input and output. LSIs may be referred to as ICs, system LSIs, super LSIs, and ultra LSIs depending on the degree of integration. The method of making an integrated circuit is not limited to LSI, and may be realized by a dedicated circuit, a general-purpose processor, or a dedicated processor. Further, an FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connection and setting of the circuit cells inside the LSI may be used. The present disclosure may be realized as digital processing or analog processing. Furthermore, if an integrated circuit technology that replaces an LSI appears due to advances in semiconductor technology or another technology derived from it, it is natural that the functional blocks may be integrated using that technology. There is a possibility of applying biotechnology.

The present disclosure can be implemented in all types of devices, devices and systems (collectively referred to as communication devices) having a communication function. Non-limiting examples of communication devices include telephones (mobile phones, smartphones, etc.), tablets, personal computers (PCs) (laptops, desktops, notebooks, etc.), cameras (digital stills / video cameras, etc.). ), Digital players (digital audio / video players, etc.), wearable devices (wearable cameras, smart watches, tracking devices, etc.), game consoles, digital book readers, telehealth telemedicines (remote health) Care / medicine prescription) devices, vehicles with communication functions or mobile transportation (automobiles, airplanes, ships, etc.), and combinations of the above-mentioned various devices can be mentioned.

Communication devices are not limited to those that are portable or mobile, but are not portable or fixed, any type of device, device, system, such as a smart home device (home appliances, lighting equipment, smart meters or It also includes measuring instruments, control panels, etc.), vending machines, and any other "Things" that can exist on the IoT (Internet of Things) network.

Communication includes data communication by a combination of these, in addition to data communication by a cellular system, a wireless LAN system, a communication satellite system, and the like. Communication devices also include devices such as controllers and sensors that are connected or connected to communication devices that perform the communication functions described in the present disclosure. For example, it includes controllers and sensors that generate control and data signals used by communication devices that perform the communication functions of the communication device.

Communication devices also include infrastructure equipment that communicates with or controls these non-limiting devices, such as base stations, access points, and any other device, device, or system. ..

The settlement device according to the present disclosure displays a first camera that photographs a product placed on a table, a product image recognition circuit that recognizes the product based on an image of the product, and a recognition result of the product. Based on a display device, an input device that receives input of a correction instruction for the recognition result of the product, a microphone that detects voice, a voice recognition circuit that recognizes the voice, and keywords included in the voice recognition result. A control circuit for extracting a candidate for the product is provided, the display device displays the candidate, the input device accepts the selection of the candidate from the user, and the control circuit is based on the selection. , Correct the recognition result of the product.

In the settlement device according to the present disclosure, the keyword is one or a combination of two or more of a manufacturer name, a product name, a brand name, a genre name, and a series name.

In the settlement device according to the present disclosure, the first camera is provided facing the table.

In the settlement device according to the present disclosure, the microphone includes a microphone array having directivity.

In the settlement device according to the present disclosure, a second camera that captures a user's face, a face image recognition circuit that recognizes the user based on an image captured by the second camera, and the recognized user. On the other hand, a settlement processing circuit for charging the price of the product is provided.

In the settlement device according to the present disclosure, the microphone is provided between the first camera and the second camera.

In the settlement device according to the present disclosure, the display device is provided adjacent to the table, the second camera, and the microphone.

In the settlement device according to the present disclosure, the control circuit changes the order of the candidates displayed by the display device according to the user or the attribute of the user.

In the settlement device according to the present disclosure, the input device includes a touch panel provided on the display surface of the display device.

In the settlement method according to the present disclosure, the camera photographs the product placed on the table, the product image recognition circuit recognizes the product based on the image of the product, and the display device recognizes the product. The result is displayed, the input device accepts the input of the correction instruction of the recognition result of the product, the microphone detects the voice, the voice recognition circuit recognizes the voice, and the control circuit recognizes the voice. Candidates for the product are extracted based on the keywords included in the result, the display device displays the candidates, the input device accepts the selection of the candidates from the user, and the control circuit is based on the selection. Then, the recognition result of the product is corrected.

The present disclosure is useful for a settlement system such as a supermarket or a convenience store where the purchase price of a plurality of products is collectively settled.

10 Settlement device 110 Settlement table 120 Top plate 130 First camera 140 Ceiling part 150 Touch panel display 160 Second camera 170 Microphone 210 Voice recognition unit 220 Product image recognition unit 230 Face image recognition unit 240 Payment processing unit 250 Product recognition device 260 Memory 270 storage 280 processor

Claims (10)

The first camera that shoots the products placed on the table,
A product image recognition circuit that recognizes the product based on the image of the product,
A display device that displays the recognition result of the product and
An input device that accepts input of correction instructions for the recognition result of the product,
With a microphone that detects voice,
A voice recognition circuit that recognizes the voice and
A control circuit that extracts the product candidates based on the keywords included in the voice recognition result, and
With
The display device displays the candidate and displays the candidate.
The input device accepts the selection of the candidate from the user and receives
The control circuit corrects the recognition result of the product based on the selection.
Settlement device. The settlement device according to claim 1, wherein the keyword is one or a combination of one or more of a manufacturer name, a product name, a brand name, a genre name, and a series name. The settlement device according to claim 1 or 2, wherein the first camera is provided so as to face the table. The settlement device according to any one of claims 1 to 3, wherein the microphone includes a microphone array having directivity. A second camera that captures the user's face,
A face image recognition circuit that recognizes the user based on the image taken by the second camera, and
A settlement processing circuit that charges the recognized user for the price of the product, and
To prepare
The settlement device according to any one of claims 1 to 4. The settlement device according to claim 5, wherein the microphone is provided between the first camera and the second camera. The payment device according to claim 5 or 6, wherein the display device is provided adjacent to the table, the second camera, and the microphone. The settlement device according to any one of claims 5 to 7, wherein the control circuit changes the order of the candidates displayed by the display device according to the user or the attribute of the user. The settlement device according to any one of claims 1 to 8, wherein the input device includes a touch panel provided on the display surface of the display device. The camera shoots the product placed on the table,
The product image recognition circuit recognizes the product based on the image of the product.
The display device displays the recognition result of the product,
The input device accepts the input of the correction instruction of the recognition result of the product, and receives the input.
The microphone detects the voice and
The voice recognition circuit recognizes the voice and
The control circuit extracts the product candidates based on the keywords included in the voice recognition result.
The display device displays the candidate,
The input device accepts the selection of the candidate from the user and receives
The control circuit corrects the recognition result of the product based on the selection.
Settlement method.

Images