JP7477664B2 - Product data processing system and product data processing method - Google Patents

Description

FIELD An embodiment of the present invention relates to a product data processing system and a product data processing method .

In retail stores such as supermarkets, checkout systems have been introduced that allow customers to perform part of the accounting process themselves in order to improve the efficiency of accounting processes using POS (Point Of Sales). For example, self-checkout systems equipped with self-checkout registers (self-checkout terminals) have been introduced. At the self-checkout register, the customer operates the register to read the product code (usually a barcode) of the product, registers the product, and executes a registration process that generates payment information for the payment process, and a payment process that makes payment based on the payment information generated by the registration process.

In recent years, personal checkout systems have been developed that register products that customers intend to purchase while they shop around the store. With this system, it is necessary that the products that are placed in the cart or basket as intended purchases match the registered products.

JP 2016-532932 A

An object of the present invention is to provide a product data processing system and a product data processing method that enable a customer to reliably register products that he or she intends to purchase through an operation.

According to an embodiment, the product data processing system has a display unit, display means, storage means, first output means , recognition means, and second output means . The display means displays the product information acquired by the product information acquisition means on the display unit. The storage means stores the product information acquired by the product information acquisition means in the storage unit. The first output means outputs a first guide to the display unit, when a pre-set predetermined time has elapsed since the product information was stored in the storage unit, to encourage the customer to store the product in a storage unit for storing the product that the customer plans to purchase. The recognition means recognizes the product stored in the storage unit. The second output means outputs a second guide to the display unit, when product information corresponding to the product recognized by the recognition means is not stored in the storage unit, to encourage the customer to acquire the product information using the product information acquisition means.

1 is a system configuration diagram illustrating an overall configuration of a system according to an embodiment of the present invention. FIG. 1 is a block diagram showing a configuration of an electronic device according to an embodiment. 1A and 1B are diagrams showing examples of use of a clip terminal according to an embodiment of the present invention. FIG. 2 is a diagram showing the appearance of the clip terminal according to the embodiment. 5 is a flowchart illustrating the operation of the electronic device according to the embodiment. 5 is a flowchart for explaining the operation of a clip terminal in the present embodiment. 5 is a flowchart for explaining the operation of a clip terminal in the present embodiment. FIG. 4 is a diagram showing an example of a product checklist in the embodiment. FIG. 4 is a diagram showing an example of a message according to the embodiment. 4A and 4B are diagrams showing examples of images captured by a camera in the present embodiment. FIG. 11 is a diagram showing an example of an object corresponding to a commodity detected from the image shown in FIG. 10 . 10 is a flowchart showing a registered position determination process for determining whether confirmation is necessary based on a registered position in the present embodiment. FIG. 1 is a diagram showing an example of the arrangement of product shelves and the positions of carts in a store.

Hereinafter, the present embodiment will be described with reference to the drawings.
Fig. 1 is a system configuration diagram showing the overall configuration of the system of this embodiment. In the system shown in Fig. 1, a system 10 installed at a headquarters and store systems 12 installed at each of a plurality of stores are connected via a network 14 including a dedicated line or the Internet.

The systems 10 installed at the headquarters include, for example, a headquarters system 10A, a customer system 10B, a payment system 10C, and a sales promotion system 10D, which are interconnected via a network 10E such as a LAN (Local Area Network).

The headquarters system 10A executes processes to manage the store systems installed in each store. The customer system 10B manages customers who use the store system to purchase products, etc. The payment system 10C executes processes (personal authentication, credit card payment, etc.) for payment processing using credit cards, etc. by customers in the store system. The sales promotion system 10D manages the provision of information such as product advertisements and in-store guides for sales promotion to customers (electronic devices used by customers) through the store system.

The store system 12 installed in the store is realized as, for example, a POS (Point Of Sales) system. The store system 12 includes, for example, a store server 20A, a clip terminal management server 20B, multiple accounting machines 20C, and multiple POS terminals 20D, which are interconnected via a network 20E such as a LAN.

The store server 20A controls the entire store system 12. The store server 20A stores data related to products sold in the store, such as product codes, product names, unit prices, weights, product positions (positions on shelves) and product images for each product. In the registration process, the data of the product corresponding to the product code read by scanning the product is received from the store server 20A and stored, and payment information (including amount, product, number, etc.) for payment processing can be generated. The clip terminal management server 20B manages the electronic device 50 (clip terminal described later) used to check whether the product (purchased product) registered by the customer using the electronic device 30 as a purchase request matches the product placed in the cart 60 or basket as a purchase request. The accounting machine 20C executes payment processing for the product registered by the customer using the electronic device 30. The POS terminal 20D executes registration processing for the customer's purchased product by the store clerk's operation, payment processing for the product registered by the registration processing, etc.

In addition, multiple access points (base stations) 20F are connected to the network 20E wired within the store. The access points 20F communicate wirelessly with various electronic devices used within the store, for example, via a wireless LAN. The electronic devices used within the store include, for example, electronic devices 30 used by customers, electronic devices 40 used by store attendants, and electronic devices 50 (clip terminals 50, described below) used while attached to, for example, a cart 60.

The electronic devices 30 and 40 may be, for example, a smartphone or a tablet personal computer (PC). The electronic device 30, in cooperation with the electronic device 50 (clip terminal), executes a registration process for products to be purchased and an individual check process for checking whether the registered products (purchased products) match the products placed in the cart 60 or basket as products to be purchased, based on an application program (including a check program) for using the services provided by the store system 12. The electronic device 40 executes a confirmation process, based on a confirmation program for confirming the registration process executed by the electronic device 30, to allow a store clerk (attendant) to easily confirm whether the product information registered by the registration process matches the products placed in the cart 60.

The electronic device 30 and the electronic device 50 (clip terminal 50) may communicate via the clip terminal management server 20B (network 20E) or directly by short-range wireless communication. Furthermore, they may be connected to each other by a communication cable to perform wired communication.

Figure 2 is a block diagram showing the configuration of electronic devices 30, 50 in this embodiment. The electronic device 30 shown in Figure 2 will be described as being realized by, for example, a smartphone. The electronic device 50 will be configured as a dedicated clip terminal 50 for executing individual check processing in the store system 12. Below, it will be described as a clip terminal 50. Note that the clip terminal 50 can also be realized by installing a check program on a general-purpose device (computer) equipped with a camera, such as a smartphone.

The electronic device 30 includes a processor 30A, a memory 30B, a storage device 30C, a touch panel 30D, a camera 30E, a communication module 30F, and a battery 30G.
The processor 30A is a circuit that controls each part to realize various processes of the electronic device 30 by executing a program recorded in the memory 30B.

Memory 30B records programs executed by processor 30A, as well as data used by processor 30A to execute various processes. Programs recorded in memory 30B include, for example, a registration processing program that executes registration processing for products to be purchased, and a check program that executes individual check processing to check whether the registered product (purchased product) matches the product placed in cart 60 or basket as to be purchased. Data stored in memory 30B includes payment information generated based on the product code read in the registration processing, product information on the product registered by the registration processing, etc.

The storage device 30C is realized by a hard disk drive (HDD) device or a solid state drive (SSD) device, and records data used by the processor 30A in performing various processes, or data generated by the processes in the processor 30A.

The touch panel 30D is a device on which a customer performs an input operation, and includes a display device and a touch sensor. The touch panel 30D displays any screen, such as a GUI (Graphical User Interface) screen, associated with the processing of the processor 30A.
Camera 30E captures images under the control of processor 30A.

The communication module 30F controls communication with external electronic devices under the control of the processor 30A. The communication module 30F can control communication such as wireless communication with the access point 20F and short-range wireless communication with other electronic devices such as the clip terminal 50.

The clip terminal 50 is provided with a processor 50A, a memory 50B, a storage device 50C, a touch panel 50D, a camera 50E, a communication module 50F, a battery 50G, an I/O interface 50H, and a reader/writer 50J.

The processor 50A is a circuit that controls each part to realize various processes as the clip terminal 50 by executing the programs recorded in the memory 50B.

The memory 50B records the programs executed by the processor 50A, as well as data used by the processor 50A to execute various processes. The programs recorded in the memory 50B include a check program that realizes an individual check process that checks whether the registered product (purchased product) matches the product placed in the cart 60 or basket as a purchase candidate. The check program includes an image processing program that realizes image processing that detects an object corresponding to the product from an image captured by the camera 50E and detects that the product has been placed in the cart 60 and that the product has been removed from the cart 60. The data stored in the memory 50B includes various data such as product information registered by the registration process of the electronic device 30, object management data that manages objects detected by image processing (see FIG. 11), and a product checklist (see FIG. 8) for checking products placed in the cart 60.

The storage device 50C is realized by a hard disk drive (HDD) or a solid state drive (SSD), and records data used by the processor 50A in performing various processes, or data generated by the processes in the processor 50A.

The touch panel 50D is a device on which a customer performs an input operation, and includes a display device and a touch sensor. The touch panel 50D displays any screen, such as a GUI (Graphical User Interface) screen, associated with the processing of the processor 50A.
Camera 50E captures images under the control of processor 50A.

The communication module 50F controls communication with external electronic devices under the control of the processor 50A. The communication module 50F can control communication such as wireless communication with the access point 20F and short-range wireless communication with other electronic devices such as the electronic device 30.

The I/O interface 50H is an interface for inputting signals from external devices, and is connected to, for example, a weight sensor 601 (a weighing scale, a pressure sensor, a weighing scale, etc.) attached to the basket bottom 60C of the cart 60. The weight sensor 601 detects the weight of a product placed (placed) on the cart 60 and outputs a detection signal.

The reader/writer 50J communicates with RF (Radio Frequency) tags attached to each product sold in the store and to product shelves in the store. The communication range of the reader/writer 50J is, for example, the range in which it can communicate with the RF tags of products placed in the cart 60, or the range in which it can communicate with RF tags attached to product shelves by moving the cart 60 around the store. The reader/writer 50J can be used to detect when a product has been placed in or removed from the cart 60, or to detect the position in the store of the cart 60 to which the clip terminal 50 is attached.

Figure 3 shows an example of how the clip terminal 50 in this embodiment is used, and Figure 4 shows the external appearance of the clip terminal 50 in this embodiment.

The clip terminal 50 is used, for example, by attaching it to the upper frame 60B of the cart 60 used in the store. As shown in FIG. 4, the clip terminal 50 is provided with a clip 60k, which can be attached and detached as desired. Therefore, it is not necessary to attach the clip terminal 50 to all carts 60 prepared in advance in the store, and the clip terminal 50 only needs to be attached as necessary to the carts 60 used by customers who use the individual check processing service using the clip terminal 50. Therefore, in order to realize the individual check processing service, large-scale investments such as remodeling the store or replacing the carts 60 are not required, and it can be realized at low cost.

In the example shown in FIG. 3, the clip terminal 50 is attached to the upper frame 60B on the handle 60A side of the cart 60. When attaching the clip terminal 50, the camera 50E on the clip terminal 50 is directed toward the basket bottom 60C, and the shooting range is adjusted so that at least all of the products placed in the cart 60 can be photographed. The shooting range of the camera 50E preferably includes the upper frame 60B, so that the areas inside and outside the basket can be clearly distinguished in the image captured by the camera 50E. Furthermore, it is desirable that the shooting range of the camera 50E be such that it is possible to capture, for example, the movement of the customer's arm (hand) so that the action of the customer putting in or taking out a product can be detected.

As shown in FIG. 4, the clip terminal 50 is attached to the upper frame 60B so that the display surface of the touch panel 50D faces the customer. This makes it easy to check the images captured by the camera 50E and information (messages, etc.) that show the check results of the individual check process using the captured images.

Next, the operation of the electronic device 30 and the clip terminal 50 (electronic device) used in the store system 12 of this embodiment will be described. FIG. 5 is a flowchart for explaining the operation of the electronic device 30 in this embodiment. FIG. 6 and FIG. 7 are flowcharts for explaining the operation of the clip terminal 50 in this embodiment.

When a customer uses the electronic device 30 to perform the registration process, the customer starts an application program (check program) for using the services provided by the store system 12. The processor 30A of the electronic device 30 then executes the registration process and individual check process described below based on the application.

First, the electronic device 30 executes a login process to connect to the store system 12 (Act 10). In the login process, the customer operates the camera 30E to read (take an image) a barcode containing a membership code unique to the member, which is attached to, for example, a membership card. The processor 30A extracts the membership code from the barcode read by the camera 30E and transmits it to the clip terminal management server 20B. The clip terminal management server 20B authenticates the member by comparing the membership code received from the electronic device 30 with the membership data managed in the customer system 10B.

When the electronic device 30 is notified by the clip terminal management server 20B that it has been authenticated as a member, it displays a selection screen for services available at the store. In addition to registration processing for planned purchases, services available at the store include displaying flyer information, displaying information about special sales items, displaying store information (maps), and using the clip terminal 50. By using the clip terminal 50, for example, if a product placed in the cart 60 has not been registered, a message is output from the clip terminal 50. This prevents the customer from forgetting to register products that they plan to purchase. In addition, since the clip terminal 50 checks the products to be purchased while the customer is shopping around the store, the burden of confirmation work by the attendant (store clerk) after the customer has completed the registration processing can be reduced.

When a customer uses a clip terminal 50, the store system 12 may provide the customer with benefits (such as points) (storing data indicating the benefits to the customer in the customer system 10B) to encourage the customer to use the clip terminal 50.

When using the clip terminal 50, the electronic device 30 and the clip terminal 50 are made to be in a state in which they can communicate with each other. For example, the clip terminal 50 is attached to the cart 60 used by the customer (or the customer is allowed to use the cart 60 with the clip terminal 50 attached in advance), and a link is set between the customer's electronic device 30 and the clip terminal 50. For example, the electronic device 30 and the clip terminal 50 are paired so that they can communicate with each other via short-distance wireless communication. Alternatively, the unique code of the clip terminal 50 is read by the electronic device 30 and transmitted from the electronic device 30 to the clip terminal management server 20B. The clip terminal management server 20B distinguishes between the electronic device 30 and the clip terminal 50 that cooperate to execute the individual check process, and enables communication between the electronic device 30 and the clip terminal 50 via the clip terminal management server 20B (network 20E). Note that other methods can also be used to connect the electronic device 30 and the clip terminal 50.

When the processor 30A of the electronic device 30 is connected to the clip terminal 50 (Act 11), it puts the product registration process into a state in which it can be executed. When the processor 50A of the clip terminal 50 is connected to the electronic device 30 (Act 40), it starts a product detection process. The product detection process is a process of detecting objects corresponding to products from an image captured by the camera 50E. The processor 50A detects objects corresponding to products, etc. from an image captured by the camera 50E using existing image processing, and stores the positions of each object. For example, when the processor 50A detects an object moving from an area corresponding to the outside of the cart (outside the cart) to the inside of the cart (inside the cart), it determines that the customer has performed an action to put in or take out a product. Based on the change in the objects present in the cart before and after detecting this action, it detects that a product has been put in or taken out. Even when a customer's action is detected, if the number of objects corresponding to products in the cart has not changed, it determines that, for example, the customer has rearranged the products in the cart. The processor 50A stores the position of the rearranged object. The processor 50A stores objects detected from images captured by the camera 50E in the product checklist.

Figure 8 is a diagram showing an example of a product checklist in this embodiment. As shown in Figure 8, the product checklist stores data (e.g., D1 to D4) that associates data on objects (detected objects) detected from an image captured by the camera 50E, data on product information (registered products) registered by a registration process in the electronic device 30, and data showing the results of checking whether the registered products match the products corresponding to the detected objects (check results "OK" or "NG").

When a customer is shopping around the store, the customer uses the camera 30E of the electronic device 30 to scan the barcode attached to the product he or she wishes to purchase, thereby registering the product (Act 12). The processor 30A detects the barcode from the image captured by the camera 30E and extracts the product code from the barcode. The processor 30A obtains product information corresponding to the product code from the store server 20A, for example, via the clip terminal management server 20B (network 20E), and displays it on the touch panel 30D. That is, the product name, price, product image, etc. of the product read by the product scan are displayed for the customer to confirm. Here, when the customer performs an instruction operation to register the product (for example, touching a product registration button displayed on the touch panel 30D) (Act 13, Yes), the processor 30A stores the product read by the product scan as a registered product to be purchased (Act 14). The processor 30A also transmits information on the registered product to the clip terminal 50 (Act 15).

On the other hand, when the clip terminal 50 receives the information of the registered product from the electronic device 30 (Act 42, Yes), it stores the information of the registered product in the product checklist (Act 43). When the processor 50A refers to the information of the registered product and detects that it is a large product, that is, when it is determined that the product does not need to (cannot) be added to the cart 60, it sets a skip flag in the product checklist (Act 45). In the example shown in FIG. 8, a skip flag is set in association with the registered product "Product C" as data D2. For the registered product "Product C" for which the skip flag is set, the processor 50A excludes it from the targets for checking whether the registered product matches the product added to the cart 60. In addition, the processor 50A transmits a notification that no product needs to be added to the electronic device 30 (Act 57).

When the processor 30A of the electronic device 30 receives a notification that no product needs to be added from the clip terminal 50 (Act 18, Yes), it displays on the touch panel 30D a message informing the customer that no product needs to be added to the cart 60, along with registered product information (product name, product image, price, etc.). The message notifying that no product needs to be added may be displayed not only on the electronic device 30, but also on the touch panel 50D of the clip terminal 50.

On the other hand, if the registered product is not a large product, the processor 50A acquires product dictionary data for identifying the registered product by image processing from the clip terminal management server 20B, for example, via the network 20E. For example, if the processor 50A does not store product dictionary data corresponding to the registered product, it outputs a transmission request for the product dictionary data corresponding to the registered product to the clip terminal management server 20B. In response to the transmission request from the clip terminal 50, the clip terminal management server 20B transmits the product dictionary data of the corresponding product to the clip terminal 50. Note that, when the electronic device 30 and the clip terminal 50 communicate via the clip terminal management server 20B, when the clip terminal management server 20B receives information on the registered product from the electronic device 30, it may transmit the product dictionary data of the corresponding product to the clip terminal 50 together with the information on the registered product. The product dictionary data may also be stored in advance in the clip terminal 50.

When the registered product is stored in the product checklist, the processor 50A starts measuring the time that has elapsed since the registered product was stored. The processor 50A determines whether the elapsed time has exceeded a preset time until it is confirmed that the product corresponding to the registered product has been placed in the cart 60 (Act 46).

When a customer purchases an item, the customer often carries out a series of actions, such as picking up the item from a shelf, scanning it using the electronic device 30, and then placing it in the cart 60. Therefore, if a preset time has passed since the item was scanned (storage of the registered item in the clip terminal 50), the customer is notified that a registered item may have been forgotten to be placed in the cart 60. If the processor 50A determines that the preset time has passed for the registered item (Act 46, Yes), it sends a request to the electronic device 30 to place the item in the cart (Act 47).

When the processor 30A of the electronic device 30 receives a product addition request from the clip terminal 50 (Act 18, Yes), it displays on the touch panel 30D a product addition request message urging the customer to add the product to the cart 60 along with registered product information (product name, product image, price, etc.) (Act 19). For example, as shown in FIG. 9(A), a message "Please add the registered product to the cart" is displayed. The product addition request message may be displayed not only on the electronic device 30, but also on the touch panel 50D of the clip terminal 50. When the product addition request message targets multiple products, the information of the multiple target products may be displayed in a list form so that the customer can clearly recognize the target products.

On the other hand, when the processor 50A of the clip terminal 50 detects through image processing of the image captured by the camera 50E that a customer has performed an action to put in or take out a product, it detects that a product has been put in or taken out based on the changes in the objects in the cart before and after this action is detected. Note that the processor 50A not only detects the action of putting in or taking out a product based on the image, but can also determine whether a product has been put in or taken out by combining this with the change in weight detected by the weight sensor 601 at the time this action is detected.

Figure 10 shows an example of an image captured by camera 50E. The example shown in Figure 10 shows a state in which three products, A, B, and E, have been placed in cart 60.

The processor 50A detects the position of the upper frame 60B of the cart 60 and the area of the basket bottom 60C from the image captured by the camera 50E. Because the clip terminal 50 is fixed to the cart 60, the areas of the upper frame 60B and the basket bottom 60C in the image do not generally change during shopping. If the processor 50A detects an object moving, for example, from the area corresponding to the outside of the cart (outside the cart) to the inside of the cart (inside the cart) based on the position of the upper frame 60B, it determines that the customer has taken an action to put in or take out a product.

In addition, the processor 50A detects objects corresponding to products, etc. from the images captured by the camera 50E using existing image processing, and stores the positions of each object as object management data.

Figure 11 shows examples of objects OJ-A, OJ-B, and OJ-E corresponding to products detected from the image shown in Figure 10. Processor 50A detects that a product has been added or removed based on the changes in the objects present in the cart before and after the customer adds or removes the product.

When an object corresponding to a product is detected from the image (Act 48, Yes), the processor 50A refers to the product checklist and determines whether a registered product with a check result of "NG" is stored. That is, it determines whether there is a product that has been registered by the electronic device 30 but has not been confirmed to have been added to the cart 60. Here, if a registered product with a check result of "NG" is stored (Act 49, Yes), the processor 50A executes a recognition process for the object detected from the image using product dictionary data corresponding to the registered product with a check result of "NG" (Act 52). The product dictionary data is created so that the product can be recognized based on features such as the barcode attached to the product, the design and characters of the package, etc.

Here, based on the results of the recognition process using the product dictionary data, it is determined whether the object detected from the image is a registered product that corresponds to the product dictionary data. For example, if the characteristics of the image represented by the object and the characteristics indicated by the product dictionary data show similarities that exceed a preset reference value, it is determined that the object is a registered product that corresponds to the product dictionary data.

If it is determined that the product is a registered product (Act 53, Yes), the processor 50A stores the object detected from the image as a detected object in association with the registered product previously stored in the product checklist, which corresponds to the product dictionary data used in the recognition process (Act 54). Note that data indicating the similarity (degree of similarity) in the recognition process may also be stored as data related to the detected object. In this way, if the product is determined to be a registered product corresponding to the product dictionary data, but the degree of similarity is low, the attendant can be prompted to check the product before the payment process.

The processor 50A also sets the check result corresponding to the pair of the registered product and the detected object to "OK" (Act 55) and stores that it has been confirmed that the registered product has been placed in the cart 60. In this case, the processor 50A transmits a product placement confirmation to the electronic device 30 (Act 56).

When the electronic device 30 receives confirmation of placing the product from the clip terminal 50 (Act 20, Yes), if a message requesting placing of the product is displayed, the electronic device 30 erases the message from the touch panel 30D (Act 28). This allows the customer to be notified that the clip terminal 50 has confirmed that the registered product has been placed in the cart 60.

When there are multiple registered products with a check result of "NG" in the product checklist, the processor 50A executes a recognition process on the objects using the product dictionary data of each registered product. Furthermore, for objects that are not determined to be registered products by the recognition process (Act 53, No), the processor 50A stores the product as an unregistered detected object with a check result of "NG" in the product checklist (Act 50). Furthermore, when an object is detected from an image (Act 48, Yes), if a registered product with a check result of "NG" is not stored (Act 49, No), the processor 50A similarly stores the product as an unregistered detected object with a check result of "NG" in the product checklist (Act 50).

Data D3 shown in Figure 8 indicates that product C has been registered in the electronic device 30, but the check result is set to "NG," indicating that it has not been confirmed that the product has been added to the cart 60. Data D4 indicates that unregistered products E and F have been added to the cart 60 earlier and have therefore been detected as objects E and F, respectively (check result "NG").

When the processor 50A stores a detection object with an unregistered product (check result "NG") in the product checklist (Act 50), it transmits a product registration request to the electronic device 30 (Act 51). When the processor 30A of the electronic device 30 receives the product registration request from the clip terminal 50 (Act 21, Yes), it displays a product registration request message on the touch panel 30D to prompt the customer to register the product. For example, as shown in FIG. 9 (B), it displays a product registration request message stating "There is an unregistered product in the cart. Please register the product." It should be noted that the product registration request message may be displayed not only on the electronic device 30 but also on the touch panel 50D of the clip terminal 50 (Act 22).

As described above, after the product is registered, the product registration request message displayed on the touch panel 50D is erased (Acts 20 and 28) when it is confirmed that the product has been placed in the cart 60 based on the recognition process using the product dictionary data of the registered product (Acts 49 to 51).

On the other hand, when returning a product that has been registered as a purchase candidate, the customer uses the camera 30E of the electronic device 30 to scan the barcode attached to the product to be returned, in the same manner as when registering the product (Act 12). The processor 30A detects the barcode from the image captured by the camera 30E and extracts the product code from the barcode. The processor 30A displays the product information (product name, price, product image, etc.) corresponding to the product code on the touch panel 30D, and allows the customer to confirm the product to be returned. Here, when the customer performs an instruction operation to cancel the product (for example, touching a product cancellation button displayed on the touch panel 30D) (Act 13, No), the processor 30A deletes the product read by the product scan from the purchase candidate (Act 16). The processor 30A also transmits a registration cancellation notification to the clip terminal 50 together with the information of the product to be canceled (returned) (Act 17).

When the clip terminal 50 receives a registration cancellation notification from the electronic device 30 (Act 60, Yes), it identifies the corresponding registered product from the product checklist based on the information of the product to be cancelled (Act 61) and deletes the information of the registered product (Act 62). The processor 50A transmits a cancellation completion notification to the electronic device 30 (Act 63). When the electronic device 30 receives the cancellation completion notification (Act 25, Yes), if a product addition request message is displayed, it erases the product addition request message (Act 28). For example, as described above, the electronic device 30 displays the product addition request message if a set time has passed without the product being added to the cart 60 after the product registration process. When the customer performs a product cancellation operation for the product that has been registered, the electronic device 30 erases the product addition request message for the corresponding product.

When a customer removes an item from the cart 60, the processor 50A of the clip terminal 50 determines that the customer has taken an action to put in or take out the item based on the image captured by the camera 50E, and detects an object corresponding to the removed item using existing image processing. In the same way as when an object is detected when an item is added, it can also be determined that an item has been removed by combining this with a change in weight detected by the weight sensor 601.

When the processor 50A detects that an object (product) has been removed from the cart 60 (Act 64, Yes), it determines whether a registered product corresponding to this detected object is stored in the product checklist (check result "OK"). Here, if a corresponding registered product is not stored (Act 65, No), the processor 50A deletes the detected object corresponding to the removed product from the product checklist (Act 68).

On the other hand, if the corresponding registered product is stored (Act 65, Yes), the processor 50A sends a product cancellation request to the electronic device 30 (Act 66). The processor 50A also sets the check result "NG" for the detection object corresponding to the removed product (Act 67).

When the processor 30A of the electronic device 30 receives a product cancellation request from the clip terminal 50 (Act 23, Yes), it displays a product cancellation request message on the touch panel 30D to prompt the customer to cancel the product (Act 24). For example, as shown in FIG. 9(C), the product cancellation request message "The product has been removed from the cart. If you wish to return the product, please perform the return operation." may be displayed. The product cancellation request message may be displayed not only on the electronic device 30, but also on the touch panel 50D of the clip terminal 50.

When an item is removed from the cart 60, the data stored in the item checklist is updated immediately. However, the data may be updated after a predetermined time (e.g., 10 seconds) has elapsed, assuming that there is a high possibility that the item will be canceled (returned). In this case, the item cancellation request message may be displayed on the electronic device 30 immediately when the item is removed, or may be displayed after the set time has elapsed.

In this way, a customer can attach the clip terminal 50 to the cart 60 and go shopping while performing the registration process using the electronic device 30. Even if the customer forgets to register a product they plan to purchase or puts a product in the cart 60, the customer can easily check this by reading the message displayed on the electronic device 30.

When the customer completes the registration process for the product they wish to purchase, they instruct the end of product registration, for example by operating the subtotal button. When the customer instructs the end of product registration (Act 26, Yes), processor 30A of electronic device 30 displays a confirmation screen (product list) for the registered products and prompts the customer to specify a payment method (credit, cash, direct debit, electronic money, etc.). Processor 30A generates payment information based on the information of the products registered by the registration process.

The processor 30A of the electronic device 30 also transmits a notification of the completion of the registration process to the clip terminal 50 (Act 27). When the processor 50A of the clip terminal 50 receives the notification of the completion of the registration process from the electronic device 30 (Act 69, Yes), it transmits the product checklist data to the electronic device 30 (Act 70). This data includes, for example, data indicating the similarity when the detected object is recognized. After transmitting the product checklist data, the processor 50A clears the product checklist data and terminates the process (Act 71).

When an instruction to end the registration process is given, if any data stored in the product checklist has a check result of "NG", a message prompting the customer to check again is displayed on the touch panel 30D of the electronic device 30 (touch panel 50D of the clip terminal 50).

When the customer issues a final confirmation instruction, the processor 30A of the electronic device 30 displays on the touch panel 30D a code (a barcode or two-dimensional code) containing payment information and data included in the product checklist that prompts the attendant to confirm the item.

The attendant uses the camera of the electronic device 40 to scan the code displayed on the touch panel 30D of the electronic device 30. The electronic device 40 displays a confirmation screen according to the payment information included in the code and the data that needs to be confirmed. For example, the electronic device 40 displays a list of products to be purchased that the customer has registered (including product names, prices, subtotals, etc.) and the number of products. This allows the attendant to check whether the products placed in the cart 60 match the registered products. Furthermore, based on the data that needs to be confirmed, the electronic device 40 clearly displays products that correspond to detected objects that had a low similarity in the recognition process of the clip terminal 50, for example. This allows the attendant to easily recognize products that particularly need confirmation, reducing the burden of the confirmation work.

In the above description, an example is described in which image processing in the clip terminal 50 is used to check whether the registered product matches the product placed in the cart 60. However, the product weight detected by the weight sensor 601 may also be used to check the product. For example, the product data provided by the store server 20A includes product weight data, so that a check can be made based on whether the weight increased when the product is placed in the cart 60 matches the weight of the registered product within an allowable range. In particular, for weighed products whose prices vary depending on their weight, it is effective to use the weight data detected by the weight sensor 601 to perform a correct check. For example, it is possible to check for fraud such as placing a product heavier than the same type of registered product in the cart 60 after a registration process is performed using a light-weight product. The product checklist stores data indicating that confirmation is required if there is a weighed product and if the difference between the weight of the registered product and the weight when it is placed in the cart 60 is greater than a set value. This allows the attendant to recognize that confirmation is particularly required, as described above.

In addition, in the above explanation, it is described that the product registered by the registration process is checked to see if it matches the product placed in the cart 60, but it is also possible to use the clip terminal 50 to check the registration of other products. For example, it is also possible to determine that confirmation is necessary when the difference between the timing of product registration in the electronic device 30 or the position in the store at the timing of detection of the product (detected object) by the clip terminal 50 and the position (product position) of the product shelf in the store on which the target product is placed is greater than a preset determination value.

Figure 12 is a flowchart showing the registration position determination process in this embodiment, which determines whether confirmation is necessary based on the registration position. The processor 50A of the clip terminal 50 executes the registration position determination process based on a check program. Note that the registration position determination process is executed in parallel with the individual check process described above.

The processor 50A detects the current location (Act 82) when it receives product information of a registered product from the electronic device 30 (Act 80, Yes), or when it is determined which registered product the detected object corresponds to (check result "OK") (Act 81, Yes). The current location can be detected, for example, by communicating with an RF tag attached to each product shelf in the store using the reader/writer 50J.

Figure 13 shows an example of the layout of product shelves 70-1 to 70-5 in a store and the position of a cart 60. The clip terminal 50 attached to the cart 60 can determine that it is near the product shelf 70-5 when it can communicate with an RF tag 72-5 attached to the product shelf 70-5 through communication within a limited range using the reader/writer 50J. Note that the current position of the cart 60 (clip terminal 50) can be detected not only using the reader/writer 50J (RF tag), but also using methods such as the difference in radio wave strength between the communication module 50F and multiple access points 20F, and image processing to detect the cart 60 in images taken inside the store.

When the processor 50A detects the current location, it next acquires data indicating the product location (product shelf location) where the registered product is being sold, for example, via the store server 20A or the electronic device 30 (Act 83). The processor 50A determines whether the difference between the current location and the product location (product shelf location) is greater than a preset determination value (Act 84). For example, if the product 74 arranged on the product shelf 70-1 shown in FIG. 13 is registered or placed in the cart 60 at a location close to the product shelf 70-5, it determines that the customer has performed an unusual action and that confirmation is required (Act 85, Yes). The clip terminal 50 stores confirmation data that corresponds to the registered product in the product checklist and the data corresponding to the detected object (Act 86). This allows the attendant to recognize that special confirmation is required, as described above.

According to at least one of the embodiments described above, by using the clip terminal 50, a match between the product information registered by the registration process by the electronic device 30 and the product corresponding to the detection object corresponding to the product put into or taken out of the cart 60 detected by the clip terminal 50 is checked. Depending on the result of this check, a message prompting the customer to confirm can be displayed as appropriate, so that the customer can reliably register the product that he or she intends to purchase.

Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims.

The processes described in the above-mentioned embodiments can be written as a program that can be executed by a computer onto a recording medium such as a magnetic disk (flexible disk, hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), or a semiconductor memory, and provided to various devices. They can also be transmitted via a communication medium and provided to various devices. The computer reads the program recorded on the recording medium, or receives the program via the communication medium, and executes the above-mentioned processes by having its operation controlled by this program.

12...store system, 20A...store server, 20B...clip terminal management server, 30, 40...electronic device, 50...clip terminal (electronic device), 50A...processor, 50B...memory, 50C...storage device, 50D...touch panel, 50E...camera, 50F...communication module, 50J...reader/writer, 60...cart, 601...weight sensor.

Claims (6)

A display unit;
A product information acquisition means for acquiring product information related to a product that a customer plans to purchase;
a display unit that displays the product information acquired by the product information acquisition unit on the display unit;
a storage unit for storing the product information acquired by the product information acquisition unit in a storage unit;
a first output means for outputting, when a predetermined time has elapsed since the product information was stored in the storage unit, a first guidance message to the customer on the display unit, which message prompts the customer to store the product in a storage unit for storing the product that the customer plans to purchase;
A recognition means for recognizing the products stored in the storage section;
a second output means for outputting, when product information corresponding to the product recognized by the recognition means is not stored in the storage unit, a second guidance to the customer on the display unit, prompting the customer to acquire the product information by the product information acquisition means;
A product data processing system having the above configuration. The recognition means includes:
image information acquiring means for acquiring image information from an imaging unit that images the storage section;
an object detection means for detecting an object corresponding to a commodity from the image information;
a determination means for determining whether product information of a product corresponding to the object is stored in the storage unit ,
2. The product data processing system according to claim 1 , wherein the second output means outputs the second guidance when the determining means determines that the product information is not stored. 3. The product data processing system according to claim 1, further comprising a third output means for outputting a third notification notifying the user that the product does not need to be stored in the storage unit when the product information indicates a specific product that does not need to be stored in the storage unit. Obtaining product information regarding the product that the customer intends to purchase;
Displaying the product information on a display unit;
storing the product information in a storage unit;
when a preset predetermined time has elapsed since the product information was stored in the storage unit, a first guidance message is output to the display unit to prompt the customer to store the product in a storage unit for storing the product that the customer plans to purchase ;
Recognizing the products stored in the storage section,
A product data processing method in which, when product information corresponding to a recognized product is not stored in the memory unit, a second message is output to the display unit to prompt the customer to obtain the product information . acquiring image information from an imaging unit that images the storage section, and detecting an object corresponding to the commodity from the image information;
determining whether product information of a product corresponding to the object is stored in the storage unit ;
5. The method for processing product data according to claim 4, wherein when it is determined that the product information is not stored, the second notification is output. A product data processing method as described in claim 4 or claim 5, wherein when the product information indicates a specific product that does not need to be stored in the storage unit, a third notification is output to notify that the product does not need to be stored in the storage unit.