JP2019204550A - Commodity data processing system and commodity data processing method - Google Patents

Abstract

To provide a check device for enabling reliable registration of a commodity to be purchased by an operation made by a customer.SOLUTION: According to an embodiment, the check device includes a receiving unit, a camera, an object detection unit, and a check unit. The receiving unit receives commodity information concerning a commodity to be purchased by a customer stored in electronic equipment from the electronic equipment. The camera captures an image. The object detection unit detects an object corresponding to the commodity from the image. The check unit checks agreement between the commodity existing in a range of imaging performed by the camera and the commodity corresponding to the commodity information on the basis of the commodity information and the object.SELECTED DRAWING: Figure 1

Description

  Embodiments described herein relate generally to a check device and a check program.

  In retail stores such as supermarkets, in order to improve the efficiency of accounting processing by POS (Point Of Sales), a checkout system in which a part of accounting processing work is performed by a customer is introduced. For example, a self-checkout system provided with a self-checkout (self-checkout terminal) has been introduced. The self-checkout registers the product by reading the product code (generally a barcode) of the product by the customer's operation, generates the payment information for the payment process, and the payment information generated by the registration process. Execute payment processing to pay the price.

  Further, in recent years, a personal checkout system has been considered in which a customer registers a product for a product to be purchased while the customer is buying the product in the store. In the case of this system, it is necessary that the product put in the cart or the basket as the purchase schedule matches the product registered in the product.

JP-T-2006-532932 Publication

  The problem to be solved by the present invention is to provide a check device and a check program for reliably registering a product to be purchased by a customer's operation.

  According to the embodiment, the check device includes a reception unit, a camera, an object detection unit, and a check unit. A receiving part receives the merchandise information regarding the goods which the customer memorize | stores in the electronic device and which is scheduled to purchase from the said electronic device. The camera takes an image. The object detection unit detects an object corresponding to a product from the image. The check unit checks, based on the product information and the object, a match between the product existing in the shooting range by the camera and the product corresponding to the product information.

1 is a system configuration diagram schematically illustrating an overall configuration of a system according to an embodiment. 1 is a block diagram illustrating a configuration of an electronic device according to an embodiment. The figure which shows the usage example of the clip terminal in this embodiment. The figure which shows the external appearance of the clip terminal in this embodiment. 6 is a flowchart for explaining the operation of the electronic apparatus according to the embodiment. The flowchart for demonstrating operation | movement of the clip terminal in this embodiment. The flowchart for demonstrating operation | movement of the clip terminal in this embodiment. The figure which shows an example of the goods check list in this embodiment. The figure which shows an example of the message in this embodiment. The figure which shows an example of the image image | photographed with the camera in this embodiment. The figure which shows an example of the object corresponded to the goods detected from the image shown in FIG. The flowchart which shows the registration position determination process which determines whether confirmation is required based on the registration position in this embodiment. The figure which shows an example of arrangement | positioning of the merchandise shelf in a store, and the position of a cart.

Hereinafter, the present embodiment will be described with reference to the drawings.
FIG. 1 is a system configuration diagram schematically showing the overall configuration of the system of the present embodiment. In the system shown in FIG. 1, a system 10 provided in the headquarters and a store system 12 provided in each of a plurality of stores are connected via a network 14 including a dedicated line or the Internet.

  The system 10 provided in the headquarters includes, for example, a headquarter system 10A, a customer system 10B, a payment system 10C, a sales promotion system 10D, and the like, and are connected to each other via a network 10E such as a LAN (Local Area Network).

  The headquarters system 10A executes a process for supervising a store system or the like installed in each store. The customer system 10B manages customers who purchase products using the store system. The settlement system 10C executes processing (such as personal authentication and credit settlement) for settlement processing using a credit card or the like by a customer in the store system. The sales promotion system 10 </ b> D performs management for providing information (such as product advertisements for sales promotion and in-store guidance) to customers (electronic devices used by customers) through the store system.

  The store system 12 provided in the store is realized as, for example, a POS (Point Of Sales) system. The store system 12 includes, for example, a store server 20A, a clip terminal management server 20B, a plurality of accounting machines 20C, a plurality of POS terminals 20D, and the like, and is connected to each other via a network 20E such as a LAN.

  The store server 20 </ b> A controls the entire store system 12. The store server 20A stores data related to products sold in the store, such as product code for each product, product name, unit price, weight, product position (product shelf position) sold in the store, product image, etc. Data is stored. In the registration process, product data corresponding to the product code read by scanning the product is received from the store server 20A and stored, and payment information for payment processing (including amount, product, number, etc.) Can be generated. In the clip terminal management server 20B, the product (purchased product) that has been registered using the electronic device 30 as a purchase schedule by the operation of the customer matches the product that has been placed in the cart 60 or the basket as the purchase schedule. The electronic device 50 (clip terminal to be described later) used for checking whether or not is checked is managed. The accounting machine 20 </ b> C executes a settlement process for a product that has been registered using the electronic device 30 by a customer operation. The POS terminal 20D executes a registration process for the customer's purchased product by the operation of the store clerk, a settlement process for the product registered by the registration process, and the like.

  A plurality of access points (base stations) 20F are connected to the network 20E wired in the store. The access point 20F performs wireless communication with various electronic devices used in the store using, for example, a wireless LAN. The electronic device used in the store includes, for example, an electronic device 30 used by a customer, an electronic device 40 used by a store clerk (attendant), etc., for example, an electronic device 50 used in a state of being attached to a cart 60 (a clip terminal described later) 50).

  For example, a smart phone or a tablet personal computer (PC) is used as the electronic devices 30 and 40. The electronic device 30 registers and purchases a product (purchased product) that has undergone registration processing and registration processing for a product to be purchased based on an application program (including a check program) for using the service provided by the store system 12. The individual check process for checking whether the product in the cart 60 or the basket as a schedule matches is executed in cooperation with the electronic device 50 (clip terminal). In the electronic device 40, based on a confirmation program for confirming the registration process executed by the electronic device 30, the clerk (attendant) matches the product information registered by the registration process with the product put in the cart 60. Execute confirmation processing to make it easy to confirm.

  The electronic device 30 and the electronic device 50 (clip terminal 50) may communicate with each other via the clip terminal management server 20B (network 20E), or may directly communicate with each other by short-range wireless communication. Further, communication cables may be connected to each other for wired communication.

  FIG. 2 is a block diagram showing a configuration of the electronic devices 30 and 50 in the present embodiment. The electronic device 30 shown in FIG. 2 is demonstrated as what is implement | achieved by the smart phone, for example. In addition, the electronic device 50 is configured as a dedicated clip terminal 50 for executing the individual check process in the store system 12. Hereinafter, the clip terminal 50 will be described. Note that the clip terminal 50 can also be realized by installing a check program in a general-purpose device (computer) on which a camera such as a smartphone is mounted.

The electronic device 30 includes a processor 30A, a memory 30B, a storage device 30C, a touch panel 30D, a camera 30E, a communication module 30F, and a battery 30G.
The processor 30A is a circuit that controls each unit in order to realize various processes as the electronic device 30 by executing a program recorded in the memory 30B.

  In addition to the program executed by the processor 30A, the memory 30B records data used when the processor 30A executes various processes. The program recorded in the memory 30B includes, for example, a registration processing program for executing registration processing for a product scheduled for purchase, a product for which registration processing has been performed (purchased product), and a product that has been placed in a cart 60 or a basket as a purchase schedule Includes a check program for executing an individual check process for checking whether or not and. The data stored in the memory 30B stores settlement information generated based on the product code read in the registration process, product information related to the product registered by the registration process, and the like.

  The storage device 30C is realized by a hard disk (HDD (Hard disk drive)) device, an SSD (solid state drive) device, or the like, and is generated by data used by the processor 30A for performing various processes or by processing by the processor 30A. Record the recorded data.

The touch panel 30D is a device that is input by a customer, and includes a display device and a touch sensor. On the touch panel 30D, an arbitrary screen such as a GUI (Graphical User Interface) screen associated with the processing of the processor 30A is displayed.
The camera 30E captures an image under the control of the processor 30A.

  The communication module 30F controls communication with an external electronic device under the control of the processor 30A. The communication module 30F can control communication such as wireless communication with the access point 20F and short-range wireless communication with other electronic devices such as the clip terminal 50.

  The clip terminal 50 includes a processor 50A, a memory 50B, a storage device 50C, a touch panel 50D, a camera 50E, a communication module 50F, a battery 50G, an I / O interface 50H, and a reader / writer 50J.

  The processor 50A is a circuit that controls each unit in order to realize various processes as the clip terminal 50 by executing a program recorded in the memory 50B.

  In addition to the program executed by the processor 50A, the memory 50B records data used when the processor 50A executes various processes. The program recorded in the memory 50B includes a check program that realizes an individual check process for checking whether a product (purchased product) that has undergone registration processing matches a product that has been put in the cart 60 or the basket as a purchase schedule. Is included. The check program detects an object corresponding to a product from an image photographed by the camera 50E, and realizes image processing for detecting that the product has been put into the cart 60 and that the product has been taken out from the cart 60. A processing program is included. The data stored in the memory 50B includes product information registered by the registration process of the electronic device 30, object management data for managing objects detected by the image processing (see FIG. 11), and products stored in the cart 60. Various data such as a product check list (see FIG. 8) for checking is stored.

  The storage device 50C is realized by a hard disk device (HDD (Hard disk drive)), an SSD (solid state drive) device, or the like, and is generated by data used by the processor 50A for performing various processes or by processing by the processor 50A. Record the recorded data.

The touch panel 50D is a device that is input by a customer, and includes a display device and a touch sensor. On the touch panel 50D, an arbitrary screen such as a GUI (Graphical User Interface) screen associated with the processing of the processor 50A is displayed.
The camera 50E captures an image under the control of the processor 50A.

  The communication module 50F controls communication with an external electronic device under the control of the processor 50A. The communication module 50F can control communication such as wireless communication with the access point 20F and short-range wireless communication with other electronic devices such as the electronic device 30.

  The I / O interface 50H is an interface for inputting a signal from an external device. For example, a weight sensor 601 (a measuring instrument, a pressure sensor, a weighing scale, etc.) attached to the bottom 60C of the cart 60 is connected to the I / O interface 50H. The weight sensor 601 detects the weight of a product placed (placed) in the cart 60 and outputs a detection signal.

  The reader / writer 50J communicates with an RF (Radio Frequency) tag attached to each product sold in the store or a product shelf in the store. The communication range by the reader / writer 50J is, for example, a range that can be communicated with an RF tag of a product placed in the cart 60, or a range that can be communicated with an RF tag attached to a product shelf by moving the cart 60 in the store. To do. The reader / writer 50J is used for detecting that a product has been put in or taken out of the cart 60, or used for detecting the position in the store of the cart 60 to which the clip terminal 50 is attached. Can do.

  FIG. 3 is a diagram illustrating an example of use of the clip terminal 50 in the present embodiment, and FIG. 4 is a diagram illustrating an appearance of the clip terminal 50 in the present embodiment.

  The clip terminal 50 is used by being mounted on an upper frame 60B of a cart 60 used in a store, for example. As shown in FIG. 4, the clip terminal 50 is provided with a clip 60k, which can be arbitrarily attached and detached. Therefore, it is not necessary to attach the clip terminal 50 in advance to all carts 60 prepared in advance in the store, and only the cart 60 used by the customer who uses the individual check v processing service using the clip terminal 50 is used. The clip terminal 50 may be attached as necessary. For this reason, a large-scale investment such as remodeling the store or exchanging the cart 60 is not necessary to realize the individual check processing service, and can be realized at low cost.

  In the example shown in FIG. 3, an example in which the cart 60 is attached to the upper frame 60 </ b> B on the handle 60 </ b> A side is shown. When the clip terminal 50 is attached, the camera 50E provided on the clip terminal 50 is pointed in the direction of the bottom 60C of the basket, and the shooting range is adjusted so that at least all products placed in the cart 60 can be shot. Is done. The shooting range by the camera 50E preferably includes the upper frame 60B so that the area inside and outside the basket can be clearly divided in the image shot by the camera 50E. Furthermore, it is desirable that the shooting range by the camera 50E is capable of shooting, for example, the movement of the customer's arm (hand) so that the movement of the product in and out by the customer can be detected.

  As shown in FIG. 4, the clip terminal 50 is attached to the upper frame 60B so that the display surface of the touch panel 50D faces the customer side. This facilitates confirmation of an image photographed by the camera 50E and information (message or the like) representing the check result of the individual check process using the photographed image.

  Next, operations of the electronic device 30 and the clip terminal 50 (electronic device) used in the store system 12 of the present embodiment will be described. FIG. 5 is a flowchart for explaining the operation of the electronic device 30 in the present embodiment. 6 and 7 are flowcharts for explaining the operation of the clip terminal 50 in the present embodiment.

  When the customer performs registration processing using the electronic device 30, the customer activates an application program (check program) for using the service provided by the store system 12. Hereinafter, the processor 30A of the electronic device 30 executes a registration process and an individual check process described below based on the application.

  First, the electronic device 30 executes a login process for connecting to the store system 12 (Act 10). In the login process, for example, a bar code including a member code unique to the member attached to the member card is read by the camera 30E by the operation of the customer (image shooting). The processor 30A extracts the member code from the barcode read by the camera 30E and transmits it to the clip terminal management server 20B. Based on the membership code received from the electronic device 30, the clip terminal management server 20B collates with the membership data managed in the customer system 10B and performs membership authentication.

  When notified by the clip terminal management server 20B that the member has been authenticated, the electronic device 30 displays a selection screen for services available in the store. Services available at the store include a flyer information display, sale product guidance display, store guidance (map) display, use of the clip terminal 50, and the like in addition to registration processing for the purchase schedule processing. By using the clip terminal 50, a message is output from the clip terminal 50 when, for example, registration processing is not performed for a product put in the cart 60. Accordingly, the customer can prevent forgetting to register the product to be purchased. In addition, since the products to be purchased are checked by the clip terminal 50 while the customer is making a purchase in the store, the burden of the confirmation work by the attendant (sales staff) after the registration process by the customer is completed. Can be reduced.

  When the customer uses the clip terminal 50, the store system 12 provides the customer with profits (points and the like) (stores data indicating the profit for the customer in the customer system 10B), thereby providing the customer with the profit. Thus, the use of the clip terminal 50 may be prompted.

  When the clip terminal 50 is used, the electronic device 30 and the clip terminal 50 are set in a communicable state. For example, the clip terminal 50 is mounted on the cart 60 used by the customer (or the customer uses the cart 60 on which the clip terminal 50 is previously mounted), and a link between the customer's electronic device 30 and the clip terminal 50 is set. For example, the electronic device 30 and the clip terminal 50 are paired so that they can communicate by short-range wireless communication. Alternatively, the unique code of the clip terminal 50 is read by the electronic device 30 and transmitted from the electronic device 30 to the clip terminal management server 20B. The clip terminal management server 20B discriminates between the electronic device 30 and the clip terminal 50 that cooperate to execute the individual check process, and sets the clip terminal management server 20B (network 20E) between the electronic device 30 and the clip terminal 50. To be able to communicate via It should be noted that other methods can be used for connecting the electronic device 30 and the clip terminal 50.

  When connected to the clip terminal 50 (Act 11), the processor 30A of the electronic device 30 makes the product registration process executable. Further, the processor 50A of the clip terminal 50 is connected to the electronic device 30 (Act 40), and starts a product detection process. In the product detection process, an object corresponding to a product is detected from an image photographed by the camera 50E. The processor 50A detects an object corresponding to a product or the like from an image photographed by the camera 50E by existing image processing, and stores the position of each object. For example, when an object moving from the area corresponding to the outside of the cart (outside the cart) to the inside of the cart (inside the cart) is detected, the processor 50A determines that the customer has performed an operation for putting in and out the product. Based on the change of the object existing in the cart before and after detecting this motion, it is detected that a product has been put in or taken out. Further, even when the movement of the customer is detected, if the number of objects corresponding to the product in the cart has not changed, it is determined that the customer has replaced the product in the cart, for example. The processor 50A stores the position of the replaced object. The processor 50A stores the object detected from the image photographed by the camera 50E in the product check list.

  FIG. 8 is a diagram showing an example of a product check list in the present embodiment. In the product check list, as shown in FIG. 8, the data of the object (detected object) detected from the image taken by the camera 50 </ b> E and the product information (registered product) registered by the registration process in the electronic device 30. Data (for example, D1 to D4) in which data (data of check results “OK” and “NG”) indicating the result of checking the match between the registered product and the product corresponding to the detected object is associated is stored.

  The customer registers the product to be purchased while buying around the store by scanning the barcode attached to the product using the camera 30E of the electronic device 30 (Act 12). The processor 30A detects a barcode from an image photographed by the camera 30E, and extracts a product code from the barcode. The processor 30A acquires product information corresponding to the product code from the store server 20A through, for example, the clip terminal management server 20B (network 20E), and displays the information on the touch panel 30D. That is, the product name, price, product image, etc. of the product read by the product scan are displayed and confirmed by the customer. Here, when a customer performs an instruction operation for product registration (for example, touching a product registration button displayed on the touch panel 30D) (Act 13, Yes), the processor 30A plans to purchase the product read by the product scan. Store as a registered product (Act 14). Further, the processor 30A transmits information on the registered product to the clip terminal 50 (Act 15).

  On the other hand, when the clip terminal 50 receives information on the registered product from the electronic device 30 (Act 42, Yes), the clip terminal 50 stores the information on the registered product in the product check list (Act 43). When the processor 50A detects that the product is a large product with reference to the information of the registered product, that is, when it is determined that the product is not necessary (cannot be entered) in the cart 60, the product checklist A skip flag is set in (Act 45). In the example shown in FIG. 8, a skip flag is set as data D2 in association with the registered product “product C”. For the registered product “product C” in which the skip flag is set, the processor 50A excludes the registered product from the target for checking the match between the registered product and the product placed in the cart 60. Further, the processor 50A transmits a product insertion unnecessary notification to the electronic device 30 (Act 57).

  When the product insertion unnecessary notification is received from the clip terminal 50 (Act 18, Yes), the processor 30A of the electronic device 30 receives information on the registered product (product name, product image, price, etc.) on the touch panel 30D and the customer. Displays a message notifying that it is not necessary to put the product in the cart 60. Note that the message indicating that the product is not required may be displayed not only on the electronic device 30 but also on the touch panel 50D of the clip terminal 50.

  On the other hand, when the registered product is not a large product, the processor 50A acquires product dictionary data for identifying the registered product by image processing from the clip terminal management server 20B through the network 20E, for example. For example, when the product dictionary data corresponding to the registered product is not stored, the processor 50A outputs a transmission request for the product dictionary data corresponding to the registered product to the clip terminal management server 20B. In response to the transmission request from the clip terminal 50, the clip terminal management server 20B transmits the product dictionary data of the corresponding product to the clip terminal 50. When the electronic device 30 and the clip terminal 50 communicate via the clip terminal management server 20B, the clip terminal management server 20B receives the registered product information from the electronic device 30. At the same time, the product dictionary data of the corresponding product may be transmitted to the clip terminal 50. The product dictionary data may be stored in the clip terminal 50 in advance.

  When the registered product is stored in the product check list, the processor 50A starts measuring the elapsed time since the registered product is stored. The processor 50A determines whether or not the elapsed time has elapsed in advance until it is confirmed that the product corresponding to the registered product has been put in the cart 60 (Act 46).

  In many cases, a customer intends to handle a product to be purchased, for example, by taking a product from a product shelf or the like, scanning the product using the electronic device 30, and placing the product on the cart 60. Therefore, when a preset time has elapsed since the product scan (storage of the registered product in the clip terminal 50), the customer may be notified because there is a possibility that the registered product has been left in the cart 60. ing. When it is determined that the elapsed time has passed for the registered product (Act 46, Yes), the processor 50A transmits a product insertion request to the electronic device 30 (Act 47).

  When the product insertion request is received from the clip terminal 50 (Act 18, Yes), the processor 30A of the electronic device 30 informs the customer along with information on the registered product (product name, product image, price, etc.) on the touch panel 30D. On the other hand, a product insertion request message prompting the user to put the product into the cart 60 is displayed (Act 19). For example, as shown in FIG. 9A, a message “Please put the registered product into the cart.” Is displayed. The product insertion request message may be displayed not only on the electronic device 30 but also on the touch panel 50D of the clip terminal 50. In addition, when there are a plurality of products to be targeted by the product insertion request message, information about the plurality of products to be targeted is displayed in a list so that the customer can surely recognize the target products. good.

  On the other hand, when the processor 50A of the clip terminal 50 detects that the customer has performed an operation for putting in and out the product by image processing on the image captured by the camera 50E, the processor 50A is placed in the cart before and after detecting the operation. Based on a change in an existing object, it is detected that a product has been put in or taken out. The processor 50A not only detects the movement of the product based on the image, but also determines the loading / unloading of the product in combination with a change in the weight detected by the weight sensor 601 at the timing when the operation is detected. You can also

  FIG. 10 shows an example of an image taken by the camera 50E. In the example shown in FIG. 10, for example, a state in which three products A, B, and E are put in the cart 60 is shown.

  The processor 50A detects the position of the upper frame 60B of the cart 60 and the area of the bottom 60C of the cart 60 from the image captured by the camera 50E. Since the clip terminal 50 is fixed to the cart 60, basically, the region of the upper frame 60B and the basket bottom portion 60C in the image does not change during buying. When the processor 50A detects, for example, an object moving from the area corresponding to the outside of the cart (outside the cart) to the inside of the cart (inside the cart) based on the position of the upper frame 60B, the customer puts in and out the merchandise. It is determined that the operation has been performed.

  Further, the processor 50A detects an object corresponding to a product or the like from an image photographed by the camera 50E by existing image processing, and stores the position of each object as object management data.

  FIG. 11 shows an example of objects OJ-A, OJ-B, and OJ-E corresponding to the products detected from the image shown in FIG. The processor 50A detects that a product has been put in or taken out based on a change in an object existing in the cart before and after the product has been put in and out by the customer.

  When an object corresponding to a product is detected from the image (Act 48, Yes), the processor 50A refers to the product check list and determines whether a registered product with the check result “NG” is stored. That is, it is determined whether there is a product that has been registered by the electronic device 30 but has not been confirmed that the corresponding product has been placed in the cart 60. Here, when the registered product of the check result “NG” is stored (Act 49, Yes), the processor 50A is detected from the image using the product dictionary data corresponding to the registered product of the check result “NG”. Recognition processing is executed for the object (Act 52). The merchandise dictionary data is created so that merchandise can be recognized based on features such as barcodes attached to the merchandise, package design, and characters.

  Here, based on the processing result of the recognition process using the product dictionary data, it is determined whether the object detected from the image is a registered product corresponding to the product dictionary data. For example, when the feature of the image represented by the object and the feature indicated by the product dictionary data indicate similarity exceeding a preset reference value, it is determined that the product is a registered product corresponding to the product dictionary data.

  If it is determined that the product is a registered product (Act 53, Yes), the processor 50A associates the registered product previously stored in the product check list with the product dictionary data used in the recognition process. Then, the object detected from the image is stored as a detected object (Act 54). Note that data indicating similarity (similarity) in the recognition processing may be stored as data related to the detected object. As a result, it is determined that the product is a registered product corresponding to the product dictionary data. However, since the similarity is low, it is possible to prompt the attendant to confirm the product before the settlement process.

  Further, the processor 50A sets a check result “OK” corresponding to the combination of the registered product and the detected object (Act 55), and stores that it has been confirmed that the registered product has been put in the cart 60. In this case, the processor 50A transmits a product insertion confirmation to the electronic device 30 (Act 56).

  When the product insertion confirmation is received from the clip terminal 50 (Act 20, Yes), the electronic device 30 deletes this message from the touch panel 30D when the product insertion request message is displayed (Act 28). Accordingly, it is possible to notify the customer that the clip terminal 50 has confirmed that the registered product has been put in the cart 60.

  When there are a plurality of registered products with the check result “NG” in the product check list, the processor 50A executes a recognition process on the object using the product dictionary data of each registered product. For an object that has not been determined to be a registered product by the recognition process (Act 53, No), the processor 50A stores the check result “NG” in the product check list as a detection object for which the product has not been registered (Act 50). ). In addition, when an object is detected from the image (Act48, Yes), if the registered product of the check result “NG” is not stored (Act49, No), the processor 50A similarly does not register the product. As a detection object, the check result “NG” is stored in the product check list (Act 50).

  The data D3 shown in FIG. 8 indicates that the product C has been registered in the electronic device 30, but the check result “NG” is set and it has not been confirmed that the product has been put in the cart 60. Yes. The data D4 indicates that unregistered products F and F are detected as objects E and F, respectively, by having been put in the cart 60 first (check result “NG”).

  When the detected object (check result “NG”) in which the product is not registered is stored in the product check list (Act 50), the processor 50A transmits a product registration request to the electronic device 30 (Act 51). When the product registration request is received from the clip terminal 50 (Act21, Yes), the processor 30A of the electronic device 30 causes the touch panel 30D to display a product registration request message that prompts the customer to register the product. For example, as shown in FIG. 9B, a product registration request message “There are unregistered products in the cart. Please register products.” Is displayed. The product registration request message may be displayed not only on the electronic device 30 but also on the touch panel 50D of the clip terminal 50 (Act 22).

  In addition, as described above, the product registration request message displayed on the touch panel 50D is registered in the cart 60, and then the product is put in the cart 60 based on the recognition process using the product dictionary data of the registered product. If it is confirmed (Act 49 to 51), it is deleted (Act 20, 28).

  On the other hand, when returning a product registered as a purchase plan, the customer scans the barcode to be attached to the product using the camera 30E of the electronic device 30 for the product to be returned in the same manner as when registering the product. (Act12). The processor 30A detects a barcode from an image photographed by the camera 30E, and extracts a product code from the barcode. The processor 30A displays product information (product name, price, product image, etc.) corresponding to the product code on the touch panel 30D, and allows the customer to confirm the product to be returned. Here, when an instruction operation for canceling the product (for example, touching a product cancel button displayed on the touch panel 30D) is performed by the customer (Act13, No), the processor 30A starts purchasing the product read by the product scan. Delete (Act 16). Further, the processor 30A transmits a notification of registration cancellation to the clip terminal 50 together with information on a product to be canceled (returned) (Act 17).

  When the clip terminal 50 receives the registration cancellation notification from the electronic device 30 (Act 60, Yes), the clip terminal 50 determines the corresponding registered product from the product check list based on the information of the cancellation target product (Act 61), and this registered product. Is deleted (Act 62). The processor 50A transmits a cancellation completion notification to the electronic device 30 (Act 63). When the electronic device 30 receives the cancellation completion notification (Act 25, Yes), the electronic device 30 deletes the merchandise insertion request message if the merchandise insertion request message is displayed (Act 28). For example, as described above, the electronic device 30 displays the product insertion request message when the set time has passed without the product being put in the cart 60 after the product registration processing has been performed. The electronic device 30 deletes the product insertion request message for the corresponding product by performing a product cancellation operation for the product for which the customer has registered.

  In addition, when the product put in the cart 60 is taken out by the customer, the processor 50A of the clip terminal 50 indicates that the customer has performed an operation for putting in and out the product based on the image taken by the camera 50E. The object corresponding to the product extracted by the existing image processing is detected. Note that it is also possible to determine that a product has been put out in combination with a change in the weight detected by the weight sensor 601 in the same manner as when an object is detected when a product is placed.

  When the processor 50A detects that the object (product) has been removed from the cart 60 (Act 64, Yes), the registered product corresponding to the detected object is stored in the product check list (check result) "OK"). If the corresponding registered product is not stored (Act 65, No), the processor 50A deletes the detection object corresponding to the extracted product from the product check list (Act 68).

  On the other hand, when the corresponding registered product is stored (Act 65, Yes), the processor 50A transmits a product cancellation request to the electronic device 30 (Act 66). Further, the processor 50A sets the check result “NG” for the detected object corresponding to the taken-out product (Act 67).

  When the product cancellation request is received from the clip terminal 50 (Act 23, Yes), the processor 30A of the electronic device 30 causes the touch panel 30D to display a product cancellation request message that prompts the customer to cancel the product (Act 24). . For example, as shown in FIG. 9C, a merchandise cancellation request message of “A merchandise has been removed from the cart. The product cancellation request message may be displayed not only on the electronic device 30 but also on the touch panel 50D of the clip terminal 50.

  Note that when the product is taken out from the cart 60, the data stored in the product checklist is immediately updated. However, after a predetermined set time (for example, 10 seconds) has elapsed, the product cancellation ( The data may be updated assuming that there is a high possibility of (return). In this case, the product cancellation request message may be displayed immediately on the electronic device 30 according to the extraction of the product, or may be displayed after a set time has elapsed.

  In this way, the customer forgets to register the product to be purchased and forget to put the product in the cart 60 by attaching the clip terminal 50 to the cart 60 and making a purchase while performing registration processing using the electronic device 30. Even if it exists, it can be easily confirmed by a message displayed on the electronic device 30.

  When the customer completes the registration process for the product to be purchased, for example, the customer instructs the end of the product registration by operating the subtotal button. When the end of the product registration is instructed by the customer (Act 26, Yes), the processor 30A of the electronic device 30 displays a confirmation screen (product list) of the registered product and a settlement method (credit, cash, account debit). , Electronic money, etc.). The processor 30A generates payment information based on information on the product registered by the registration process.

  Further, the processor 30A of the electronic device 30 transmits a registration processing end notification to the clip terminal 50 (Act 27). When the notification of the end of the registration process is received from the electronic device 30 (Act 69, Yes), the processor 50A of the clip terminal 50 transmits the product checklist data to the electronic device 30 (Act 70). This data includes, for example, data indicating the similarity when the detected object is recognized. After transmitting the product checklist data, the processor 50A clears the product checklist data and ends the processing (Act 71).

  When the registration process is instructed to end and the data of the check result “NG” remains in the data stored in the product check list, a message for prompting the customer to confirm again is sent electronically. It is displayed on the touch panel 30D of the device 30 (the touch panel 50D of the clip terminal 50).

  When the final confirmation instruction is given by the customer, the processor 30A of the electronic device 30 touches the settlement information and a code (bar code or two-dimensional code) including data prompting the attendant included in the product check list for confirmation. Display on 30D.

  The attendant scans the code displayed on the touch panel 30 </ b> D of the electronic device 30 using the camera of the electronic device 40. The electronic device 40 displays a confirmation screen corresponding to the payment information included in the code and the data that needs to be confirmed. For example, the electronic device 40 displays a list of products to be purchased (including product names, prices, subtotal amounts, etc.) and product scores that the customer has registered. Thereby, the attendant can confirm whether the product put in the cart 60 matches the registered product. Furthermore, based on the data that needs to be confirmed, the electronic device 40 explicitly displays the product corresponding to the detected object having a low similarity in the recognition process of the clip terminal 50, for example. Thereby, it is possible to easily recognize a product that particularly needs to be confirmed with respect to the attendant, thereby reducing the burden of confirmation work.

  In the above description, an example is described in which the image processing in the clip terminal 50 is used to check the coincidence between the registered product and the product placed in the cart 60. However, the weight of the product detected by the weight sensor 601 is described. You can also use it to check products. For example, since the data related to the product provided from the store server 20A includes the product weight data, the weight increased when the product is put in the cart 60 matches the weight of the registered product within an allowable range. You can check based on. In particular, it is effective to use the weight data detected by the weight sensor 601 for a weighing product whose price varies depending on the weight in order to make a correct check. For example, after performing registration processing with a product having a small weight, it is possible to confirm fraud in which a product heavier than the same type of registered product is placed in the cart 60. The product checklist stores data indicating that confirmation is necessary when there is a weighing product and the difference between the weight of the registered product and the weight when put in the cart 60 is larger than the set value. Keep it. As a result, as described above, it is possible to recognize that the attendant particularly needs to confirm.

  Further, in the above description, the matching between the product registered by the registration process and the product put in the cart 60 is checked. However, the clip terminal 50 may be used to check the registration of other products. Is possible. For example, the position in the store at the timing of product registration in the electronic device 30 or the timing at which the product (detected object) is detected by the clip terminal 50, and the position of the product shelf (product position) in the store where the target product is placed. It is also possible to determine that confirmation is necessary when the difference between is greater than a preset determination value.

  FIG. 12 is a flowchart showing a registered position determination process for determining whether confirmation is necessary based on the registered position in the present embodiment. The processor 50A of the clip terminal 50 performs a registered position determination process based on the check program. It is assumed that the registration position determination process is executed in parallel with the individual check process described above.

  The processor 50A receives the product information of the registered product from the electronic device 30 (Act 80, Yes), or when it is determined which registered product the detected object corresponds to (Check result “OK”) (Act 81) , Yes), the current position is detected (Act 82). The current position can be detected, for example, by communicating with the RF tag attached to each of the product shelves in the store by the reader / writer 50J.

  FIG. 13 is a diagram illustrating an example of the arrangement of the merchandise shelves 70-1 to 70-5 in the store and the position of the cart 60. When the clip terminal 50 attached to the cart 60 can communicate with the RF tag 72-5 attached to the product shelf 70-5 by communication within a limited range by the reader / writer 50J, the product shelf 70 It can be determined that it is near -5. Note that the detection of the current position of the cart 60 (clip terminal 50) is not only a method using the reader / writer 50J (RF tag), but also the difference in radio wave strength between the communication module 50F and the plurality of access points 20F, It is possible to use a method such as image processing for detecting the cart 60 with respect to an image to be captured.

  When the processor 50A detects the current position, the processor 50A acquires data indicating the product position (position of the product shelf) where the registered product is sold next through, for example, the store server 20A or the electronic device 30 (Act 83). The processor 50A determines whether or not the difference between the current position and the product position (the position of the product shelf) is larger than a preset determination value (Act 84). For example, when the product 74 arranged in the product shelf 70-1 shown in FIG. 13 is registered in the position close to the product shelf 70-5 or put into the cart 60, the customer performs an operation different from normal. It is determined that confirmation is necessary (Act, Yes). The clip terminal 50 stores confirmation data in association with the corresponding registered product in the product checklist and data corresponding to the detected object. As a result, as described above, the attendant can recognize that confirmation is particularly necessary.

  According to at least one embodiment described above, by using the clip terminal 50, the product information registered by the registration process by the electronic device 30 and the product put in and out of the cart 60 detected by the clip terminal 50 are used. A match with the product corresponding to the corresponding detected object is checked. According to the check result, a message prompting the customer to confirm can be displayed as appropriate, so that it is possible to reliably register the product to be purchased by the customer's operation.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  In addition, the processing described in the above-described embodiment is a program that can be executed by a computer, for example, a recording medium such as a magnetic disk (flexible disk, hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), and a semiconductor memory. And can be provided to various devices. It is also possible to transmit to a variety of devices by transmitting via a communication medium. The computer reads the program recorded on the recording medium or receives the program via the communication medium, and the operation is controlled by this program, thereby executing the above-described processing.

  DESCRIPTION OF SYMBOLS 12 ... Store system, 20A ... Store server, 20B ... Clip terminal management server, 30, 40 ... Electronic device, 50 ... Clip terminal (electronic device), 50A ... Processor, 50B ... Memory, 50C ... Storage device, 50D ... Touch panel, 50E ... camera, 50F ... communication module, 50J ... reader / writer, 60 ... cart, 601 ... weight sensor.

Embodiments described herein relate generally to a product data processing system and a product data processing method .

The problem to be solved by the present invention is to provide a product data processing system and a product data processing method for reliably registering a product to be purchased by a customer's operation.

According to the embodiment, the product data processing system includes product information acquisition means, display means, storage means, and output means. The product information acquisition means acquires product information related to a product that the customer plans to purchase. The display unit displays the product information acquired by the product information acquisition unit on a display unit. The storage unit stores the product information acquired by the product information acquisition unit in a storage unit. The output means outputs a guide for encouraging the storage unit to store the product when a predetermined time elapses after the product information is acquired by the product information acquisition unit.

The data D3 shown in FIG. 8 indicates that the product C has been registered in the electronic device 30, but the check result “NG” is set and it has not been confirmed that the product has been put in the cart 60. Yes. Data D4 indicates that unregistered products E 1 and F are detected as objects E and F, respectively, when they are first placed in the cart 60 (check result “NG”).

Claims (6)

A receiving unit for receiving, from the electronic device, product information related to a product to be purchased by a customer stored in the electronic device;
A camera for taking images,
An object detection unit for detecting an object corresponding to a product from the image;
A check device comprising: a check unit that checks a match between a product existing within a photographing range of the camera and a product corresponding to the product information based on the product information and the object.   The check device according to claim 1, further comprising: a transmission unit that transmits a check result by the check unit to another electronic device. A detection unit for detecting a change in weight including a product existing in a shooting range by the camera;
The check device according to claim 1, wherein the check unit checks according to a change in the weight. It further has a discriminating unit that discriminates products that are not to be checked based on the product information,
The check device according to claim 1, wherein the check unit does not check the product determined by the determination unit. Computer
A receiving unit for receiving, from the electronic device, product information related to a product to be purchased by a customer stored in the electronic device;
An object detection unit for detecting an object corresponding to a product from an image taken by a camera;
A check program for functioning as a check unit for checking a match between a product existing within a photographing range of the camera and a product corresponding to the product information based on the product information and the object. Computer
A registration unit for registering product information of products to be purchased,
A transmitting unit that transmits the product information to a check device that checks for a match between the product corresponding to the product information and a product that exists in a predetermined range;
A receiving unit that receives a check result based on the transmission of the product information from the check device;
A check program for functioning as a display unit for displaying the check result.