JP2023026627A - Check apparatus and check program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a check device for surely enabling registration of products to be purchased due to customer operation.

SOLUTION: According to an embodiment, the checking device includes a receiving unit, a camera, an object detection unit, and a check unit. The receiving unit receives, from the electronic device, product information on products to be purchased by a customer stored in the electronic device. The camera takes pictures. The object detection unit detects an object corresponding to a product from the image. Based on the product information and the object, the check unit checks a match between a product existing within the imaging range by the camera and an item corresponding to the product information.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2023,JPO&INPIT

Description

TECHNICAL FIELD Embodiments of the present invention relate to a check device and a check program.

Retail stores such as supermarkets have introduced a checkout system that allows customers to perform part of the accounting process in order to improve the efficiency of accounting by POS (Point Of Sales). For example, a self-checkout system with a self-checkout terminal (self-checkout terminal) has been introduced. In a self-checkout, a product is registered by reading the product code (generally a bar code) of the product by the customer's operation, and the registration process to generate the payment information for the payment process, as well as the payment information generated by the registration process. Settlement processing is executed to make the payment.

Furthermore, in recent years, a personal checkout system has been considered in which a customer registers products to be purchased while shopping around in the store. In the case of this system, it is necessary that the product put in the cart or basket as a purchase plan matches the product registered.

Japanese translation of PCT publication No. 2016-532932

SUMMARY OF THE INVENTION An object of the present invention is to provide a checking device and a checking program for ensuring that a product to be purchased is registered by a customer's operation.

According to an embodiment, the checking device comprises detection means, determination means and storage means. The detecting means detects the current position when it is determined to which product the detected object detected from the image corresponds. The determination means determines whether the difference between the current position and the product position where the product is placed is greater than a preset determination value. The storage means stores data in which the product and the detected object are associated with each other when the difference is greater than the determination value.
Also, according to an embodiment, the checking device comprises detection means, determination means, and storage means. The detection means detects the current position when product information is received from the electronic device. The determination means determines whether the difference between the current position and the product position where the product corresponding to the product information is placed is greater than a preset determination value. If the difference is greater than the determination value, the storage means stores data in which the product and the detected object detected from the image are associated with each other.

1 is a system configuration diagram schematically showing the overall configuration of a system according to an embodiment; FIG. 1 is a block diagram showing the configuration of an electronic device according to the embodiment; FIG. FIG. 4 is a diagram showing a usage example of a clip terminal according to the present embodiment; The figure which shows the external appearance of the clip terminal in this embodiment. 4 is a flowchart for explaining the operation of the electronic device according to the embodiment; 4 is a flowchart for explaining the operation of the clip terminal according to the embodiment; 4 is a flowchart for explaining the operation of the clip terminal according to the embodiment; The figure which shows an example of the goods checklist in this embodiment. The figure which shows an example of the message in this embodiment. FIG. 4 is a diagram showing an example of an image captured by a camera according to the embodiment; FIG. 11 is a diagram showing an example of an object corresponding to a product detected from the image shown in FIG. 10; 4 is a flowchart showing registered position determination processing for determining whether confirmation is necessary based on the registered position according to the present embodiment. FIG. 4 is a diagram showing an example of the arrangement of product shelves in a store and the positions of carts;

Hereinafter, this embodiment will be described with reference to the drawings.
FIG. 1 is a system configuration diagram schematically showing the overall configuration of the system of this embodiment. In the system shown in FIG. 1, a system 10 provided at a headquarters and a store system 12 provided at each of a plurality of stores are connected via a dedicated line or a network 14 including the Internet.

The system 10 provided at the head office includes, for example, a head office system 10A, a customer system 10B, a settlement system 10C, a sales promotion system 10D, and the like, which are interconnected via a network 10E such as a LAN (Local Area Network).

The headquarters system 10A executes processing for controlling the store systems installed in each store. The customer system 10B manages customers who purchase products using the store system. The settlement system 10C executes processing (individual authentication, credit settlement, etc.) for settlement processing using a credit card or the like by a customer in the store system. The sales promotion system 10D performs management for providing information such as product advertisements for sales promotion and in-store guidance to customers (electronic devices used by customers) through the store system.

A store system 12 provided in a store is implemented as, for example, a POS (Point Of Sales) system. The store system 12 includes, for example, a store server 20A, a clip terminal management server 20B, a plurality of checkout machines 20C, a plurality of POS terminals 20D, etc., which are interconnected via a network 20E such as a LAN.

The store server 20A controls the store system 12 as a whole. The store server 20A stores data related to products sold in the store, such as product codes, product names, unit prices, weights, product locations sold in the store (product shelf positions), and product images for each product. data is stored. In the registration process, product data corresponding to the product code read by scanning the product is received from the store server 20A and stored, and payment information (including amount, product, quantity, etc.) for payment processing is received. can be generated. The clip terminal management server 20B confirms that the product (purchased product) that has been registered using the electronic device 30 as a purchase plan by the customer's operation and the product put in the cart 60 or basket as a purchase plan match. It manages the electronic device 50 (clip terminal, which will be described later) used to check whether the The checkout machine 20C executes payment processing for products registered using the electronic device 30 by the customer's operation. The POS terminal 20D executes registration processing for products purchased by a customer operated by a store clerk, settlement processing for products registered by the registration processing, and the like.

A plurality of access points (base stations) 20F are connected to the network 20E wired in the store. The access point 20F wirelessly communicates with various electronic devices used in the store, for example, by wireless LAN. Electronic devices used in the store include, for example, an electronic device 30 used by a customer, an electronic device 40 used by a store clerk (attendant), and an electronic device 50 (clip terminal, which will be described later) used while attached to a cart 60, for example. 50) are included.

The electronic devices 30 and 40 are, for example, smart phones, tablet personal computers (PCs), and the like. Based on an application program (including a check program) for using the services provided by the store system 12, the electronic device 30 performs registration processing for products to be purchased, registered products (purchased products), and purchased products. In cooperation with the electronic device 50 (clip terminal), an individual check process for checking whether or not the products put in the cart 60 or the basket match as planned is executed. The electronic device 40 allows a store clerk (attendant) to check that the product information registered by the registration process matches the product placed in the cart 60 based on a confirmation program for confirming the registration process executed by the electronic device 30. Execute confirmation processing and the like for easy confirmation.

The electronic device 30 and the electronic device 50 (clip terminal 50) may communicate via the clip terminal management server 20B (network 20E), or may communicate directly by short-range wireless communication. Furthermore, communication cables may be interconnected for wired communication.

FIG. 2 is a block diagram showing the configuration of the electronic devices 30 and 50 in this embodiment. The electronic device 30 shown in FIG. 2 will be described as being realized by, for example, a smart phone. It is also assumed that the electronic device 50 is configured as a dedicated clip terminal 50 for executing individual check processing in the store system 12 . The clip terminal 50 will be described below. Note that the clip terminal 50 can also be realized by installing a check program in a general-purpose device (computer) having a camera such as a smart phone.

The electronic device 30 is provided with a processor 30A, a memory 30B, a storage device 30C, a touch panel 30D, a camera 30E, a communication module 30F, and a battery 30G.
The processor 30A is a circuit that controls each part in order to realize various processes as the electronic device 30 by executing a program recorded in the memory 30B.

The memory 30B records programs executed by the processor 30A and data used when the processor 30A executes various processes. The programs recorded in the memory 30B include, for example, a registration processing program for executing registration processing for products to be purchased, products that have undergone registration processing (purchased products), and products that have been put into the cart 60 or basket as products to be purchased. contains a check program that performs an individual check process to check if the The data stored in the memory 30B includes payment information generated based on the product code read in the registration process, product information related to the product registered in the registration process, and the like.

The storage device 30C is realized by a hard disk (HDD (Hard disk drive)) device, an SSD (solid state drive) device, or the like, and stores data used when the processor 30A performs various processes, or data generated by the processing in the processor 30A. Record the data received.

The touch panel 30D is a device for input operation by the customer, and includes a display device and a touch sensor. The touch panel 30D displays an arbitrary screen such as a GUI (Graphical User Interface) screen associated with the processing of the processor 30A.
Camera 30E takes images under the control of processor 30A.

The communication module 30F controls communication with external electronic devices under the control of the processor 30A. The communication module 30F can control communication such as wireless communication with the access point 20F and short-range wireless communication with other electronic devices such as the clip terminal 50. FIG.

The clip terminal 50 is provided with a processor 50A, a memory 50B, a storage device 50C, a touch panel 50D, a camera 50E, a communication module 50F, a battery 50G, an I/O interface 50H, and a reader/writer 50J.

The processor 50A is a circuit that controls each part in order to realize various processes as the clip terminal 50 by executing programs recorded in the memory 50B.

The memory 50B records programs executed by the processor 50A as well as data used when the processor 50A executes various processes. The program recorded in the memory 50B includes a check program for realizing individual check processing for checking whether or not the registered product (purchased product) matches the product put in the cart 60 or basket as a planned purchase. is included. The check program includes an image for realizing image processing for detecting an object corresponding to the product from the image captured by the camera 50E and detecting that the product has been put into the cart 60 and taken out from the cart 60. A processing program is included. The data stored in the memory 50B includes product information registered by the registration processing of the electronic device 30, object management data (see FIG. 11) for managing objects detected by image processing, and products placed in the cart 60. Various data such as a product checklist (see FIG. 8) for checking is stored.

The storage device 50C is implemented by a hard disk device (HDD (Hard disk drive)), an SSD (solid state drive) device, or the like, and stores data used when the processor 50A performs various processes, or data generated by processing in the processor 50A. Record the data received.

The touch panel 50D is a device for input operation by the customer, and includes a display device and a touch sensor. Touch panel 50D displays an arbitrary screen such as a GUI (Graphical User Interface) screen associated with the processing of processor 50A.
Camera 50E takes images under the control of processor 50A.

The communication module 50F controls communication with external electronic devices under the control of the processor 50A. The communication module 50F can control communication such as wireless communication with the access point 20F and short-range wireless communication with other electronic devices such as the electronic device 30. FIG.

The I/O interface 50H is an interface for inputting signals from external equipment, and is connected to, for example, a weight sensor 601 (weighing scale, pressure sensor, weighing scale, etc.) attached to the basket bottom 60C of the cart 60 . The weight sensor 601 detects the weight of the product placed (placed) in the cart 60 and outputs a detection signal.

The reader/writer 50J communicates with RF (Radio Frequency) tags attached to each product sold in the store and product shelves in the store. The communication range of the reader/writer 50J is, for example, the range in which it can communicate with the RF tags of the products placed in the cart 60, or the range in which it can communicate with the RF tags attached to product shelves by moving the cart 60 within the store. do. The reader/writer 50J can be used to detect when a product is put into or taken out of the cart 60, or used to detect the position of the cart 60 to which the clip terminal 50 is attached in the store. can be done.

FIG. 3 is a diagram showing a usage example of the clip terminal 50 in this embodiment, and FIG. 4 is a diagram showing an appearance of the clip terminal 50 in this embodiment.

The clip terminal 50 is used, for example, by attaching it to the upper frame 60B of the cart 60 used in the store. As shown in FIG. 4, the clip terminal 50 is provided with a clip 60k that can be arbitrarily attached and detached. Therefore, it is not necessary to attach the clip terminal 50 to all the carts 60 prepared in advance in the store, and only the cart 60 used by the customer who uses the individual check processing service using the clip terminal 50 can be The clip terminal 50 may be attached as required. Therefore, in order to implement the individual check processing service, large-scale investment such as remodeling the store or exchanging the cart 60 is not required, and the service can be implemented at low cost.

The example shown in FIG. 3 shows an example in which the cart 60 is mounted on the upper frame 60B on the side of the handle 60A. When attaching the clip terminal 50, the camera 50E provided on the clip terminal 50 is directed toward the basket bottom 60C, and the photographing range is adjusted so that at least all the products put in the cart 60 can be photographed. be done. The photographing range of the camera 50E preferably includes the upper frame 60B so that the inside and outside of the cage can be clearly distinguished in the image photographed by the camera 50E. Furthermore, it is desirable that the photographing range of the camera 50E be capable of photographing, for example, the movement of the customer's arm (hand) so that the customer's movement of putting in and taking out products can be detected.

Moreover, as shown in FIG. 4, the clip terminal 50 is attached to the upper frame 60B so that the display surface of the touch panel 50D faces the customer. This facilitates confirmation of the image captured by the camera 50E and the information (message, etc.) indicating the check result of the individual check process using the captured image.

Next, operations of the electronic device 30 and the clip terminal 50 (electronic device) used in the store system 12 of this embodiment will be described. FIG. 5 is a flow chart for explaining the operation of the electronic device 30 in this embodiment. 6 and 7 are flowcharts for explaining the operation of the clip terminal 50 in this embodiment.

When the customer uses the electronic device 30 to perform the registration process, the customer activates an application program (check program) for using services provided by the store system 12 . Thereafter, the processor 30A of the electronic device 30 executes registration processing and individual check processing described below based on the application.

First, the electronic device 30 executes login processing for connecting to the store system 12 (Act 10). In the login process, the customer operates the camera 30E to read, for example, a bar code including a membership code specific to the member attached to the membership card (image capture). The processor 30A extracts the membership code from the bar code read by the camera 30E and transmits it to the clip terminal management server 20B. Based on the member code received from the electronic device 30, the clip terminal management server 20B checks the member data managed by the customer system 10B to authenticate the member.

When the clip terminal management server 20B notifies the electronic device 30 that the member has been authenticated, the electronic device 30 displays a selection screen of services available at the store. Services that can be used at the store include registration processing for processing of purchase plans, leaflet information display, bargain product guide display, store guide (map) display, use of the clip terminal 50, and the like. By using the clip terminal 50, a message is output from the clip terminal 50 when, for example, the product put in the cart 60 has not been registered. Therefore, the customer can prevent forgetting to register the product to be purchased. In addition, since the product to be purchased is checked by the clip terminal 50 while the customer is shopping around in the store, the burden of confirmation work by the attendant (clerk) after the customer completes the registration process is reduced. can be mitigated.

When the customer uses the clip terminal 50, the store system 12 provides the customer with a benefit (points, etc.) (stores data indicating the benefit to the customer in the customer system 10B). The use of the clip terminal 50 may be urged.

When using the clip terminal 50, the electronic device 30 and the clip terminal 50 are put into a communicable state. For example, the clip terminal 50 is attached to the cart 60 used by the customer (or the customer is allowed to use the cart 60 pre-attached with the clip terminal 50), and the link between the customer's electronic device 30 and the clip terminal 50 is set. For example, the electronic device 30 and the clip terminal 50 are paired so as to be able to communicate by short-range wireless communication. Alternatively, the unique code of the clip terminal 50 is read by the electronic device 30 and transmitted from the electronic device 30 to the clip terminal management server 20B. The clip terminal management server 20B distinguishes between the electronic device 30 and the clip terminal 50 that cooperate to execute the individual check process, and establishes the clip terminal management server 20B (network 20E) between the electronic device 30 and the clip terminal 50. to allow communication via Note that it is also possible to use other methods for connecting the electronic device 30 and the clip terminal 50 .

When the processor 30A of the electronic device 30 is connected to the clip terminal 50 (Act 11), the processor 30A enables the product registration process. Also, when the processor 50A of the clip terminal 50 is connected to the electronic device 30 (Act 40), it starts product detection processing. The product detection process is a process of detecting an object corresponding to the product from the image captured by the camera 50E. The processor 50A detects objects corresponding to products or the like from the image captured by the camera 50E by existing image processing, and stores the positions of the respective objects. When the processor 50A detects an object moving into the inside of the cart (inside the cart) from the area corresponding to the outside of the cart (outside the cart), for example, the processor 50A determines that the customer has taken the product in and out. Based on the changes in the objects existing in the cart before and after this action is detected, it is detected that the product has been put in or taken out. Also, even when the customer's motion is detected, if the number of objects corresponding to the products in the cart does not change, it is determined that the customer has rearranged the products in the cart, for example. Processor 50A stores the position of the repositioned object. The processor 50A stores the objects detected from the image taken by the camera 50E in the product checklist.

FIG. 8 is a diagram showing an example of a product checklist according to this embodiment. In the product checklist, as shown in FIG. 8, data of an object (detected object) detected from an image captured by the camera 50E and product information (registered product) registered by registration processing in the electronic device 30 are stored. Data (for example, D1 to D4) in which the data indicating the results of checking the match between the registered product and the product corresponding to the detected object (check results "OK" and "NG") are associated with each other are stored.

The customer uses the camera 30E of the electronic device 30 to scan the barcode attached to the product to register the product to be purchased while shopping around the store (Act 12). The processor 30A detects the barcode from the image captured by the camera 30E and extracts the product code from the barcode. The processor 30A acquires product information corresponding to the product code from the shop server 20A through the clip terminal management server 20B (network 20E), for example, and displays it on the touch panel 30D. That is, the product name, price, product image, etc. of the product read by product scanning are displayed for the customer to confirm. Here, when the customer performs a product registration instruction operation (for example, touches the product registration button displayed on the touch panel 30D) (Act 13, Yes), the processor 30A selects the product read by the product scan as the product to be purchased. It is stored as a registered product (Act 14). Also, the processor 30A transmits information on the registered product to the clip terminal 50 (Act 15).

On the other hand, when the clip terminal 50 receives the registered product information from the electronic device 30 (Act 42, Yes), the clip terminal 50 stores the registered product information in the product checklist (Act 43). If the processor 50A refers to the registered product information and detects that it is a large product, that is, if it determines that the product does not need to be (cannot be) put in the cart 60, the product checklist (Act 45). In the example shown in FIG. 8, a skip flag is set in association with the registered product "product C" as the data D2. The processor 50</b>A excludes the registered product “product C” for which the skip flag is set from the objects to be checked for matching between the registered product and the product placed in the cart 60 . Also, the processor 50A transmits a notification that the product is not required to be placed in the electronic device 30 (Act 57).

When the processor 30A of the electronic device 30 receives the product insertion unnecessary notification from the clip terminal 50 (Act 18, Yes), on the touch panel 30D, the registered product information (product name, product image, price, etc.) and the customer to display a message notifying that it is not necessary to put the product in the cart 60.例文帳に追加It should be noted that the message notifying that the product is not required to be placed may be displayed not only on the electronic device 30 but also on the touch panel 50</b>D of the clip terminal 50 .

On the other hand, if the registered product is not a large product, the processor 50A acquires product dictionary data for identifying the registered product by image processing, for example, from the clip terminal management server 20B through the network 20E. For example, when the processor 50A does not store the product dictionary data corresponding to the registered product, the processor 50A outputs a transmission request for the product dictionary data corresponding to the registered product to the clip terminal management server 20B. The clip terminal management server 20</b>B transmits the product dictionary data of the corresponding product to the clip terminal 50 in response to the transmission request from the clip terminal 50 . Note that when the electronic device 30 and the clip terminal 50 communicate via the clip terminal management server 20B, the clip terminal management server 20B, when receiving the registered product information from the electronic device 30, At the same time, the commodity dictionary data of the corresponding commodity may be transmitted to the clip terminal 50 . Moreover, the product dictionary data may be stored in the clip terminal 50 in advance.

Also, when the registered product is stored in the product checklist, the processor 50A starts measuring the elapsed time after storing the registered product. The processor 50A determines whether or not the preset time has elapsed until it is confirmed that the product corresponding to the registered product has been placed in the cart 60 (Act 46).

Merchandise to be purchased by a customer is often handled by a series of operations such as, for example, picking up the merchandise from a merchandise shelf or the like, scanning the merchandise using the electronic device 30 , and placing the merchandise on the cart 60 . Therefore, when a preset time has passed since the product scan (storage of the registered product in the clip terminal 50), there is a possibility that the registered product has been forgotten to be put in the cart 60, so that the customer can be notified. ing. When the processor 50A determines that the preset time has elapsed for the registered product (Act 46, Yes), the processor 50A transmits a product insertion request to the electronic device 30 (Act 47).

When the processor 30A of the electronic device 30 receives the product insertion request from the clip terminal 50 (Act 18, Yes), the touch panel 30D sends the registered product information (product name, product image, price, etc.) to the customer. In response, a product insertion request message is displayed to prompt the user to add the product to the cart 60 (Act 19). For example, as shown in FIG. 9A, the message "Please add the registered product to the cart" is displayed. The product insertion request message may be displayed not only on the electronic device 30 but also on the touch panel 50</b>D of the clip terminal 50 . In addition, if there are multiple products that are the target of the product insertion request message, it is possible to make sure that the customer can recognize the target product by displaying the information of the multiple target products in a list. good.

On the other hand, the processor 50A of the clip terminal 50 performs image processing on the image taken by the camera 50E, and when it detects that the customer has taken the product in and out of the cart, it detects the movement in the cart before and after the detection of this movement. Based on the changes in existing objects, it detects when a product is put in or taken out. Note that the processor 50A not only detects the action of putting in and taking out the product based on the image, but also determines the putting in and out of the product in combination with the change in weight detected by the weight sensor 601 at the timing when this action is detected. You can also

FIG. 10 shows an example of an image captured by the camera 50E. In the example shown in FIG. 10, for example, three commodities A, B, and E are placed in the cart 60 .

The processor 50A detects the position of the upper frame 60B of the cart 60 and the region of the basket bottom 60C from the image captured by the camera 50E. Since the clip terminal 50 is fixed to the cart 60, basically the areas of the upper frame 60B and the basket bottom 60C in the image do not change during shopping. When the processor 50A detects an object moving into the inside of the cart (inside the cart) from, for example, the area corresponding to the outside of the cart (outside the cart) with reference to the position of the upper frame 60B, the customer puts in and takes out the product. It is determined that the action was performed for the purpose.

Further, the processor 50A detects objects corresponding to products or the like from the image captured by the camera 50E by existing image processing, and stores the positions of the respective objects as object management data.

FIG. 11 shows an example of objects OJ-A, OJ-B, and OJ-E corresponding to products detected from the image shown in FIG. The processor 50A detects that a product has been put in or taken out based on changes in the objects existing in the cart before and after the product is put in and taken out by the customer.

When the object corresponding to the product is detected from the image (Act48, Yes), the processor 50A refers to the product checklist to determine whether the registered product with the check result of "NG" is stored. That is, it is determined whether there is any product that has been registered by the electronic device 30 but has not been confirmed to have been placed in the cart 60 . Here, if the registered product with the check result of “NG” is stored (Act 49, Yes), the processor 50A uses the product dictionary data corresponding to the registered product with the check result of “NG” to Recognition processing is executed for the object (Act 52). The product dictionary data is created so that the product can be recognized based on features such as the bar code attached to the product, package design, and characters.

Here, it is determined whether the object detected from the image is a registered product corresponding to the product dictionary data based on the processing result of the recognition processing using the product dictionary data. For example, when the feature of the image represented by the object and the feature indicated by the product dictionary data show similarity exceeding a preset reference value, it is determined that the product is a registered product corresponding to the product dictionary data.

Here, if the product is determined to be a registered product (Act 53, Yes), the processor 50A associates the product with the registered product previously stored in the product checklist corresponding to the product dictionary data used in the recognition process. , the object detected from the image is stored as the detected object (Act 54). Data indicating similarity (degree of similarity) in recognition processing may be stored as data related to the detected object. As a result, it is determined that the registered product corresponds to the product dictionary data, but since the degree of similarity is low, it is possible to prompt the attendant to confirm the product before settlement processing.

Also, the processor 50A sets the check result "OK" corresponding to the combination of the registered product and the detected object (Act 55), and stores that it has been confirmed that the registered product has been placed in the cart 60. FIG. In this case, the processor 50A transmits a product placement confirmation to the electronic device 30 (Act 56).

When the electronic device 30 receives the product insertion confirmation from the clip terminal 50 (Act 20, Yes), the electronic device 30 deletes the message from the touch panel 30D if the product insertion request message is displayed (Act 28). Thus, the customer can be notified that the clip terminal 50 has confirmed that the registered product has been placed in the cart 60 .

Note that if there are a plurality of registered products with the check result of "NG" in the product checklist, the processor 50A executes recognition processing on the object using the product dictionary data of each registered product. For objects that are not determined to be registered products by the recognition processing (Act 53, No), the processor 50A stores them in the product checklist as the detected objects for which the products are unregistered as the check result "NG" (Act 50, No). ). Also, when an object is detected from the image (Act 48, Yes), if the registered product with the check result of "NG" is not stored (Act 49, No), the processor 50A similarly determines whether the product is unregistered. As a detected object, the check result "NG" is stored in the product check list (Act 50).

Data D3 shown in FIG. 8 indicates that product C has already been registered in the electronic device 30, but the check result is set to "NG", and it is not confirmed that the product has been placed in the cart 60. there is Data D4 also indicates that unregistered products E and F have been detected as objects E and F, respectively, because they were first put in the cart 60 (check result “NG”).

When the processor 50A causes the product checklist to store a detected object with an unregistered product (check result “NG”) (Act 50), the processor 50A transmits a product registration request to the electronic device 30 (Act 51). When the product registration request is received from the clip terminal 50 (Act 21, Yes), the processor 30A of the electronic device 30 causes the touch panel 30D to display a product registration request message prompting the customer to register the product. For example, as shown in FIG. 9B, a product registration request message stating "There are unregistered products in the cart. Please register the products." is displayed. The product registration request message may be displayed not only on the electronic device 30 but also on the touch panel 50D of the clip terminal 50 (Act 22).

As described above, the product registration request message displayed on the touch panel 50D indicates that the product is placed in the cart 60 based on the recognition process using the product dictionary data of the registered product after the product has been registered. If it is confirmed (Acts 49-51), it is erased (Acts 20, 28).

On the other hand, when returning a product that has already been registered as a purchase plan, the customer uses the camera 30E of the electronic device 30 to scan the barcode attached to the product to be returned, in the same manner as when registering the product. (Act 12). The processor 30A detects the barcode from the image captured by the camera 30E and extracts the product code from the barcode. The processor 30A causes the touch panel 30D to display product information (product name, price, product image, etc.) corresponding to the product code so that the customer can confirm the product to be returned. Here, when the customer performs a product cancellation instruction operation (for example, touches the product cancellation button displayed on the touch panel 30D) (Act 13, No), the processor 30A removes the product read by the product scan from the purchase plan. Delete (Act 16). In addition, the processor 30A transmits a registration cancellation notice to the clip terminal 50 together with information on the product to be canceled (returned goods) (Act 17).

When the clip terminal 50 receives the registration cancellation notification from the electronic device 30 (Act 60, Yes), it determines the corresponding registered product from the product checklist based on the information of the product to be canceled (Act 61), and confirms the registered product. is deleted (Act 62). The processor 50A transmits a cancellation completion notification to the electronic device 30 (Act 63). When the electronic device 30 receives the cancellation completion notification (Act 25, Yes), if the product insertion request message is displayed, it erases the product insertion request message (Act 28). For example, as described above, the electronic device 30 displays a product insertion request message when a set time elapses without the product being placed in the cart 60 after product registration processing. When the customer performs a product cancellation operation for a product for which registration processing has been performed, the electronic device 30 erases the product insertion request message for the corresponding product.

When the customer takes out the product from the cart 60, the processor 50A of the clip terminal 50 detects that the customer has taken the product in and out based on the image captured by the camera 50E. Detect objects corresponding to the products that have been discriminated and extracted by existing image processing. It should be noted that it is also possible to determine that the product has been put out by combining the change in weight detected by the weight sensor 601 in the same way as detecting an object when the product is put in.

When the processor 50A detects that the object (product) has been taken out of the cart 60 (Act 64, Yes), the registered product corresponding to this detected object is stored in the product checklist (check result "OK"). Here, if the corresponding registered product is not stored (Act 65, No), the processor 50A deletes the detected object corresponding to the taken-out product from the product checklist (Act 68).

On the other hand, if the corresponding registered product is stored (Act65, Yes), the processor 50A transmits a product cancellation request to the electronic device 30 (Act66). Also, the processor 50A sets the check result "NG" for the detected object corresponding to the product that was taken out (Act 67).

When the product cancellation request is received from the clip terminal 50 (Act 23, Yes), the processor 30A of the electronic device 30 causes the touch panel 30D to display a product cancellation request message prompting the customer to operate to cancel the product (Act 24). . For example, as shown in FIG. 9(C), a product cancellation request message of "The product has been taken out of the cart. If you want to return the product, please perform the return operation." The product cancellation request message may be displayed not only on the electronic device 30 but also on the touch panel 50</b>D of the clip terminal 50 .

Although the data stored in the product checklist is immediately updated when the product is taken out of the cart 60, after a predetermined set time (for example, 10 seconds) elapses, the product is canceled ( The data may be updated assuming that there is a high possibility of returned goods. In this case, the product cancellation request message may be displayed on the electronic device 30 immediately after the product is taken out, or may be displayed after a set time has elapsed.

In this way, the customer attaches the clip terminal 50 to the cart 60 and makes purchases while performing the registration process using the electronic device 30, thereby preventing the customer from forgetting to register the product to be purchased and forgetting to put the product in the cart 60. Even if there is, it can be easily confirmed by the message displayed on the electronic device 30 .

When the customer completes the registration process for the product to be purchased, the customer instructs the end of the product registration by operating the subtotal button, for example. When the customer instructs the end of the product registration (Act 26, Yes), the processor 30A of the electronic device 30 displays a confirmation screen (product list) of the registered products and also displays a payment method (credit, cash, direct debit). , electronic money, etc.). The processor 30A generates payment information based on the product information registered by the registration process.

In addition, the processor 30A of the electronic device 30 transmits a registration processing end notification to the clip terminal 50 (Act 27). When the processor 50A of the clip terminal 50 receives the registration processing end notification from the electronic device 30 (Act 69, Yes), the processor 50A transmits the product checklist data to the electronic device 30 (Act 70). This data also includes, for example, data indicating the degree of similarity when the detection object is recognized. After transmitting the product checklist data, the processor 50A clears the product checklist data and terminates the process (Act 71).

When the end of the registration process is instructed, if the check result "NG" data remains in the data stored in the product checklist, a message is sent electronically to prompt the customer to confirm again. It is displayed on the touch panel 30D of the device 30 (the touch panel 50D of the clip terminal 50).

When the customer instructs the final confirmation, the processor 30A of the electronic device 30 transmits the payment information and a code (a bar code or a two-dimensional code) including data for prompting the confirmation of the attendant included in the product checklist to the touch panel. Display on 30D.

The attendant uses the camera of the electronic device 40 to scan the code displayed on the touch panel 30</b>D of the electronic device 30 . The electronic device 40 displays a confirmation screen corresponding to the payment information included in the code and the data requiring confirmation. For example, the electronic device 40 displays a list of products that the customer has registered and plans to purchase (including product names, prices, subtotal amounts, etc.), the number of products, and the like. Thereby, the attendant can confirm whether the product put in the cart 60 and the registered product match. Furthermore, the electronic device 40 clearly displays the product corresponding to the detected object whose similarity is low in the recognition processing of the clip terminal 50, for example, based on the data that needs to be confirmed. As a result, the burden of confirmation work can be reduced by making it possible for the attendant to easily recognize commodities that particularly require confirmation.

In the above description, an example of checking the match between the registered product and the product put in the cart 60 using the image processing in the clip terminal 50 is explained, but the weight of the product detected by the weight sensor 601 is You may make it check goods using also. For example, since the product data provided by the store server 20A includes product weight data, the weight added when the product is placed in the cart 60 matches the weight of the registered product within an allowable range. can be checked based on In particular, it is effective to use the weight data detected by the weight sensor 601 for a weighed product whose price varies depending on the weight, in order to make a correct check. For example, it is possible to check whether or not an item heavier than the registered item of the same kind is added to the cart 60 after the registration process has been performed using an item with a smaller weight. The product check list stores data indicating that confirmation is required when there are weighed products and when the difference between the weight of the registered product and the weight when put in the cart 60 is greater than a set value. Keep Thereby, as described above, it is possible to make the attendant recognize that confirmation is especially required.

Furthermore, in the above description, it is assumed that the product registered by the registration process matches the product put in the cart 60, but the clip terminal 50 can also be used to check the registration of other products. It is possible. For example, the position in the store at the timing of product registration in the electronic device 30 or the timing at which the product (detection object) is detected by the clip terminal 50, and the position of the product shelf in the store where the target product is placed (product position) is greater than a preset determination value, it is also possible to determine that confirmation is required.

FIG. 12 is a flowchart showing registered position determination processing for determining whether confirmation is necessary based on the registered position in this embodiment. The processor 50A of the clip terminal 50 executes registration position determination processing based on the check program. It should be noted that the registered position determination process is executed in parallel with the above-described individual check process.

The processor 50A receives the product information of the registered product from the electronic device 30 (Act80, Yes), or if it is determined which registered product the detected object corresponds to (check result "OK") (Act81 , Yes), the current position is detected (Act 82). The current position can be detected, for example, by communicating with the RF tag attached to each product shelf in the store using the reader/writer 50J.

FIG. 13 is a diagram showing an example of the arrangement of product shelves 70-1 to 70-5 in the store and the position of the cart 60. As shown in FIG. When the clip terminal 50 attached to the cart 60 can communicate with the RF tag 72-5 attached to the product shelf 70-5 through communication within a limited range by the reader/writer 50J, the product shelf 70 It can be determined when it is near -5. The current position of the cart 60 (clip terminal 50) can be detected not only by the method using the reader/writer 50J (RF tag), but also by the difference in radio wave intensity between the communication module 50F and the plurality of access points 20F, It is possible to use a method such as image processing to detect the cart 60 for the image taken.

After detecting the current position, the processor 50A obtains data indicating the product position (product shelf position) where the registered product is next sold, for example, through the store server 20A or the electronic device 30 (Act 83). The processor 50A determines whether the difference between the current position and the product position (position on the product shelf) is greater than a preset determination value (Act 84). For example, when the product 74 placed on the product shelf 70-1 shown in FIG. 13 is registered or placed in the cart 60 at a position near the product shelf 70-5, the customer performs an unusual action. It is determined that confirmation is required as a thing (Act 85, Yes). The clip terminal 50 stores the confirmation data associated with the data corresponding to the corresponding registered product and the detected object in the product checklist (Act 86). Thereby, as described above, it is possible to make the attendant recognize that confirmation is especially required.

According to at least one of the embodiments described above, by using the clip terminal 50, the product information registered by the registration processing by the electronic device 30 and the products put in and out of the cart 60 detected by the clip terminal 50 can be combined. Corresponding detected objects are checked for matches with corresponding products. Since a message prompting the customer for confirmation can be appropriately displayed according to the check result, it is possible to reliably register the product to be purchased by the customer's operation.

While several embodiments of the invention have been described, these embodiments have been presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

Further, the processing described in the above-described embodiments can be performed by using a recording medium such as a magnetic disk (flexible disk, hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), a semiconductor memory, etc. as a program that can be executed by a computer. can be written to and provided to various devices. Moreover, it is also possible to transmit it by a communication medium and provide it to various devices. The computer reads the program recorded on the recording medium or receives the program via the communication medium, and executes the above-described processes by being controlled by the program.

DESCRIPTION OF SYMBOLS 12... Store system, 20A... Store server, 20B... Clip terminal management server, 30, 40... Electronic equipment, 50... Clip terminal (electronic equipment), 50A... Processor, 50B... Memory, 50C... Storage device, 50D... Touch panel, 50E...Camera, 50F...Communication module, 50J...Reader/writer, 60...Cart, 601...Weight sensor.

Claims (8)

detection means for detecting a current position when it is determined to which product the detection object detected from the image corresponds;
determination means for determining whether the difference between the current position and the product position where the product is placed is greater than a preset determination value;
and storage means for storing data associating the product with the detected object when the difference is greater than the judgment value. detection means for detecting a current position when product information is received from an electronic device;
determination means for determining whether the difference between the current position and the product position where the product corresponding to the product information is placed is greater than a preset determination value;
and storage means for storing data in which the detected object detected from the image is associated with the product when the difference is greater than the judgment value. list storage means for storing the detected objects in a list;
3. The checking device according to claim 1, further comprising transmitting means for transmitting a product registration request to an electronic device when said detected object is stored in said list. 3. The checking device according to claim 1, wherein said detecting means detects said current position by communication with an RF tag attached to the position of the article where the article is placed. 3. The checking device according to claim 1, wherein said detecting means detects said current position based on a difference in radio field intensity between a communication module and a plurality of access points. 3. The checking device according to claim 1, wherein said detecting means detects said current position by image processing of an image taken inside a store where said merchandise is sold. the computer,
detection means for detecting a current position when it is determined to which product the detection object detected from the image corresponds;
determination means for determining whether the difference between the current position and the product position where the product is placed is greater than a preset determination value;
A check program for functioning as storage means for storing data that associates the product with the detection object when the difference is greater than the judgment value. the computer,
detection means for detecting a current position when product information is received from an electronic device;
determination means for determining whether the difference between the current position and the product position where the product corresponding to the product information is placed is greater than a preset determination value;
A check program for functioning as storage means for storing data in which the detected object detected from the image is associated with the product when the difference is greater than the judgment value.

Images