JP2023179088A - Program, information terminal, and system - Google Patents

Abstract

To implement the registration of articles using mobile terminals in a suitable manner.SOLUTION: A program for allowing a computer to function as an information terminal used for registering articles to be purchased includes: image acquisition means for acquiring images; registration means for registering articles based on article identification information in the images acquired by the image acquisition means; vibration detecting means for detecting vibration of the information terminal; and output means for outputting information on the purchase of the articles to an operator of the information terminal. The output means outputs information on the purchase of the articles when a predetermined vibration is detected by the vibration detection means and the articles have not been registered by the registration means before the detection of the vibration.SELECTED DRAWING: Figure 1

Description

The present invention relates to a program, an information terminal, and a system.

2. Description of the Related Art A system is known in which a customer registers a product by himself/herself using a mobile terminal (see, for example, Patent Document 1).

Japanese Patent Application Publication No. 2018-147252

However, there is room for improvement in the above-described system, for example, in measures against malicious customers or inexperienced customers.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a technique for suitably registering products using a mobile terminal.

A program that is one aspect of solving the above-mentioned problem is a program that causes a computer to function as an information terminal used for registering a product to be purchased, and the program is a program that causes a computer to function as an information terminal that is used for registering a product to be purchased, and that includes an image acquisition unit that acquires an image, and an image acquisition unit that acquires an image. as a registration means for registering a product based on the product identification information in the image acquired by , a vibration detection means for detecting the vibration of the own information terminal, and an output means capable of outputting information regarding the purchase of the product to the operator of the own information terminal. and the output means outputs information regarding the purchase of the product when a predetermined vibration is detected by the vibration detection means and the product is not registered by the registration means before the detection of the vibration. This is a program featuring the following.

It is a conceptual diagram of a network for explaining a sales system. FIG. 2 is an explanatory diagram illustrating a cart to which a mobile terminal can be attached. It is a sequence diagram explaining the outline of operation of a mobile terminal, a cloud server, and a payment device. This is an example of a display on a mobile terminal. This is an example of a display on a mobile terminal. It is a flowchart explaining an example of operation of a portable terminal. It is a flowchart explaining an example of operation of a portable terminal. It is a flowchart explaining an example of operation of a portable terminal. It is a flowchart explaining another example of operation of a portable terminal. It is a flowchart explaining another example of operation of a portable terminal. It is a flowchart explaining another example of operation of a portable terminal.

(First embodiment)
FIG. 1 is a conceptual diagram of a network for explaining a sales system 1 according to the first embodiment (the second embodiment is also the same). The sales system 1 shown in FIG. 1 includes a headquarters server 10, a cloud server 20, a store controller 30, a transaction status management device 40 (for example, a personal computer, etc.), a payment device 50, and a mobile terminal 60.

The sales system 1 can be introduced into various types of businesses (supermarkets, convenience stores, drug stores, home centers, home electronics retailers, etc.). In FIG. 1, a broken line T indicates the inside of the store (product display area, checkout area, backyard, etc.). As shown in FIG. 1, a store controller 30, transaction status management device 40, and payment device 50 are installed within the store. These are communicably connected via a LAN 19 (which may be wired or wireless). A mobile terminal 60 is present in the store. The mobile terminal 60 is basically used while being attached (set) to a cart (shopping cart) 70. The headquarters server 10 and the cloud server 20 are installed outside the store (for example, at the headquarters, data center, etc.). Devices installed inside the store (store controller 30, transaction status management device 40, payment device 50) and devices installed outside the store (headquarters server 10, cloud server 20) communicate (directly communicate with each other). , or communication via the store controller 30).

Note that in FIG. 1, the number of each device (terminal) is an example. For example, in FIG. 1, "..." is shown on the right side of the payment device 50, but the number of payment devices 50 installed in the store may be one or more. For example, although two mobile terminals 60 are shown in FIG. 1, the number of mobile terminals 60 used in the store changes depending on the number of customers. Additionally, other devices may be installed within the store. For example, in a store, there are semi-self devices (registration devices, payment devices) that register products based on the operations of store staff and make payments based on the operations of customers, and devices that register products based on operations of customers. A fully self-service device (registered payment device) that performs payment based on customer operations may be installed.

(Mobile terminal 60)
Mobile terminal 60 is used by a customer. Specifically, the mobile terminal 60 is used when a customer registers a product in the store. The mobile terminal 60 is, for example, a smartphone, but may also be a tablet terminal. The mobile terminal 60 may be owned by the customer or may be lent by the store.

Although not shown, the mobile terminal 60 includes, for example, a CPU, a storage unit, a display unit, an operation unit, an audio output unit (speaker), an audio input unit (microphone), an imaging unit (camera), and a communication unit. A vibration detection unit that detects vibration (displacement, speed, or acceleration) of the mobile terminal 60 (casing) is provided. The vibration detection unit may be any sensor (device) that can detect vibrations of the mobile terminal 60 (own mobile terminal 60), such as an acceleration sensor or a gyro sensor. When the vibration detection section detects vibration, it outputs the detection result to the CPU. For example, the vibration detection section outputs the magnitude of the detected vibration to the CPU as a detection result. Note that the magnitude of vibration is sometimes referred to as a vibration value (or vibration amount).The magnitude of vibration (vibration value) detected by the vibration detection unit can be any of displacement, velocity, or acceleration. In addition, the vibration detection unit detects vibrations caused by, for example, movement (running) of the cart 70, in addition to vibrations caused by putting products into the cart 70 (described later).The mobile terminal 60 further includes a light emitting unit ( light) may be provided.

The CPU is a central processing unit, and controls the entire mobile terminal 60 by reading and executing programs stored in the storage unit. The storage unit stores various information (for example, information referenced or generated by the CPU, information acquired from the outside, etc.). The storage unit is a general term for one or more various storage members (for example, memory or storage such as RAM, ROM, hard disk, SSD (Solid State Drive), etc.). The display unit is, for example, a liquid crystal display, and displays various information (including image information). The display unit is configured as a touch panel that can accept operations on the display screen. The operation unit receives operations from an operator (customer). The operation unit is a general term for one or more various operation members (various operators (hard buttons) provided on the housing.The display unit is configured as a touch panel, so it can be considered as a part of the operation unit. The audio output unit outputs audio. The audio input unit inputs audio. The imaging unit acquires captured images (still images, moving images). The communication unit transmits and receives various information. The communication unit is a general term for one or more various communication members (for example, communication members for mobile wireless communication such as 5G, wireless LAN, short-range wireless communication, non-contact communication, etc.).

The mobile terminal 60 includes a code recognition function (scanner function) that reads various codes (for example, barcodes such as JAN codes) from captured images acquired by the imaging unit. Registration by the mobile terminal 60 is realized by the code recognition function. Furthermore, the mobile terminal 60 executes control according to the detection of vibration by the vibration detection unit, the status of processing, and the like. That is, the mobile terminal 60 executes control according to the situation when vibration is detected, and control according to whether vibration is detected in a certain situation. Details will be described later.

FIG. 2 is an explanatory diagram illustrating a cart 70 to which a mobile terminal 60 can be attached. As shown in FIG. 2, the cart 70 has a shopping basket placement section 71 on which a shopping basket 80 (not shown) can be placed, and a holder section 72 on which a mobile terminal 60 can be attached. Although it has been explained that the mobile terminal 60 is basically used while being attached to the cart 70, specifically, it is used while being attached to the holder part 72.

In the example shown in FIG. 2, the shopping basket 80 is not placed on the shopping basket placement section 71, and the mobile terminal 60 is not attached to the holder section 72. The shopping basket 80 is placed on the shopping basket placement section 71, and the mobile terminal 60 is attached to the holder section 72. Additionally, a special bag 90 is attached to the cart 70 for temporarily storing products that require confirmation by a store clerk at the time of checkout (pending products that are held for take-out by the customer or return to the store). .

When the mobile terminal 60 is attached to the holder part 72, the imaging part acquires a captured image (through image (live view)) in a direction looking down on the shopping basket 80 from the position of the holder part 72. When a customer finds a desired product (product to purchase) in the product display area (display shelf), the customer picks up the product and places it in a holder so that the barcode attached to the product can be read by the scanner function. part 72 (space between holder part 72 and shopping basket 80).

(scan operation)
The operation of bringing the product below the holder section 72 so that the barcode attached to the product can be read by the scanner function is referred to as a product scanning operation (or simply scanning operation). Note that FIG. 4(B) shows a scene where a scanning operation is being performed. The mobile terminal 60 acquires a product code (product identification information) through a scanning operation.

The mobile terminal 60 may recognize the product scanning operation. In other words, the mobile terminal 60 may determine whether or not a product scanning operation is being performed (that is, the presence or absence of a scanning operation). For example, the mobile terminal 60 uses an image recognition function using image recognition technology (for example, pattern recognition technology, etc.) to perform a scanning operation based on whether or not something that looks like a product, a hand, or a barcode is recognized in the captured image. You may decide whether or not it exists.

(Payment device 50)
The payment device 50 is used by a customer. Specifically, the payment device 50 pays for the products registered by the mobile terminal 60 based on the customer's operation. In other words, the sales system 1 provides a service in which the customer registers the product himself using the mobile terminal 60 and pays the bill himself using the settlement device 50.

Although not shown, the payment device 50 includes a CPU, a storage unit, a display unit, a scanner unit, a card payment unit (non-cash payment unit), a change machine (cash payment unit), a printing unit, an audio output unit, an imaging unit, Equipped with communication section and sign pole. Note that the payment device 50 may further include a key operation section made up of one or more various operation members, a sensor (a human sensor, etc.) that detects (detects, recognizes) the presence of a person, and the like.

The CPU is a central processing unit, and controls the entire payment device 50 by reading and executing programs stored in the storage unit. The storage unit stores various information. The storage unit is a general term for one or more various storage members. The display unit is, for example, a liquid crystal display, and displays various information. The display unit is configured as a touch panel that can accept operations on the display screen. The scanner section scans various codes (for example, payment codes (described later), product codes, store clerk codes attached to gift certificates, point cards, store clerk name tags, etc.) and reads each code.

The card payment unit is a payment mechanism using various cards (credit cards, prepaid cards such as transportation cards, cash cards for debit payments, point cards for point payments, etc.). A change machine is a payment mechanism using cash. The printing unit prints and issues various media (receipts, etc.). The audio output unit outputs audio (eg, confirmation sound, warning sound, audio guidance, etc.). The imaging unit (camera) acquires captured images (still images, moving images). The communication unit is a general term for one or more various communication members. The sign pole has a light emitting section at its tip, and depending on the light emitting mode of the light emitting section, it notifies various information (such as a call to a store clerk, operating status, etc.).

(Headquarters server 10)
The headquarters server 10 is a server that manages the sales system 1 as a whole. The headquarters server 10 stores, for example, various information (for example, a product master file (product master), etc.) and various performance information (for example, sales performance information, product sales performance information, etc.). Further, the headquarters server 10 supplies various information (for example, product master etc.) to the cloud server 20 and store controller 30, for example. The product master is a file that stores product names, prices, etc. in association with product codes. Although not shown, the headquarters server 10 includes at least a CPU, a storage section, and a communication section. The headquarters server 10 may be composed of a plurality of servers.

(Cloud server 20)
The cloud server 20 is a server that mainly supports product registration by the mobile terminal 60. The cloud server 20 stores various information (eg, product master, transaction information (cart information), etc.). Further, the cloud server 20 supplies various information (eg, screen information, etc.) to the mobile terminal 60 and the payment device 50, for example. Although not shown, the cloud server 20 includes at least a CPU, a storage unit, and a communication unit. The cloud server 20 may be composed of multiple servers.

(Store controller 30)
The store controller 30 mainly stores information between devices installed inside the store (transaction status management device 40, settlement device 50) and devices installed outside the store (headquarters server 10, cloud server 20). This is a server in charge of communication. Although not shown, the store controller 30 includes at least a CPU, a storage section, and a communication section.

(Transaction status management device 40)
The transaction status management device 40 is a device (for example, a personal computer) that manages (monitors and controls the processing status, operation status, etc.) devices installed in the store (store controller 30, payment device 50). Although not shown, the transaction status management device 40 includes at least a CPU, a storage section, a display section (which may be configured as a touch panel), an operation section (keyboard, mouse, etc.), and a communication section. The transaction status management device 40 can communicate with a printing device (not shown), and may output various information from the printing device.

In addition to managing the devices installed in the store, the transaction status management device 40 also accesses the head office server 10, etc., and maintains the product master stored in the head office server 10, etc. (product settings, etc.). Alternatively, the transaction information etc. stored in the cloud server 20 may be displayed or modified by accessing the cloud server 20.

(Summary of operations of mobile terminal 60, cloud server 20, and payment device 50)
FIG. 3 is a sequence diagram illustrating an overview of the operations of the mobile terminal 60, the cloud server 20, and the payment device 50. The sequence diagram in FIG. 3 shows that after the mobile terminal 60 displays the initial registration screen (registration screen before registering one item of product; the registration screen in FIG. 4A), the payment device 50 displays the product list screen ( The operation of each device (mobile terminal 60, cloud server 20, payment device 50) is shown until displaying the information (not shown). Note that since the sequence diagram in FIG. 3 mainly explains the cooperation of each device, the sequence diagram in FIG. .

Note that the mobile terminal 60 may, for example, display a service start screen (menu screen, not shown) based on a customer's operation, and display an initial registration screen based on an operation of a shopping start instruction button on the service start screen. do. In addition, when displaying the initial registration screen (based on the operation of the shopping start instruction button), transaction identification information that identifies the transaction (or transaction identification information of the transaction) by the mobile terminal 60 is acquired from the cloud server 20 ( (receiving).

Step S1: The mobile terminal 60 acquires the product code through the customer's scanning operation.
Step S2: The mobile terminal 60 transmits the transaction identification information of the transaction and the product code acquired in step S1 to the cloud server 20.

Step S3: Cloud server 20 receives transaction identification information and product code from mobile terminal 60.
Step S4: The cloud server 20 updates the transaction information of the transaction (the transaction corresponding to the transaction identification information received in step S) based on the product code. That is, the cloud server 20 acquires (extracts) the product name and price corresponding to the product code received in step S3 from the product master, and stores them in the transaction information of the transaction.
Step S5: The cloud server 20 transmits the updated transaction information to the mobile terminal 60. For example, the cloud server 20 generates a registration screen (for example, the registration screen in FIG. 4C) that reflects the updated transaction information, and transmits the screen information of the registration screen to the mobile terminal 60.

Step S6: The mobile terminal 60 receives screen information (transaction information) from the cloud server 20.
Step S7: The mobile terminal 60 displays a registration screen based on the screen information received in step S6. Specifically, in the case of the first product, the mobile terminal 60 displays a registration screen that reflects the first product in place of the initial registration screen, and in the case of the second or subsequent products, the mobile terminal 60 displays a registration screen that reflects the first product. The terminal 60 displays a registration screen that reflects the second and subsequent products.

After step S7, when the product code is obtained by the customer's scanning operation (when returning to step S1), and when the customer operates the checkout instruction button BT630 (see FIG. 4(C)), a checkout code is generated and displayed. There is a case where the process proceeds to step S8. That is, steps S1 to S7 are repeatedly executed every time a product code is obtained.

Step S8: The mobile terminal 60 generates and displays a payment code according to the customer's operation of the payment instruction button BT630. The payment code is coded (for example, two-dimensional coded) the payment information. The settlement information is information necessary for acquiring transaction information of the transaction from the cloud server 20 (for example, transaction identification information that identifies the transaction).

The customer causes the payment device 50 to read the payment code displayed on the mobile terminal 60. It is assumed that the payment device 50 displays a guide screen for guiding the reading of the payment code (reading operation) in the standby state.

Step S9: The payment device 50 reads the payment code displayed on the mobile terminal 60 based on the customer's reading operation. That is, the settlement device 50 acquires transaction identification information that identifies the transaction.
Step S10: The settlement device 50 requests transaction information of the transaction from the cloud server 20. For example, the payment device 50 transmits request information (transaction information acquisition request information) including transaction identification information that identifies the transaction to the cloud server 20.

Step S11: The cloud server 20 receives the request (transaction information acquisition request information).
Step S12: The cloud server 20 transmits the transaction information of the transaction (the transaction corresponding to the transaction identification information included in the transaction information acquisition request information received in step S11) to the settlement device 50. For example, the cloud server 20 generates a product list screen (not shown) that reflects the transaction information of the transaction, and transmits the screen information of the product list screen to the payment device 50.

Step S13: The payment device 50 and the mobile terminal 60 receive screen information (transaction information) from the cloud server 20.
Step S14: The payment device 50 displays a product list screen based on the screen information received in step S13.

(Example of display on mobile terminal 60)
4 and 5 are display examples of the mobile terminal 60. It is assumed that the mobile terminal 60 is already attached to the cart 70 (holder section 72). Note that when the scanning operation is not performed, the captured image of the direction looking down on the shopping basket 80 from the position of the holder part 72 includes images of the shopping basket 80 and the products that have already been placed in the shopping basket 80, and when the scanning operation is not performed. When the scanning operation is being performed, images of the target product of the scanning operation and the shopping basket 80 and the products already placed in the shopping basket 80 are captured behind the target product, but in FIGS. 4 and 5, for simplicity, Items other than the target items of the scanning operation (such as the shopping basket 80 and items already placed in the shopping basket 80) are not shown.

FIG. 4A shows a registration screen (initial registration screen) in a state where no scanning operation is performed. Note that, for example, the mobile terminal 60 displays a service start screen (menu screen, not shown) based on the customer's operation, and displays a registration screen (initial registration screen) based on the operation of a shopping start instruction button on the service start screen. indicate.

The registration screen in FIG. 4A (the same applies to the registration screens in FIGS. 4B and 4C) is provided with a captured image display area HR610 and a registered information display area HR620. The captured image display area HR610 is an area that mainly displays captured images. The registration information display area HR620 is an area that mainly displays product registration information.

FIG. 4(B) is a scene following the scene of FIG. 4(A). Specifically, FIG. 4B shows a scene where a customer is scanning a product (white bread).

FIG. 4(C) is a scene following the scene of FIG. 4(B). Specifically, FIG. 4(C) shows a scene where a product (white bread) is registered by the scanning operation shown in FIG. 4(B). That is, the mobile terminal 60 that has acquired the product code through the scanning operation transmits the product code, etc. to the cloud server 20 (step S2 in FIG. 3), and displays a registration screen as shown in FIG. 3 step S6).

After registering the product (one item of product), the mobile terminal 60 clicks the checkout instruction button BT630 (in this example, a button displaying "Proceed to checkout") that accepts the checkout instruction operation, as shown in FIG. 4(C). ) to be displayed in an operational manner. The customer operates the checkout instruction button BT630 when proceeding to payment (accounting) (that is, when completing product registration).

FIG. 5(A) shows the scene of FIG. 4(A) or another scene following FIG. 4(B). The mobile terminal 60 is in a state where the scanning operation as shown in FIG. 4(B) is not performed, or in a state where the scanning action is performed as shown in FIG. 4(B) but no product is registered. In this case, when the vibration detection unit detects a vibration equal to or higher than the second threshold, which is larger than the first threshold, a small screen SG10 is displayed in front of the registration screen (FIG. 4(A)), as shown in FIG. 5(A). indicate.

(first threshold, second threshold, accuracy)
The first threshold value and the second threshold value will be explained. The first threshold value and the second threshold value are values that are compared with the magnitude of vibration (vibration value) detected by the mobile terminal 60 (vibration detection section), respectively. The mobile terminal 60 uses the first threshold value and the second threshold value to determine (determine) the possibility that the product is placed in the cart 70 (shopping basket 80) (product insertion possibility). Specifically, the mobile terminal 60 uses the first threshold value and the second threshold value to classify the possibility of product introduction into three stages (no product introduction, product introduction (low probability), product introduction possibility (high probability)). to decide. “No product input” is the determination result that no product is input. “Product inserted (low accuracy)” is a judgment result in which it was determined that a product was inserted, and the accuracy of the judgment result (determination that a product was inserted) was determined to be relatively low. “Product inserted (high accuracy)” is a judgment result in which it was determined that a product was inserted, and the accuracy of the judgment result (determination that a product was inserted) was determined to be relatively high.

If the detected vibration value is less than the first threshold value, the mobile terminal 60 determines that no product has been inserted. If the detected vibration value is greater than or equal to the first threshold and less than the second threshold, the mobile terminal 60 determines that a product has been inserted (low accuracy). If the detected vibration value is greater than or equal to the second threshold, the mobile terminal 60 determines that the product has been inserted (high accuracy). That is, the first threshold value is a threshold value for determining whether or not a product has been inserted, and the second threshold value is a threshold value for determining the accuracy of the determination itself when it is determined that a product has been inserted.

Note that the first threshold (same as the second threshold) may be determined by experiment, for example. For example, the vibration value detected by the vibration detection unit when the product is actually placed in the cart 70 (shopping basket 80) (vibration value at the time of product insertion), and the vibration value detected by the vibration detection unit when the cart 70 is moved etc. A large number of detected vibration values (vibration values when no product is inserted) are collected, and a first threshold value and a second threshold value are set based on the distribution of vibration values when a product is inserted and the distribution of vibration values when a product is not inserted. A threshold value may be determined.

As an example, according to the distribution of vibration values when a product is inserted and the distribution of vibration values when a product is not inserted, vibrations below a certain vibration value (vibration value A) are vibrations when the product is placed in the cart 70. If there is a very low possibility that the vibration is caused by moving the cart 70, etc., the vibration value A may be set as the first threshold value. Furthermore, according to the distribution of vibration values when a product is inserted and the distribution of vibration values when a product is not inserted, vibrations greater than a certain vibration value (vibration value B) can be caused by vibrations when the product is placed in the cart 70. If the probability of vibration being caused by moving the cart 70 is extremely low, the vibration value B may be set as the second threshold value.

Return to FIG. 5(A). The small screen SG10 includes a message MS11 informing that the product should be placed in the special bag 90, an image GA12 informing that the product should be placed in the special bag 90, and a message informing that the product could not be registered. MS13 and a message MS14 notifying that a store clerk will handle the payment at the payment machine (payment device 50) are displayed. Further, an OK button BT15 is displayed in an operable manner on the small screen SG10.

The customer puts the product into the special bag 90 and operates the OK button BT15 (the customer may put the product into the special bag 90 after operating the OK button BT15). The small screen SG10 is erased by operating the OK button BT15 (the display becomes as shown in FIG. 4(A)).

FIG. 5(B) is the scene of FIG. 4(A) or another scene following FIG. 4(B). The mobile terminal 60 is in a state where the scanning operation as shown in FIG. 4(B) is not performed, or in a state where the scanning action is performed as shown in FIG. 4(B) but no product is registered. In this case, when the vibration detection unit detects a vibration that is equal to or greater than the first threshold value and less than the second threshold value, as shown in FIG. 5(B), a small screen SG20 is displayed in front of the registration screen (FIG. 4(A)). Display.

Displayed on the small screen SG20 are a message MS21 informing that it is a confirmation, a message MS23 informing that the product could not be registered, and a message MS24 confirming whether or not to register the product. . Further, on the small screen SG20, a button BT25 for not registering the product and a button BT26 for registering the product are operable.

If the customer does not want to register the product, the customer operates the "Do not register" button BT25, and if the customer wants to register the product, the customer operates the "Register" button BT26. The small screen SG20 is erased by operating the non-registration button BT25 or the registration button BT26 (it becomes a display as shown in FIG. 4(A)).

FIG. 5(C) is a scene following the scene of FIG. 4(C). If the vibration detection unit does not detect a vibration equal to or higher than the first threshold value in the state after the product has been registered as shown in FIG. 4C, the mobile terminal 60 displays the registration screen (FIG. 4C). C) Display the small screen SG30 in front of the screen.

On the small screen SG30, a message MS31 notifying the user of confirmation and a message MS32 confirming whether or not the product has been inserted are displayed. Furthermore, an OK button BT35 is displayed in an operable manner on the small screen SG30.

The customer confirms that the product has been inserted (or inserts the product if it has not been introduced) and operates the OK button BT35 (the customer may confirm that the product has been inserted after operating the OK button BT35). The small screen SG30 is erased by operating the OK button BT35 (it becomes a display as shown in FIG. 4(C)).

(Example of mobile terminal 60 (Example 1))
The mobile terminal 60 has, for example, at least one of the following functions a to e, and operates to realize at least one function. Note that details will be explained using flowcharts in FIGS. 6 to 8.

(Function a) The mobile terminal 60 determines that a product has been added to the cart 70 even though the product is not registered (that is, without obtaining a product code) (detects a vibration equal to or higher than the first threshold), and If the accuracy of the judgment is relatively high (if the vibration is equal to or higher than the second threshold), an error sound (a strong tone sound) is output, an instruction is given to put the product into the dedicated bag 90, and the payment device At step 50, a notification is given that the store clerk will respond (steps 450 and 460 in FIG. 8). For example, the mobile terminal 60 outputs an error sound and displays the small screen SG10 in FIG. 5(A).
(Function b) Although the mobile terminal 60 determines that a product has been added to the cart 70 even though the product is not registered (despite detecting vibrations equal to or higher than the first threshold), the accuracy of this determination is relatively low. (if the vibration is less than the second threshold), a notification sound (soft tone sound) is output, and it is confirmed whether or not to register the product (steps 470 and 480 in FIG. 8). For example, the mobile terminal 60 outputs a notification sound and displays the small screen SG20 in FIG. 5(B).
(Function c) The mobile terminal 60 determines that the registered product has been placed in the cart 70 (although it detects vibrations of a value equal to or higher than the first threshold), but the accuracy of the determination is relatively low (the vibrations exceed the first threshold). If the value is less than the second threshold), it is confirmed that the product has been put in (step 420 in FIG. 7(A)). For example, the mobile terminal 60 displays the small screen SG30 in FIG. 5(C).
(Function d) If the mobile terminal 60 determines that the registered product has not been added to the cart 70 (if no vibration of more than the first threshold is detected), the mobile terminal 60 confirms that the product has been inserted (Fig. 7 (B) step 520). For example, the mobile terminal 60 displays the small screen SG30 in FIG. 5(C).
(Function e) When the mobile terminal 60 acquires the product code of a certain product and determines that a product registered before the product has not been added to the cart 70 (detects a vibration of more than the first threshold) If not, the user confirms that the product has been inserted (step 220 in FIG. 6(B)). For example, the mobile terminal 60 displays the small screen SG30 in FIG. 5(C).

6 to 8 are flowcharts illustrating an example of the operation of the mobile terminal 60. The flowchart in FIG. 6A starts based on the operation of the shopping start instruction button on the service start screen. The flowcharts in FIGS. 6(B), 7(A), and 7(B) transition from the flowchart in FIG. 6(A), respectively. The flowchart in FIG. 8 transitions from the flowchart in FIG. 7(A). Note that the flowcharts in FIGS. 6 to 8 (same as the flowcharts in FIGS. 9 to 11) are related to steps S1, S2, and S8 in the sequence diagram in FIG. 3 (mainly related to step S1).

(Explanation of flags and timers)
First, let's start with the "Product code last acquisition flag (hereinafter abbreviated as "Product code acquisition flag")" and "Product code acquisition flag clear" that appear in the flowcharts of Figures 6 to 8 (the same applies to the flowcharts of Figures 9 to 11). ``Timer'' and ``Salesperson Confirmation Required Flag'' will be explained.
Product code acquisition flag: A flag indicating whether or not a product code has been acquired within a predetermined period of time (for example, 10 seconds) immediately before. If the product code acquisition flag is set (for example, the value "1"), it indicates that the product code was acquired within the previous predetermined time; if the product code acquisition flag is not set (for example, the value If it is "0"), it indicates that the product code has not been acquired within the previous predetermined time. The product code acquisition flag is set based on the product code being acquired.
Product code acquisition flag clear timer: This is a timer that manages the timing to clear the product code acquisition flag (for example, change the value from "1" to "0"). Specifically, the product code acquisition flag clear timer is a timer that counts down (or counts up) the predetermined time (for example, 10 seconds). The product code acquisition flag clear timer is set based on the product code being acquired.
Clerk confirmation required flag: This is a flag indicating whether or not a clerk's response is required in the processing of the payment device 50. When the clerk confirmation required flag is set, it indicates that the clerk's response is necessary, and when the clerk confirmation necessary flag is cleared, it indicates that the clerk's response is not necessary. The clerk confirmation flag is set based on the fact that it is determined that the possibility of product introduction is relatively high in a state in which a product code has not been obtained within the previous predetermined time (that is, in a state in which no product is registered).

(Flowchart in Figure 6(A))
At the start of the flowchart of FIG. 6A, the product code acquisition flag, clerk confirmation flag, and product code acquisition flag clear timer are initialized (cleared).

Step S200: The mobile terminal 60 determines whether the product code has been acquired. If the product code is acquired (step S200: YES), the process advances to step S201 in FIG. 6(B). If the product code has not been acquired (step S200: NO), the process advances to step S400.
Step S400: The mobile terminal 60 determines whether or not a vibration equal to or higher than the first threshold is detected. In other words, the mobile terminal 60 determines whether or not there is a vibration of a magnitude equal to or greater than the first threshold value. If a vibration equal to or greater than the first threshold is detected (step S400: YES), the process advances to step S401 in FIG. 7(A). If vibrations greater than or equal to the first threshold have not been detected (step S400: NO), the process advances to step S502.

Step S502: The mobile terminal 60 determines whether or not it is in a state where the product code has been obtained within the previous predetermined time (post-product code acquisition state). Specifically, if the product code acquisition flag is set, the mobile terminal 60 determines that it is in the post-product code acquisition state, and if the product code acquisition flag is not set, it is in the post-product code acquisition state. I judge that it is not. The same applies to step S201 in FIG. 6(B) and step S401 in FIG. 7(A). If the product code is in the post-acquisition state (step S502: YES), the process advances to step S503. If the product code is not in the post-acquisition state (step S502: NO), the process advances to step S600. Note that the post-product code acquisition state is also a state in which the product is scheduled to be introduced (product introduction wait state).

Step S503: The mobile terminal 60 determines whether it is time to end the state after obtaining the product code (wait state for product insertion). Specifically, if the product code acquisition flag clear timer times out, the mobile terminal 60 determines that it is time to end the post-product code acquisition state, and if the product code acquisition flag clear timer has not timed out, It is determined that it is not time to end the state after obtaining the product code. If it is the end timing (step S503: YES), the process advances to step S509 in FIG. 7(B). If it is not the end timing (step S503: NO), the process advances to step S600.

Step S600: The mobile terminal 60 determines whether the checkout instruction button BT630 has been operated. If the checkout instruction button BT630 is operated (step S600: YES), the process advances to step S700. If the checkout instruction button BT630 is not operated (step S600: NO), the process returns to step S200.
Step S700: The mobile terminal 60 generates and displays a payment code. Then, this flowchart ends. Note that the customer causes the payment device 50 to read the payment code displayed on the mobile terminal 60.

(Flowchart in Figure 6(B))
Step S201: The mobile terminal 60 determines whether it is in a state after obtaining a product code (a state in which it waits for product insertion). If the product code is in the post-acquisition state (step S201: YES), the process advances to step S208. If the product code is not in the post-acquisition state (step S201: NO), the process advances to step S235.
Step S208: The mobile terminal 60 clears the product code acquisition flag clear timer (stops counting). The process then proceeds to step S209.
Step S209: The mobile terminal 60 clears the product code acquisition flag. The process then proceeds to step S220.
Step S220: The mobile terminal 60 confirms with the customer that the product has been inserted. For example, the mobile terminal 60 displays the small screen SG30 in FIG. 5(C). The same applies to step S420 in FIG. 7(A) and step S520 in FIG. 7(B). The process then proceeds to step S221.
Step S221: The mobile terminal 60 determines whether a confirmation operation has been performed. For example, the mobile terminal 60 determines whether or not the OK button BT35 on the small screen SG30 in FIG. 5(C) has been operated. The same applies to step S421 in FIG. 7(A) and step S521 in FIG. 7(B). If there is a confirmation operation (step S221: YES), the process advances to step S225. If there is no confirmation operation (step S221: NO), the process returns to step S221.

Step S235: The mobile terminal 60 transmits the product code etc. acquired in step S200 of FIG. 6(A) to the cloud server 20. The process then proceeds to step S238.
Step S238: The mobile terminal 60 sets a product code acquisition flag. The process then proceeds to step S239.
Step S239: The mobile terminal 60 sets a product code acquisition flag clear timer. Subsequently, the process returns to step S200 in FIG. 6(A).

(Flowchart in Figure 7(A))
Step S401: The mobile terminal 60 determines whether it is in a state after obtaining a product code (a state in which it waits for product insertion). If the product code is in the post-acquisition state (step S401: YES), the process advances to step S408. If the product code is not in the post-acquisition state (step S401: NO), the process advances to step S440 in FIG.
Step S408: The mobile terminal 60 clears the product code acquisition flag clear timer. The process then proceeds to step S409.
Step S409: The mobile terminal 60 clears the product code acquisition flag. The process then proceeds to step S410.

Step S410: The mobile terminal 60 determines whether or not a vibration equal to or higher than the second threshold is detected. That is, although a vibration having a magnitude greater than or equal to the first threshold value is detected in step S400 of FIG. 6(A), it is determined whether the magnitude of the vibration is greater than or equal to the second threshold value. If a vibration equal to or higher than the second threshold is detected (step S410: YES), the process returns to step S200 in FIG. 6(A). If vibrations greater than or equal to the second threshold have not been detected (step S410: NO), the process advances to step S420.
Step S420: The mobile terminal 60 confirms with the customer that the product has been inserted. The process then proceeds to step S421.
Step S421: The mobile terminal 60 determines whether a confirmation operation has been performed. If there is no confirmation operation (step S421: NO), the process returns to step S421.

(Flowchart in Figure 7(B))
Step S509: The mobile terminal 60 clears the product code acquisition flag. The process then proceeds to step S520.
Step S520: The mobile terminal 60 confirms with the customer that the product has been inserted. The process then proceeds to step S451.
Step S521: The mobile terminal 60 determines whether a confirmation operation has been performed. If there is a confirmation operation (step S521: YES), the process returns to step S200 in FIG. 6(A). If there is no confirmation operation (step S521: NO), the process returns to step S521.

(Flowchart in Figure 8)
Step S440: The mobile terminal 60 determines whether or not a vibration equal to or higher than the second threshold is detected. That is, although a vibration having a magnitude greater than or equal to the first threshold value is detected in step S400 of FIG. 6(A), it is determined whether the magnitude of the vibration is greater than or equal to the second threshold value. If a vibration equal to or higher than the second threshold is detected (step S440: YES), the process proceeds to step S450. If vibrations greater than or equal to the second threshold have not been detected (step S440: NO), the process advances to step S470.

Step S450: The mobile terminal 60 outputs (for example, outputs for 1 second) an error sound (warning sound). The error sound is a sound with a stronger tone than the notification sound output in step S470 (for example, a sound that can be described as "beep"). The process then proceeds to step S460.

Step S460: The mobile terminal 60 instructs the customer to put the product into the dedicated bag 90, and the payment device 50 notifies the customer that a store clerk will respond. For example, the mobile terminal 60 displays the small screen SG10 in FIG. 5(A). The process then proceeds to step S461.
Step S461: The mobile terminal 60 determines whether a confirmation operation has been performed. For example, the mobile terminal 60 determines whether or not the OK button BT15 on the small screen SG10 in FIG. 5(A) has been operated. If there is a confirmation operation (step S461: YES), the process advances to step S469. If there is no confirmation operation (step S461: NO), the process returns to step S461.

Step S469: The mobile terminal 60 sets a clerk confirmation flag. Note that if the clerk confirmation required flag has already been set, the process of step S469 may be omitted. Subsequently, the process returns to step S200 in FIG. 6(A).

Step S470: The mobile terminal 60 outputs (for example, outputs for 1 second) a notification sound (confirmation sound). The notification sound is a sound with a softer tone than the error sound output in step S450 (for example, a sound described as "poro loan"). The process then proceeds to step S480.

Step S480: The mobile terminal 60 confirms with the customer whether or not to register the product. For example, the mobile terminal 60 displays the small screen SG20 in FIG. 5(B). The process then proceeds to step S481.
Step S481: The mobile terminal 60 determines whether there is an operation not to register. For example, the mobile terminal 60 determines whether the non-registration button BT25 on the small screen SG20 in FIG. 5(B) has been operated. If there is an operation not to register (step S481: YES), the process returns to step S200 in FIG. 6(A). If there is no operation indicating not to register (step S481: NO), the process advances to step S482.

Step S482: The mobile terminal 60 determines whether there is an operation to register. For example, the mobile terminal 60 determines whether or not the registration button BT26 on the small screen SG20 in FIG. 5(B) has been operated. If there is an operation to register (step S482: YES), the process advances to step S483. If there is no operation to register (step S482: NO), the process returns to step S481.

Step S483: The mobile terminal 60 determines whether the product code has been acquired. If the product code is acquired (step S483: YES), the process advances to step S485. If the product code has not been acquired (step S483: NO), the process returns to step S483.
Step S285: The mobile terminal 60 transmits the product code etc. acquired in step S483 to the cloud server 20. Subsequently, the process returns to step S200 in FIG. 6(A).

(Other embodiments of the mobile terminal 60 (Example 2))
The operation (function) shown in the first embodiment did not change the subsequent operation depending on the presence or absence of the scan operation, but in the second embodiment, the subsequent operation is changed depending on the presence or absence of the scan operation. Specifically, the mobile terminal 60 operates to realize function a' of the second embodiment instead of function a of the first embodiment, and operates to realize function b' of the second embodiment instead of function b of the first embodiment. It operates to realize the following. Note that details will be explained using flowcharts in FIGS. 9 to 11.

(Function a') The mobile terminal 60 determines that a product has been added to the cart 70 even though the product is not actually registered through a scanning operation (step S402 (NO) in FIG. 11), and If the accuracy of the judgment is relatively high, an error sound is output, an instruction is given to put the product into the dedicated bag 90, and the checkout device 50 notifies that the store clerk will respond (steps 450 and 460 in FIG. 8).

In other words, in the first embodiment, if a vibration equal to or higher than the second threshold is detected in a state where no product is registered, regardless of whether or not the scanning operation shown in FIG. 4(B) has been performed, for example , displays the small screen SG10 as shown in FIG. In this case, the small screen SG10 is not displayed (if a scanning operation is being performed, the small screen SG10 is displayed).

(Function b') Although the mobile terminal 60 determines that a product has been added to the cart 70 even though the product is not actually registered through a scanning operation (step S402 (NO) in FIG. 11), If the accuracy is relatively low, a notification sound is output and it is confirmed whether or not to register the product (steps 470 and 480 in FIG. 8).

In other words, in the first embodiment, if a vibration of the first threshold value or more and less than the second threshold value is detected in a state where no product is registered, regardless of whether a scanning operation is performed or not, the vibration shown in FIG. However, in the second embodiment, even if a vibration of the first threshold value or more and less than the second threshold value is detected in a state where no product is registered, the small screen SG20 as shown in FIG. Screen SG20 is not displayed (if a scan operation is being performed, small screen SG20 is displayed).

9 to 11 are flowcharts illustrating other examples of the operation of the mobile terminal 60. The flowchart in FIG. 9A starts based on the operation of the shopping start instruction button on the service start screen. The flowcharts in FIGS. 9(B), 10(A), 10(B), and 11 transition from the flowchart in FIG. 9(A), respectively.

(Explanation of flags and timers)
First, the "immediate scan operation flag (hereinafter abbreviated as "scan operation flag")" and "scan operation flag clear timer" that appear in the flowcharts of FIGS. 9 to 11 will be explained.
Scan operation flag: A flag indicating whether or not a scan operation has been performed within the previous predetermined time (for example, 10 seconds). If the scan operation flag is set (for example, the value "1"), it indicates that the scan operation was performed within the previous predetermined time; if the scan operation flag is not set (for example, the value "0") ” indicates that no scan operation has been performed within the previous predetermined time period. The scan operation flag is set based on detection of a scan operation.
Scan operation flag clear timer: This is a timer that manages the timing of clearing the scan operation flag (for example, changing the value from "1" to "0"). Specifically, the scan operation clear timer is a timer that counts down (or counts up) the predetermined time (for example, 10 seconds). The scan operation clear timer is set based on detecting a scan operation.

(Flowchart in Figure 9(A))
At the start of the flowchart in FIG. 9(A), each flag (product code acquisition flag, clerk confirmation flag, scan operation flag) and each timer (product code acquisition flag clear timer, scan operation flag clear timer) are initialized. be done.

The flowchart of FIG. 9(A) has step S100, step S302, and step S303 added to the flowchart of FIG. 6(A). Note that in the flowchart of FIG. 9(A), illustration of steps after step S600 (step S600, step S700) is omitted. Steps S200, S400, S502, S503, S600 (not shown), and S700 (not shown) in the flowchart in FIG. 9(A) are the same as in the flowchart in FIG. Omit part or all.

Step S100: The mobile terminal 60 determines whether a scanning operation has been performed. If there is a scan operation (step S100: YES), the process advances to step S106 in FIG. 9(B). If there is no scanning operation (step S100: NO), the process advances to step S200.

Step S200: The mobile terminal 60 determines whether or not the product code has been acquired. If the product code has been acquired (step S200: YES), the mobile terminal 60 executes the process of step S201 in FIG. If not acquired (step S200: NO), the process of step S302 is executed.

Step S302: The mobile terminal 60 determines whether it is in a state where a scanning operation was performed within the previous predetermined time (post-scanning operation state). Specifically, if the scan operation flag is set, the mobile terminal 60 determines that it is in the post-scan operation state, and if the scan operation flag is not set, it determines that it is not in the post-scan operation state. . The same applies to step S402 in FIG. If it is in the post-scanning state (step S302: YES), the process advances to step S303. If it is not in the post-scanning state (step S302: NO), the process advances to step S400.

Step S303: The mobile terminal 60 determines whether it is the end timing of the post-scan operation state. Specifically, if the scan operation flag clear timer has timed out, the mobile terminal 60 determines that it is time to end the post-scan operation state, and if the scan operation flag clear timer has not timed out, the mobile terminal 60 determines that it is time to end the post-scan operation state. It is determined that it is not time to end the state. If it is the end timing (step S303: YES), the process advances to step S306 in FIG. 10(B). If it is not the end timing (step S303: NO), the process advances to step S400.

Step S400: The mobile terminal 60 determines whether or not a vibration equal to or higher than the first threshold is detected, and if a vibration equal to or higher than the first threshold is detected (step S400: YES), executes the process of step S401 in FIG. However, if vibrations greater than or equal to the first threshold have not been detected (step S400: NO), the process of step S502 is executed.

Step S502: The mobile terminal 60 determines whether or not it has acquired a product code within the previous predetermined time (post-product code acquisition state), and if it is in the post-product code acquisition state (step S502: YES), the process of step S503 is executed, and if the product code is not in the post-acquisition state (step S502: NO), the process of step S600 is executed.
Step S503: The mobile terminal 60 determines whether or not it is the end timing of the product code acquisition state (product insertion waiting state), and if it is the end timing (step S503: YES), the step of FIG. 7(B) The process of S509 is executed, and if it is not the end timing (step S503: NO), the process of step S600 is executed. Further, as described above, although illustration of steps after step S600 is omitted in the flowchart of FIG. 9(A), the steps after step S600 of the flowchart of FIG. 6(A) are the same.

(Flowchart in Figure 9(B))
Step S106: The mobile terminal 60 sets a scan operation flag. The process then proceeds to step S107.
Step S107: The mobile terminal 60 sets a scan operation flag clear timer. Subsequently, the process returns to step S100 in FIG. 9(A).

(Flowchart in Figure 10(A))
The flowchart of FIG. 10(A) has step S236 and step S236 added to the flowchart of FIG. 6(B). In the flowchart of FIG. 10(A), the contents of the steps (steps S201 to S221) within the broken line frame P of the flowchart of FIG. 6(B) are collectively abbreviated as "P". Steps S201 to S221 (abbreviated as "P"), step S235, step S238, and step S239 in the flowchart in FIG. 10(A) are the same as in the flowchart in FIG. omitted.

Step S235: The mobile terminal 60 transmits the product code etc. acquired in step S200 of FIG. 9(A) to the cloud server 20. The process then proceeds to step S236.
Step S236: The mobile terminal 60 clears the scan operation flag clear timer. The process then proceeds to step S237.
Step S237: The mobile terminal 60 clears the scan operation flag. The process then proceeds to step S238.

The mobile terminal 60 sets a product code acquisition flag (step S238), sets a product code acquisition flag clear timer (step S239), and returns to step S100 in FIG. 9(A).

(Flowchart in Figure 10(B))
Step S306: The mobile terminal 60 clears the scan operation flag. Subsequently, the process returns to step S100 in FIG. 9(A).

(Flowchart in Figure 11)
The flowchart of FIG. 11 has steps S402, S436, and S437 added to the flowchart of FIG. 7(A). Note that in the flowchart of FIG. 11, illustrations of steps after step S408 (steps S408 to S410, step S420, and step S421) of the flowchart of FIG. 7(A) are omitted. Steps S401, S408 and subsequent steps (not shown) in the flowchart in FIG. 11 are the same as in the flowchart in FIG. 7A, so a part or all of the explanation will be omitted.

Step S401: The mobile terminal 60 determines whether or not it is in the product code acquisition state (product insertion waiting state), and if it is in the product code acquisition state (step S401: YES), executes the processes from step S408 onwards. However, if the product code is not in the post-acquisition state (step S401: NO), the process of step S402 is executed.

Step S402: The mobile terminal 60 determines whether it is in the post-scanning state. If it is in the post-scanning state (step S402: YES), the process advances to step S436. If it is not in the post-scanning state (step S402: NO), the process returns to step S100 in FIG. 9(A).

Step S436: The mobile terminal 60 clears the scan operation flag clear timer. The process then proceeds to step S437.
Step S437: The mobile terminal 60 clears the scan operation flag. The process then proceeds to step S440 in FIG. That is, the process following step S437 in FIG. 11 is the same as the process following step S401 (NO) in FIG. 7(A).

The first embodiment has been described above, and according to the first embodiment, it is possible to suitably register a product using the mobile terminal 60. For example, in the mobile terminal 60 of the first embodiment, if the vibration detected by the vibration detection unit (vibration value acquired by the CPU) is greater than or equal to the first threshold and less than the second threshold in a state where no product is registered, the mobile terminal 60 is not registered. It is determined (low accuracy) that a registered product has been inserted (so-called cart omission), and if it is equal to or higher than the second threshold value, it is determined that cart omission has occurred (high accuracy) (function a and function b). According to the mobile terminal 60 of the first embodiment, when it is determined that an unregistered product has been placed in the cart 70 (shopping basket 80), appropriate measures can be taken depending on the accuracy of the determination. Further, in the mobile terminal 60 of the second embodiment, the vibration detected by the vibration detection unit (the vibration value acquired by the CPU) in the state where the product is not registered is the first If it is greater than or equal to the threshold and less than the second threshold, it is determined that an unregistered product has been inserted (missing the cart) (low accuracy), and if it is greater than or equal to the second threshold, it is determined that the cart has been left out (high accuracy) (Function a' and function b'). According to the mobile terminal 60 of the second embodiment, malicious actions such as falsifying product registration by performing a scanning operation while ignoring the effects of vibrations caused by the running of the cart 70 or collisions of people or objects with the cart 70 can be avoided. Able to deal effectively with certain customers. Further, according to the mobile terminal 60 of the first embodiment (the same applies to the mobile terminal 60 of the second embodiment), it is possible to prevent forgetting to add registered products to the cart 70 (functions c to e).

Other configurations related to the first embodiment will be described below. Note that among the other configurations related to the first embodiment described below, the configurations are also applicable to the second embodiment (in the case where the mobile terminal 60 acquires vibration values from the cart-side vibration detection unit (described later)). What is possible may be applied to the second embodiment.

Note that, as is common to Embodiments 1 and 2, although it was explained that the mobile terminal 60 confirms with the customer that the registered product has been inserted in step S220 of FIG. 6(B), You may also check whether the registered product has been returned to the shelf without being added. For example, in step S220, instead of the small screen SG30 in FIG. SG40 (not shown) may be displayed. The same applies to step S420 in FIG. 7(A) and step S520 in FIG. 7(B).

When a button indicating that the item has been returned to the shelf is operated on the small screen SG40 (not shown), the mobile terminal 60 may notify that a store clerk will take action on the payment machine (payment device 50). However, it may be possible to notify that the product returned to the shelf will be placed in the dedicated bag 90. For example, when a button indicating that the item has been returned to the shelf is operated on the small screen SG40 (not shown), the mobile terminal 60 may display the small screen SG10 in FIG. 5(A).

Also, as is common to Embodiments 1 and 2, if there is an operation to not register in step S481 of FIG. If the button BT25 is operated), the clerk confirmation required flag may be set, and in the case of the first embodiment, the process may return to step S200 in FIG. ) may return to step S100).

Also, common to Embodiments 1 and 2, if the mobile terminal 60 does not acquire the product code within a predetermined time in step S483 of FIG. 8, the mobile terminal 60 determines whether to register the product again. may be confirmed (you may return to step S480). If the mobile terminal 60 does not acquire the product code within a predetermined time, instead of or in addition to confirming whether to register the product again, it may set a clerk confirmation flag. .

Also, as is common to Embodiments 1 and 2, when the accounting instruction button BT630 is operated (step S600: YES), the mobile terminal 60 performs the steps (steps) within the broken line frame P in FIG. Steps S201 to S221) may be executed, and then a payment code may be generated and displayed (step S700).

That is, the mobile terminal 60 operates to realize the following function f instead of or in addition to the functions described above (function a, function b, function a', function b', function c, function d, function e). You may.

(Function f) When the mobile terminal 60 determines that a product registered before the relevant product has not been added to the cart 70 when the checkout instruction button BT630 is operated (vibrates at or above the first threshold) If the product is not detected), confirm that the product has been inserted.

Also, common to Embodiment 1 and Embodiment 2, in the above flowcharts (flowcharts in FIGS. 6 to 11), each timer corresponding to each flag is generated (started), and each flag is controlled (cleared). ), but each flag may be controlled based on the elapsed time from the time recorded as a log without generating a timer. In other words, the flag may be controlled by a timestamp. As an example, the mobile terminal 60 sets a product code acquisition flag in step S238 of FIG. 6(B), sets a product code acquisition flag clear timer as a timer to control the product code acquisition flag in the subsequent step S239, and In step S503 of FIG. 6A, the end timing is determined by referring to the value of the product code acquisition flag clear timer, but the product code acquisition flag clear timer is not set in step S239 (in other words, In step S503 of FIG. 6A, refer to the elapsed time from the time when the product code acquisition flag was set (the processing time in step S238) or the time when the product code, etc. was transmitted (the processing time in step S235). The end timing may also be determined by

In addition, in the above flowchart, an example was explained in which control is performed differently based on the value of each flag (whether it is set or not), but control is also performed based on the state of the timer (that is, whether it is counting or not). It may be different. As an example, the mobile terminal 60 sets a product code acquisition flag in step S238 of FIG. 6(B), sets a product code acquisition flag clear timer as a timer to control the product code acquisition flag in the subsequent step S239, and In step S502 of FIG. 6A, it is determined whether or not the product code acquisition state is reached by referring to the value of the product code acquisition flag, but in step S238, the product code acquisition flag is not set ( In other words, without performing the process in step S238), in step S502 of FIG. You can judge.

In addition, instead of or in addition to one or both of the flag and the timer, the status (operation status information) may be used (changed, referenced, etc.) to control the operation. The status that changes is, for example, a state before a scanning operation (absent), a state after a scanning operation (present), a state after obtaining a product code (a state waiting for product insertion), and the like.

Also, as is common to Embodiments 1 and 2, when the mobile terminal 60 determines that the cart has been left out (with high accuracy) (step S440 in FIG. 8), after the confirmation operation (step S461 (YES) ), the clerk confirmation flag is set (step S469), but if it is determined (with high accuracy) that a cart has been left out, the clerk confirmation flag is set unconditionally (for example, before outputting an error sound). Flags may be set.

Also, as is common to Embodiments 1 and 2, when the mobile terminal 60 determines that a cart has been left out (high accuracy), it sets a clerk confirmation flag (step S469 in FIG. 8). In place of or in addition to setting the clerk confirmation flag, information indicating that it has been determined (with high accuracy) that a cart has been left out is sent to other devices (e.g., cloud server 20, transaction status management device 40) together with transaction identification information. You may.

The transaction status management device 40 displays information on the occurrence of a cart omission in a transaction (information on whether it is determined that a cart omission has occurred) (information indicating that a cart omission has occurred (high accuracy) is displayed on the cloud server 20). If the information is not transmitted to the transaction status management device 40, the information may be displayed by accessing the cloud server 20). Note that when the mobile terminal 60 determines that a cart has been left out (high accuracy), it may transmit information indicating that it has been determined (high accuracy) that a cart has been left out to the clerk's mobile terminal.

The judgment result of the mobile terminal 60 in each transaction may be counted. For example, in each transaction, the number of times with low accuracy that it was determined that a cart was left out (low accuracy) (the number of times it was determined that step S440 (NO) in FIG. 8), or the number of times that it was determined that a cart was left out (high accuracy) was high One or both of the number of times (the number of times it is determined that step S440 (YES) in FIG. 8) may be counted.

The counting of the low-accuracy count and the high-accuracy count may be performed by the mobile terminal 60, or may be performed by a device other than the mobile terminal 60 (for example, the cloud server 20, the transaction status management device 40). In other words, in each transaction, if the mobile terminal 60 determines that a cart has been left out (low accuracy), it may count up the number of low accuracy by itself, or it can indicate that it has determined that a cart has been left out (low accuracy). The information may be sent to the cloud server 20 along with the transaction identification information, and the cloud server 20 may count up the number of times of low accuracy, or the information indicating that it has been determined that the cart has been left out (low accuracy) may be sent along with the transaction identification information. The transaction status management device 40 may transmit the information to the transaction status management device 40, and count up the number of times of low certainty on the transaction status management device 40 side. The same applies when it is determined (with high accuracy) that a cart has been left out.

In addition, in the mode in which the mobile terminal 60 counts up the low accuracy number or the high accuracy number, when the mobile terminal 60 counts up the low accuracy number or the high accuracy number, the mobile terminal 60 counts up the number of times after counting up (or , information indicating the number of times the transaction was counted up (that is, whether the number of times of low accuracy or the number of high accuracy was counted up) may be sent to the cloud server 20 or the transaction status management device 40 together with the transaction identification information. . Instead of or in addition to counting up the low-accuracy count or high-accuracy count, the mobile terminal 60 sends the final count together with the transaction identification information to the cloud server when requesting transaction information (step S10 in FIG. 3). 20 or the transaction status management device 40. Note that the mobile terminal 60 may transmit the low-accuracy count and the high-accuracy count together with transaction identification information to the clerk's mobile terminal.

In addition, the cloud server 20 (same as the transaction status management device 40) counts the number of low accuracy and high accuracy times based on the information (low accuracy/high accuracy) received from the mobile terminal 60 indicating that it has been determined that a cart omission has occurred. In the aspect of counting up, when the cloud server 20 (same as the transaction status management device 40) counts up the low accuracy number or the high accuracy number, the cloud server 20 (the same applies to the transaction status management device 40), when counting up the low accuracy number or the high accuracy number, the cloud server 20 (the same applies to the transaction status management device 40). information) may be transmitted to the mobile terminal 60.

The transaction status management device 40 displays information (low accuracy frequency, high accuracy frequency) of cart omission occurrence in each transaction (the cloud server 20 counts up the low accuracy frequency and high accuracy frequency, and the transaction status management device 40 counts up the low accuracy frequency and high accuracy frequency). If it is not uploaded, the cloud server 20 may be accessed and displayed).

Further, in the payment device 50, for example, even if the high accuracy count is 0, if the low accuracy count is a predetermined number or more, a clerk may intervene. In this case, the payment device 50 may display a message such as ``A clerk will check randomly'' or ``Since the count of erroneous operations has exceeded 0, a clerk will check.''

Note that, instead of or in addition to the comparison result of the vibration (vibration value) detected by the vibration detection unit with the threshold value (vibration comparison result), the mobile terminal 60 uses the comparison result of the image acquired (imaged) by the imaging unit. The accuracy of cart omission determination may be determined based on (image comparison results). For example, the mobile terminal 60 has a front image that is an image before vibration (e.g., an image captured before a vibration of more than a first threshold value is detected) and a after image (an image of a vibration more than a first threshold value) that is an image after vibration. The accuracy of cart omission determination may be determined based on the image comparison result obtained by comparing the image (image taken after the vibration was detected). Further, the mobile terminal 60 may determine whether the cart has been left out based on the image comparison result instead of or in addition to the vibration comparison result. In other words, the mobile terminal 60 determines the occurrence of a cart omission based on at least one of the vibration comparison result and the image comparison result, and determines the accuracy of the cart omission determination based on at least one of the vibration comparison result and the image comparison result. may be determined (there are 9 combinations (3×3)).

As an example, in the case of a configuration in which the occurrence of cart omission is determined based on the image comparison result, and the accuracy of cart omission determination is determined based on the image comparison result, the mobile terminal 60 may detect the occurrence of cart omission in a state where no product is registered. The degree of difference between two images (before image, after image) (for example, calculates a value indicating the difference between the feature amount of the entire front image and the feature amount of the entire after image.The above degree of difference does not exist in the front image. The value is larger when an object that does not exist in the front image exists in the rear image than when the object does not exist in the rear image.Therefore, the mobile terminal 60 determines whether the product is registered or not. If the calculated degree of difference is greater than or equal to the first threshold and less than the second threshold, it is determined that the cart has been left out (low accuracy), and if the calculated degree of difference is greater than or equal to the second threshold, the cart has been left out. It may be determined that this has occurred (high accuracy).

As described above, the mobile terminal 60 is used while being attached to the holder section 72, but it may be determined whether or not it is attached to the holder section 72 using the vibration detection section. For example, it may be determined that the mobile terminal 60 is attached to the holder portion 72 when there is no change in its posture for a predetermined period of time (that is, when no acceleration or the like is detected). Note that the cart 70 may include a charging function for charging the mobile terminal 60 attached to the holder portion 72. In this case, when the mobile terminal 60 is being charged by the charging function (for example, when it recognizes that it is being charged, or when the mobile terminal 60 is connected to the connector (terminal or cable) on the side of the cart 70 related to the charging function ), it may be determined that it is attached to the holder part 72. Further, the cart 70 includes a sensor and a communication unit that can recognize whether or not the mobile terminal 60 is attached to the holder part 72. Information indicating that it is attached to the holder portion 72 may be transmitted (transmitted at predetermined time intervals). In this case, if the mobile terminal 60 has not received the above information for a predetermined period of time, it may be determined that the mobile terminal 60 is not attached to the holder section 72.

When the mobile terminal 60 is attached to the holder part 72, the cart omission determination function for determining whether or not a cart omission has occurred may be enabled, and when it is not attached to the holder part 72, the cart omission determination function may be disabled. . In other words, when the mobile terminal 60 is removed from the holder section 72, the cart omission determination function is switched from valid to invalid, and when it is reattached to the holder section 72, the cart omission determination function is switched from invalid to valid. Good too. If the mobile terminal 60 is not the customer's property but is configured as a part of the cart 70, there is no need to enable/disable the cart omission determination function (in other words, the cart omission determination function is always activated). may be valid).

Further, product registration by the mobile terminal 60 may be restricted depending on whether the mobile terminal 60 is attached to the holder section 72 or not. For example, when the mobile terminal 60 is attached to the holder section 72, product registration may be permitted, and when the mobile terminal 60 is not attached to the holder section 72, product registration may be prohibited. The method of prohibition may be, for example, a method of prohibiting image capture by the image capture unit, a method of prohibiting acquisition of a product code (step S1 in FIG. 3), or a method of prohibiting the acquisition of a product code (step S1 in FIG. 3). It may be a method of prohibiting the transmission of product codes, etc. (step S2 in FIG. 3), or a method of prohibiting updating of transaction information in the cloud server 20 (step S4 in FIG. 1).

Furthermore, when the mobile terminal 60 is not attached to the holder section 72, registration by the mobile terminal 60 may be restricted depending on the type of product. Specifically, for products that cannot be registered with the mobile terminal 60 attached to the holder part 72 or specific products that are difficult to register (large products, products that require reading barcodes on price tags on display shelves, etc.) , registration is permitted even when the mobile terminal 60 is attached to the holder section 72, and for products other than specific products (that is, products that can be easily registered even when attached to the holder section 72), the holder section 72, registration may be prohibited. Whether or not it is a specific product may be set in advance in the product master, for example.

(Second embodiment)
In the first embodiment, the mobile terminal 60 detects vibrations, but in the second embodiment, the cart 70 detects vibrations. The cart 70 includes at least a vibration detection section and a communication section. For convenience of explanation, the vibration detection section provided in the mobile terminal 60 may be referred to as a terminal-side vibration detection section, and the vibration detection section provided in the cart 70 may be referred to as a cart-side vibration detection section. The vibration detection unit (cart-side vibration detection unit) included in the cart 70 detects vibrations of the cart 70 (own cart 70) and vibrations of the shopping basket 80 placed on the cart 70, for example, an acceleration sensor, It is a gyro sensor.

The cart 70 outputs the vibration value (vibration detection result) detected by the cart-side vibration detection section to the mobile terminal 60 through the communication section. Communication between the communication unit of the cart 70 and the communication unit of the mobile terminal 60 may be, for example, communication using a wired connection such as a device controller and a USB, or communication using a wireless connection such as Bluetooth (registered trademark). There may be.

When the mobile terminal 60 acquires the vibration value from the cart 70, the mobile terminal 60 determines the occurrence of cart omission based on the acquired vibration value, as in the first embodiment, and determines the accuracy of the cart omission determination. The threshold value (first threshold value or second threshold value) to be compared with the vibration value obtained from the cart 70 is the vibration value (the first threshold value or the second threshold value) detected by the cart-side vibration detection unit when the product is actually placed in the cart 70 (shopping basket 80). A large number of vibration values detected by the cart-side vibration detection unit when the cart 70 is moved (vibration values when the product is not inserted) are collected, and vibration values when the product is inserted are collected. It may be determined based on the vibration value or the distribution of vibration values when the product is not inserted.

(Detection of vibration by other sensors)
Although it has been explained that the cart-side vibration detection unit is, for example, an acceleration sensor or a gyro sensor, the cart-side vibration detection unit can detect vibrations of the cart 70 and the shopping basket 80 placed on the cart 70. It may be a sensor (e.g., a load cell) that can detect weight (e.g., the weight of the product in the shopping basket 80), or it may be a sensor (e.g., a load cell) that can detect the weight (e.g., the weight of the product in the shopping basket 80), or it may be a sensor (e.g., a load cell) that can detect the weight (e.g., the weight of the product in the shopping basket 80). It may be a sensor that detects the pressure applied from the outside) or a sensor that detects the sound pressure (for example, the sound pressure around the cart 70 or the shopping basket 80). In other words, the cart-side vibration detection section may be an acceleration sensor or a gyro sensor like the terminal-side vibration detection section, but vibrations caused when a product is put in or when the cart 70 moves are caused by changes in weight, Since it is possible to detect a change in pressure or a change in sound pressure, the cart-side vibration detection section may be a sensor capable of detecting weight, a sensor that detects pressure, or a sensor that detects sound pressure. In other words, the cart 70 may include a weight detection section, a pressure detection section, or a sound pressure detection section as a sensor for detecting vibration (cart-side vibration detection section).

The number of threshold values used to determine the occurrence of a cart omission using a vibration value determined from a change in weight and the accuracy of the cart omission determination may be two or more. In other words, the threshold value to be compared with the vibration value ascertained from the change in weight may be two or more, as long as it is possible to determine the occurrence of cart dropout and to determine the accuracy of the determination. In other words, the threshold value to be compared with the vibration value determined from the change in weight is, for example, the change in weight detected when the product is actually placed in the cart 70 (shopping basket 80) (change in weight when the product is placed), A large number of weight changes detected when the cart 70 is moved, etc. (weight changes when no products are inserted) are collected, and a pattern of weight changes when products are inserted and weight changes when no products are inserted is determined. You may decide based on. The same applies to the threshold value to be compared with the vibration value ascertained from the change in pressure, and the threshold value to be compared with the vibration value ascertained from the change in sound pressure.

(Judgment based on factors other than vibration)
Note that instead of or in addition to vibration, the cart 70 may detect (weigh) the weight (for example, the weight of the products in the shopping basket 80) and output the weight value (weight detection result) to the mobile terminal 60. good. Further, instead of or in addition to vibration or weight, the cart 70 detects pressure (for example, pressure applied from above to the cart 70 or shopping basket 80), and outputs a pressure value (pressure detection result) to the mobile terminal 60. You may. Further, the cart 70 detects sound pressure (for example, sound pressure around the cart 70 or shopping basket 80) instead of or in addition to vibration, weight, or pressure, and obtains a sound pressure value (sound pressure detection result). may be output to the mobile terminal 60. In other words, the cart 70 may include a weight detection section, a pressure detection section, or a sound pressure detection section as a sensor for detecting vibration (cart-side vibration detection section); A weight detection section is provided to detect (measure) the weight), a pressure detection section is provided to detect pressure (for example, the pressure applied from above to the cart 70 or the shopping basket 80), and a pressure detection section is provided to detect the sound pressure (for example, the pressure applied from above to the cart 70 or the shopping basket 80). A sound pressure detection section may be provided to detect the sound pressure around the cart 70 or the shopping basket 80.

The mobile terminal 60 may determine the occurrence of a cart omission based on various information acquired from the cart 70 and determine the accuracy of the cart omission determination. In other words, the mobile terminal 60 detects at least one of the vibration values (vibration detection results), weight values (weight detection results), pressure values (pressure detection results), and sound pressure values (sound pressure detection results) acquired from the cart 70. It is also possible to determine whether or not a product has been introduced, and to determine the accuracy of the determination, based on one threshold value and each threshold value (first threshold value, second threshold value).

Regarding the threshold values (first threshold value and second threshold value) to be compared with the weight value obtained from the cart 70, for example, the weight that can be determined to be that the increased weight is not due to the addition of the product (specifically, the lightest The first threshold is the weight (smaller than the product weight), and the second threshold is the weight for which it can be determined that the increased weight is due to the addition of the product (in other words, the weight cannot be considered to have increased due to an error). You can also use it as

The threshold value (first threshold value or second threshold value) to be compared with the pressure value obtained from the cart 70 is the pressure detected by the pressure detection unit included in the cart 70 when the product is actually placed in the cart 70 (shopping basket 80). Collect a large number of pressure values (pressure values when products are inserted) and pressure values detected by a pressure detection unit provided in the cart 70 when the cart 70 is moved (pressure values when products are not inserted), respectively. It may be determined based on the distribution of the pressure value when the product is inserted or the pressure value when the product is not inserted. The same applies to the threshold value to be compared with the sound pressure value acquired from the cart 70.

Note that instead of the mobile terminal 60, the cart 70 may determine the occurrence of a cart omission and determine the accuracy of the cart omission determination. For example, the cart 70 includes a CPU and a storage unit, and stores the detection results detected by itself (at least one of vibration detection results, weight detection results, pressure detection results, and sound pressure detection results) in the storage unit. The occurrence of cart omission may be determined based on the threshold value (first threshold value, second threshold value) for each detection result, and the accuracy of cart omission determination may be determined. Cart 70 transmits the determination result to mobile terminal 60.

Although it has been described that the cart 70 includes a vibration detection section (cart-side vibration detection section), a vibration detector including a vibration detection section may be attached to the cart 70. Furthermore, the shopping basket 80 may include a vibration detection section, or a vibration detector may be attached to the shopping basket 80. The same applies to the weight detection section, pressure detection section, and sound pressure detection section.

Note that two or more vibration detection units (cart-side vibration detection units) may be provided on the cart 70 side (vibration detection units may be provided at two or more locations on the cart 70 side). The same applies to the weight detection section, pressure detection section, and sound pressure detection section.

The second embodiment has been described above. According to the second embodiment, it is possible to suitably register a product using a mobile terminal, as in the first embodiment. Furthermore, the accuracy of judgment is improved by using multiple types of sensing information (vibration, weight, pressure, sound pressure) and by sensing at multiple locations.

Each embodiment has been described above, and according to each embodiment, it is possible to suitably register a product using a mobile terminal.

(Modified examples, etc.)
Although each embodiment has been described above, the device configuration, data configuration, processing flow, display and output mode, etc. can be changed as appropriate, for example, as shown in (1) to (8) below. . Furthermore, the following (1) to (8) may be combined as appropriate.

(1) In the first embodiment, an example will be described in which it is determined whether or not a product has been inserted based on the detection results (vibration, etc.) of the terminal-side vibration detection unit provided in the mobile terminal 60, and the accuracy of the determination is determined, In the second embodiment, an example has been described in which it is determined whether or not a product has been inserted based on the detection result (vibration, etc.) of the cart-side vibration detector, etc., and the accuracy of this determination is determined. The forms may be combined. In other words, the mobile terminal 60 detects vibrations detected by the terminal-side vibration detection unit of the mobile terminal 60 itself, captured images acquired by the imaging unit of the mobile terminal 60 itself, vibrations acquired from the cart-side vibration detection unit, and vibrations acquired from the weight detection unit. Based on at least one of the weight, the pressure obtained from the pressure detection section, and the sound pressure obtained from the sound pressure detection section, it may be determined whether or not a product has been inserted, and the accuracy of the judgment may be determined. The cart 70 may make the determination by collecting information in the cart 70 (sending the detection results from the mobile terminal 60 to the cart 70).

In addition, in an embodiment in which the detection result of the vibration detection unit of the mobile terminal 60 is used to make a determination without using the detection result of the cart-side vibration detection unit of the cart 70, the detection result is not transmitted from the mobile terminal 60 to the cart 70. The cart 70, instead of the mobile terminal 60, may determine whether or not a product has been inserted and determine the accuracy of the determination. The same applies to the contents described as the operation of the mobile terminal 60 in (2) and subsequent sections below.

(2) In the first embodiment (same as the second embodiment), an example was explained in which there are two thresholds (first threshold, second threshold) related to vibration (vibration detection result), but there are three or more thresholds. You can. In other words, in the first embodiment (the second embodiment is similar), if the vibration is less than the first threshold value, it is determined that the product has not been inserted, and if the vibration is greater than or equal to the first threshold value and less than the second threshold value, it is determined that the product has been inserted (with low accuracy). ), and if it is equal to or higher than the second threshold, it is determined that the product has been introduced (high accuracy), and the case where it is determined that the product has been introduced (low accuracy) is different from the case where it is determined that the product has been introduced (high accuracy). Although an example of outputting information has been described, it is also possible to make a judgment to classify the information more finely and output information corresponding to each category. The same applies to thresholds related to images (image comparison results). The same applies to the threshold value related to weight (weight detection result), the threshold value related to pressure (pressure detection result), and the threshold value related to sound pressure detection result in the second embodiment.

(3) In the first embodiment (the second embodiment is similar), it may be possible to consider putting products into the moving cart 70 (shopping basket 80). The vibration caused by placing a product into the cart 70 that is vibrating is considered to be larger than the vibration caused by putting a product into the cart 70 that is stationary. Similarly to the case where the insertion of products into the moving cart 70 is not considered, if the vibration is less than the first threshold value, it is determined that the product has not been inserted, and if it is greater than or equal to the first threshold and less than the second threshold, it is determined that the product has been inserted ( If the value is equal to or higher than the second threshold value, it may be determined that the product has been introduced (high accuracy).

Alternatively, the vibration detected by the vibration detection unit when the product is actually put into the stationary cart 70 (vibration value when the product is put in the stationary cart) and the vibration detected when the product is actually put into the moving cart 70. Vibration detected by the vibration detection unit (vibration value when the product is inserted during movement), vibration detected by the vibration detection unit when the cart 70 is simply moved (vibration value when the product is not inserted during movement), A large number of each are collected, and two or more thresholds are determined based on the distribution of each vibration (vibration value when a product is inserted while stationary, vibration value when a product is inserted while moving, vibration value when a product is not inserted while moving). Then, by comparing the vibration detected by the vibration detection section with the two or more threshold values, it may be determined whether the product has been inserted, whether the product has been inserted (low accuracy), or whether the product has been inserted (high accuracy).

A member different from the terminal-side vibration detection unit (or the cart-side vibration detection unit) (for example, a position sensor such as a GPS included in the mobile terminal 60 or the cart 70, a sensor that detects movement of wheels or the like included in the cart 70, and a sound pressure detection unit) It may be detected whether the cart 70 is stationary or moving using an image recognition function (recognition of changes in the background part of a captured image, etc.). When it is determined that the cart 70 is being moved by the different members mentioned above, the vibration detected by the vibration detection unit (vibration value in the moving cart 70) and the threshold value for cart movement (the threshold value for the cart movement) are determined. 1 threshold and a second threshold for cart movement), it may be determined whether a product has been inserted, whether a product has been inserted (low accuracy), or whether a product has been inserted (high accuracy). Note that the threshold value for when the cart is moving may be determined based on the distribution of vibrations during movement (vibration values when a product is inserted during movement and when a product is not inserted during movement).

(4) In the first embodiment (same as the second embodiment), the collapse (movement) of products in the cart 70 (shopping basket 80) may be taken into consideration. For example, it is assumed (set) that when the number of products existing in the cart 70 is more than a predetermined number (for example, 5 products), the load is likely to collapse, and the vibration detected by the vibration detection unit and the Based on the number of products that have already been added to the cart 70 (or the number of products that have already been registered) and the threshold value, it is determined whether the product has not been added, the product has been inserted (low accuracy), or the product has been inserted (high accuracy). You can. As an example, the vibration detected by the vibration detection unit is equal to or higher than the second threshold, and the number of products already placed in the cart 70 (or the number of products already registered) before the vibration detection is less than a predetermined number. In this case, it is determined that a product has been inserted (high accuracy), and even if the vibration detected by the vibration detection unit is equal to or higher than the second threshold, the number of products that have already been inserted into the cart 70 (or If the number of products that have already been registered is greater than or equal to a predetermined number, it may be determined that the product has been introduced (low accuracy). Note that instead of or in addition to the number of products that have already been placed in the cart 70 (or the number of products that have already been registered), the total weight of the products that have already been placed in the cart 70 (or the products that have already been registered) The size and shape of the products (or products already registered) placed in the container may also be taken into consideration to determine whether or not the cargo is likely to collapse. In other words, based on the vibration detected by the vibration detection unit and the situation at the time of vibration detection (one or more of the number of products, total weight, product size and shape), it is determined whether the product is not inserted, the product is present (low or low). (accuracy), and whether or not a product has been introduced (high accuracy) may be determined. Note that it may be determined whether or not the cargo has collapsed based on the image. For example, if an object that does not exist in the image before vibration exists in the image after vibration, or if a new object is recognized in a predetermined area (periphery) of the image before vibration (product If the entrance of a hand or hand is recognized), it may be determined that the load has not collapsed.

(5) In the first embodiment (the second embodiment is the same), whether or not a product is inserted is determined based on one value (instantaneous value) of certain sensing information (for example, vibration acquired from one vibration detection unit). Although an example has been described in which it is determined, whether a product has been introduced or not may be determined based on a plurality of values (for example, a pattern of change in value over time, a feature amount obtained from a plurality of values).

As an example, the case of vibration will be explained. For judgment purposes, a vibration change pattern (vibration change when the product is inserted) that is continuously obtained from the terminal-side vibration detection unit (or cart-side vibration detection unit) when the product is actually placed in the cart 70 (shopping basket 80). Collect a large number of patterns) and calculate the feature amount of the vibration change pattern when the product is introduced. In the judgment scene, the mobile terminal 60 calculates the feature amount of the vibration change pattern continuously obtained from the terminal side vibration detection unit (or the cart side vibration detection unit), and uses the calculated feature amount and the value at the time of product insertion. The feature quantity of the vibration change pattern is compared, and if the degree of similarity (degree of coincidence) between the two is less than the first threshold, it is determined that the product has not been introduced, and if it is greater than or equal to the first threshold and less than the second threshold, it is determined that the product has been introduced ( If the value is equal to or higher than the second threshold value, it may be determined that the product has been introduced (high accuracy). Alternatively, for judgment purposes, a large number of vibration change patterns (vibration change patterns when no products are inserted) that are continuously obtained from the terminal-side vibration detection unit (or cart-side vibration detection unit) when the cart 70 is moved etc. Collect information and calculate the feature amount of the vibration change pattern when no product is inserted. In the case of judgment, the mobile terminal 60 calculates the feature amount of the vibration change pattern continuously obtained from the terminal-side vibration detection unit (or the cart-side vibration detection unit), and uses the calculated feature amount and the time when the product is not inserted. If the degree of difference between the two is less than the first threshold, it is determined that the product has not been introduced, and if it is greater than or equal to the first threshold and less than the second threshold, the product has been introduced (low accuracy). If the value is equal to or higher than the second threshold value, it may be determined that the product has been introduced (high accuracy). The mobile terminal 60 uses both the feature amount of the vibration change pattern when the product is inserted and the feature amount of the vibration change pattern when the product is not inserted, to determine whether the product is not inserted, the product is inserted (low accuracy), or the product is inserted ( high accuracy).

(6) Although related to (5) above, a learned model generated by machine learning may be used to determine whether or not to introduce a product. The trained model (hereinafter referred to as trained model 1000) for determining whether or not to introduce a product may be a model expressed using a neural network, for example. Specifically, the trained model 1000 acquires each feature amount (vibration If one or more of the feature values related to values, ..., captured images, etc.) are input, the probability that the product is inserted (likelihood) and the probability that the product is not inserted (likelihood) are calculated. It may also be something that is output.

The trained model 1000 is generated by a learning model generation device (for example, a personal computer). For example, the learning model generation device generates feature quantities (features without insertion) based on sensing information, etc. in a large number of scenes in which no products are inserted into the cart 70, and features in each of the scenes in which products are not inserted. The teacher data indicating that there is no input (that is, the teacher data corresponding to the non-input feature), and the respective feature amounts (features with input) based on sensing information etc. in numerous situations in which products are placed in the cart 70. , is a model generated based on training data indicating that a product has been introduced in each scene (that is, training data corresponding to the introduced feature).

The mobile terminal 60 calculates feature amounts from sensing information and the like. The mobile terminal 60 inputs the calculated feature amount to the input layer of the trained model 1000, and obtains an output value (probability that a product has been inserted, probability that a product has not been introduced) from the output layer. For example, the mobile terminal 60 determines that the product is not inserted if the probability that the product obtained from the output layer is inserted is less than a predetermined first probability (for example, 10%), and if the probability that the product obtained from the output layer is inserted is less than a predetermined first probability (for example, 10%), the mobile terminal 60 determines that the product is not inserted. If the probability is lower than a predetermined second probability (for example, 50%), it may be determined that a product has been inserted (low probability), and if the second probability is a predetermined second probability, it may be determined that a product has been inserted (high probability).

(7) Although related to (6) above, a trained model generated by machine learning may be used to determine the presence or absence of a scanning operation. The trained model (hereinafter referred to as trained model 1100) for determining the presence or absence of a scanning operation may be a model expressed using a neural network, for example. Specifically, the trained model 1100 calculates the probability (likelihood) that a scan operation is performed, the probability (likelihood) that a scan operation is not performed, when a feature amount based on a captured image is input. It may also be something that outputs .

The trained model 1100 is generated by a learning model generation device. For example, the learning model generation device generates feature quantities based on captured images in a large number of scenes in which no scanning operation is performed (feature quantities without scanning operation), and features in which no scanning operation is performed in each of the scenes. teacher data indicating that the scan operation is performed (i.e., teacher data corresponding to the scan operation feature-free quantity), and respective feature quantities based on captured images in a number of scenes in which the scan operation was performed (scan-operation present feature quantities); This model is generated based on teacher data indicating that a scan operation was performed in each scene (that is, teacher data corresponding to a feature amount having a scan operation). The trained model 1100 generated by the learning model generation device is stored in a location where the mobile terminal 60 can refer to it.

The mobile terminal 60 calculates feature amounts from the captured image. The mobile terminal 60 inputs the calculated feature amount to the input layer of the learned model 1100, and obtains an output value (probability that a scan operation is performed, probability that a scan operation is not performed) from the output layer. For example, the mobile terminal 60 determines that the scan operation is not performed if the probability that the scan operation was performed obtained from the output layer is less than a predetermined probability (for example, 50%), and if the probability that the scan operation is performed is less than a predetermined probability (for example, 50%), the mobile terminal 60 determines that the scan operation is not performed. If so, it is determined that a scanning operation is being performed.

(8) Some or all of the functions (input/output, storage, processing (including judgment)) of each device (mobile terminal 60, payment device 50, cart 70, etc.) are the same as the device described as the main body executing the function. may be realized in other different devices. For example, each device may function as a so-called thin client specialized in input/output interfaces and the like. As an example, a cloud server (which may be the same as the cloud server 20 or another cloud server) executes a program to control the operation of each device, and each device uses a browsing function to display the screen. It may be displayed. As a specific example, even if the cloud server 20 acquires (receives) information necessary for determination from the mobile terminal 60 and determines whether a product has been inserted, whether a product has been inserted (low accuracy), or whether a product has been inserted (high accuracy), good.

<Summary of embodiments>
[Technical field]
The present invention relates to a program, an information terminal, and a system.
[Background technology]
2. Description of the Related Art A system is known in which a customer registers a product by himself/herself using a mobile terminal (see, for example, Patent Document 1).
[Prior art documents]
[Patent document]
[Patent Document 1] Japanese Patent Application Publication No. 2018-147252 [Summary of the invention]
[Problem to be solved by the invention]
However, there is room for improvement in the above-described system, for example, in countermeasures against malicious or inexperienced customers. The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a technique for suitably registering products using a mobile terminal.
[Means to solve the problem]

(1) One aspect of solving the above-mentioned problems is a program that causes a computer to function as an information terminal (for example, a mobile terminal 60) used for registering products to be purchased, an acquisition means, a registration means for registering a product based on product identification information in an image acquired by the image acquisition means, a vibration detection means for detecting vibrations of the own information terminal (for example, a terminal-side vibration detection section), an own information terminal. functions as an output means (for example, a display section, an audio output section, a communication section) capable of outputting information regarding the purchase of a product to an operator, and the output means is configured to detect a predetermined vibration by the vibration detection means, and If the product is not registered in the registration means before the vibration is detected (for example, if step S401 in FIG. 7 is NO), information regarding the purchase of the product (for example, step S450, step S460, step S470, This program is characterized in that it outputs (see step S480).
According to the program (1), it is possible to suitably register a product using a mobile terminal. For example, if there is a possibility that a cart has been left out, necessary information can be notified to the operator or the like.

(2) The output means is characterized in that output content varies depending on the vibration detected by the vibration detection means (see, for example, steps S440, S450, S470, and S480 in FIG. 7). The program described in 1).
According to the program (2), it is possible to suitably register a product using a mobile terminal. For example, it is possible to notify the operator of suitable information depending on the possibility (accuracy) of cart omission occurring. For example, as shown in FIG. 8, if there is a relatively high possibility that a cart will be left out, the operator is notified of information requesting that the operator take action against the cart (step S460). If the value is relatively low, the system notifies the operator of information that simply requires confirmation (step S480), so that countermeasures against cart omissions are maintained (without loosening) and the operator is not unnecessarily made to feel uncomfortable. You can do it like this. Furthermore, it is also possible to effectively reduce the frequency of store staff responding to cart omissions.

(3) The computer further functions as a behavior recognition means (for example, an image recognition function) that recognizes the behavior of the operator of the own information terminal from the image acquired by the image acquisition means, and the output means (1) or (2) characterized in that when the behavior recognized by the means is a predetermined behavior (for example, a scanning motion) (for example, see step S402 in FIG. 11), information regarding the purchase of the product is output. This is the program described in .
According to the program (3), it is possible to suitably register a product using a mobile terminal. For example, it is possible to effectively deal with malicious customers who falsify product registration by performing a scanning operation while ignoring the effects of vibrations caused by the running of the cart 70 or the collision of people or objects with the cart 70. can.

(4) The output means may output, as information regarding the purchase of the product, information indicating that the intervention of a store clerk is required when purchasing the product (for example, the small screen SG10 in FIG. 5(A)), or information regarding the necessity of purchasing the product. The program according to any one of (1) to (3) is characterized in that it outputs information (for example, small screen SG20 in FIG. 5(B)) for confirming whether or not the program is rejected.
According to the program (4), it is possible to suitably register a product using a mobile terminal. It is possible to output information according to the scene (situation).

(5) Store clerk intervention necessity determining means (for example, step S440 in FIG. 8 , step S460, step S469).
According to the program (5), it is possible to suitably register a product using a mobile terminal. For example, it is also possible to effectively reduce the frequency of store staff responding to carts being left out.

(6) One aspect of solving the above-mentioned problems is a program that causes a computer to function as an information terminal (for example, a mobile terminal 60) used for registering products to be purchased, an acquisition means, a registration means for registering a product based on the product identification information in the image acquired by the image acquisition means, a vibration acquisition means for acquiring vibration from a source other than the own information terminal (for example, a vibration value is acquired from a cart-side vibration detection section); (a communication unit that acquires information), and functions as an output unit (for example, a display unit, an audio output unit, a communication unit) capable of outputting information regarding the purchase of a product to an operator of the own information terminal, and the output unit If a predetermined vibration is acquired in , and the product is not registered in the registration means before the vibration is acquired (for example, if step S401 in FIG. 7 is NO), information regarding the purchase of the product (for example, This program is characterized by outputting steps (see step S450, step S460, step S470, and step S480).
According to the program (6), the same effect as the program (1) can be obtained.

(7) One aspect of solving the above-mentioned problem is an information terminal (for example, a mobile terminal 60) used for registering a product to be purchased, which includes an image acquisition means for acquiring an image, and an image acquisition means for acquiring an image. A registration means for registering a product based on the product identification information in the image, a vibration detection means for detecting the vibration of the own information terminal, and an output means (for example, a display section) capable of outputting information regarding the purchase of the product to the operator. , an audio output section, and a communication section), and the output means is configured such that the vibration detecting means detects a vibration of a predetermined value or more, and the product is not registered by the registration means before the detection of the vibration. (for example, when step S401 in FIG. 7 is NO), information regarding the purchase of the product (for example, see step S450, step S460, step S470, and step S480), and information regarding the purchase of the product are output. It is an information terminal.
According to the information terminal (7), the same effects as the program (1) can be obtained.

(8) One aspect of solving the above-mentioned problems is a system including an information terminal (for example, a mobile terminal 60) used for registering a product to be purchased, an image acquisition means for acquiring an image, and the image acquisition means A registration means for registering a product based on the product identification information in the image acquired by the user, a vibration detection means for detecting the vibration of the own information terminal, an output means for outputting information regarding the purchase of the product to the operator of the own information terminal ( For example, a display section, an audio output section, a communication section), and the output means is configured to detect whether a predetermined vibration is detected by the vibration detection means and the product is not registered by the registration means before the detection of the vibration. This system is characterized by outputting information regarding the purchase of the product (for example, see steps S450, S460, S470, and S480) when the product is purchased (for example, when step S401 in FIG. 7 is NO).
According to the system (8), the same effects as the program (1) can be obtained.

(9) Attitude detection means for detecting the attitude of the own information terminal (for example, a vibration detection unit of the mobile terminal 60, a connector on the cart 70 side, a sensor), and the registration means according to the attitude detected by the attitude detection means. Restriction means for restricting the registration of products (e.g., means for prohibiting image capture with an imaging unit, means for prohibiting acquisition of product codes, means for prohibiting transmission of product codes, etc., means for prohibiting updating of transaction information) According to the system (9) as described in (8), for example, in the case where the mobile terminal 60 detects vibration, the possibility that the cart is left out is reliably prevented. Necessary information can be reported to the operator in certain cases.

Note that a computer-readable recording medium is used to store a program for realizing the functions of each device described above (headquarters server 10, store controller 30, transaction status management device 40, payment device 50, mobile terminal 60, cart 70, etc.). The programs recorded on the recording medium may be read into a computer system and executed to perform the processing of each of the above devices. Here, "reading a program recorded on a recording medium into a computer system and executing it" includes installing the program on the computer system. The "computer system" here includes hardware such as an OS and peripheral devices. Further, a "computer system" may include a plurality of computer devices connected via a network including the Internet, a WAN, a LAN, a communication line such as a dedicated line, etc. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. In this way, the recording medium storing the program may be a non-transitory recording medium such as a CD-ROM. The recording medium also includes a recording medium provided internally or externally that can be accessed from the distribution server to distribute the program. The code of the program stored in the recording medium of the distribution server may be different from the code of the program in a format executable by the terminal device. That is, as long as it can be downloaded from the distribution server and installed in an executable form on the terminal device, the format in which it is stored on the distribution server does not matter. Note that the program may be divided into a plurality of parts, downloaded at different timings, and then combined on a terminal device, or the distribution servers that deliver each of the divided programs may be different. Furthermore, a ``computer-readable recording medium'' refers to a storage medium that retains a program for a certain period of time, such as volatile memory (RAM) inside a computer system that serves as a server or client when a program is transmitted via a network. This shall also include things. Moreover, the above-mentioned program may be for realizing a part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

1...Sales system 10...Head office server 20...Cloud server 30...Store controller 40...Transaction status management device 50...Payment device 60...Mobile terminal 70...Shopping cart 90...Special bag

Claims (9)

A program that causes a computer to function as an information terminal used to register products to be purchased,
The computer,
image acquisition means for acquiring images;
registration means for registering a product based on product identification information in the image acquired by the image acquisition means;
vibration detection means for detecting vibrations of the own information terminal;
Functions as an output means that can output information regarding product purchases to the operator of the own information terminal,
The output means is
A program characterized in that, when a predetermined vibration is detected by the vibration detection means and the product is not registered by the registration means before the detection of the vibration, information regarding the purchase of the product is output. The output means is
2. The program according to claim 1, wherein the output content varies depending on the vibration detected by the vibration detection means. The computer,
Further functioning as a behavior recognition means for recognizing the behavior of the operator of the own information terminal from the image acquired by the image acquisition means,
The output means is
3. The program according to claim 1, wherein when the behavior recognized by the behavior recognition means is a predetermined behavior, information regarding the purchase of the product is output. The output means is
2. The program according to claim 1, wherein the program outputs, as the information regarding the purchase of the product, information indicating that the intervention of a store clerk is required when purchasing the product, or information confirming whether or not the purchase of the product is necessary. The computer,
2. The apparatus further functions as a store clerk intervention necessity determining means for determining whether or not a store clerk's intervention is required when purchasing a product based on the vibration detected by the vibration detecting means. program. A program that causes a computer to function as an information terminal used to register products to be purchased,
The computer,
image acquisition means for acquiring images;
registration means for registering a product based on product identification information in the image acquired by the image acquisition means;
a vibration acquisition means for acquiring vibrations from a source other than the own information terminal;
Functions as an output means that can output information regarding product purchases to the operator of the own information terminal,
The output means is
A program characterized in that, when a predetermined vibration is acquired by the vibration acquisition means and the product is not registered by the registration means before the acquisition of the vibration, information regarding the purchase of the product is output. An information terminal used for registering products to be purchased,
an image acquisition means for acquiring an image;
registration means for registering a product based on product identification information in the image acquired by the image acquisition means;
vibration detection means for detecting vibrations of the own information terminal;
and an output means capable of outputting information regarding the purchase of the product to the operator,
The output means is
The information terminal is characterized in that when the vibration detecting means detects a vibration of a predetermined value or more and the product is not registered by the registration means before the detection of the vibration, the information terminal outputs information regarding the purchase of the product. . A system including an information terminal used for registering products to be purchased,
image acquisition means for acquiring images;
registration means for registering a product based on product identification information in the image acquired by the image acquisition means;
vibration detection means for detecting vibrations of the own information terminal;
Equipped with an output means that can output information regarding product purchases to the operator of the own information terminal,
The output means is
A system characterized in that when a predetermined vibration is detected by the vibration detection means and the product is not registered by the registration means before the detection of the vibration, information regarding the purchase of the product is output. Attitude detection means for detecting the attitude of the own information terminal;
9. The system according to claim 8, further comprising: restriction means for restricting registration of products in said registration means according to the orientation detected by said orientation detection means.

Images