JP7289613B2 - Merchandise management system and merchandise management method - Google Patents

Description

One embodiment relates to a merchandise management system and merchandise management method .

The introduction of unmanned cash registers, in which a purchaser who purchases an item in a store checks the item to be purchased by himself and pays for it, is progressing. Unmanned cash registers are expected to improve the efficiency of store operations and shorten the time it takes to make payments when purchasing products, so there is a possibility that they will be introduced more and more in the future.

JP 2016-532932 A

Some of the above-mentioned unmanned cash registers make the store user bear the work necessary for purchasing the product, such as having the store user himself/herself check the purchased product when paying for the product to be purchased.

In addition, there is a system that eliminates the check of the purchased product at the time of payment by determining the product selected by the store user as the purchased product by image recognition. However, this system may not be able to correctly identify the product selected by the store user by image recognition depending on the positional relationship between the product and the store user when selecting the product.

Therefore, in the present embodiment, it is an object to provide a product management system and a product management method in an unmanned cash register that reduces the work burden on the store user when purchasing products and correctly identifies the product selected by the store user. and

A merchandise management system of one embodiment comprises a server, first, second and third sensors , and assistance requesting means . The server manages the products displayed in each of the individual areas obtained by dividing the display area of the store employing the unmanned register into one or a plurality of areas and the users of the store. The first sensor detects a change in the weight of the product displayed in the individual area at the placement position and notifies the server as a first notification, and the second sensor displays the product in the individual area. A change in the distance between the product and the product is detected and notified to the server as a second notification, and a third sensor captures an image including the face and feet of the user existing in the individual area. and notifying the server of the photographed image including the face and feet as a third notification , and receiving and notifying the server of the user's operation assistance request when entering and leaving the store . The server detects that the product is taken out from the individual area when both the change in the weight and the change in the distance are detected by the first notification and the second notification from the first sensor and the second sensor. The user is identified by comparing the video including the face and feet notified as the third notification with the pre-registered video of the face and feet, and the product determined to have been taken out. performs registration processing for the user identified as such, performs settlement processing at the unmanned cash register based on the result of the registration processing, and performs assistance processing for entering and exiting the store when the assistance request is received. do

FIG. 1A is an overall view of a product movement detection device according to one embodiment. FIG. 1B is a diagram showing an example of an entry confirmation operation panel. FIG. 1C is a diagram showing an example of the arrangement of rails arranged above the display area. FIG. 1D is a diagram illustrating an example of a store exit confirmation operation panel. FIG. 2 shows behavioral patterns of users of a store equipped with the product movement detection device of this embodiment, from pre-registration for using the store to actually purchasing products and leaving the store, and what is necessary at that time. It is a diagram showing the relationship with the device to be. FIG. 3 is a diagram showing the pre-registration procedure of S1 in FIG. FIG. 4A is a diagram showing how the store entry confirmation in S2 of FIG. 2 is performed. FIG. 4B is a diagram showing a processing flow of store entry confirmation. FIG. 5A is a diagram showing the state of product selection in S3 of FIG. FIG. 5B is a top view of user A freely viewing and selecting products displayed in the display area of the shelf. FIG. 6A is a diagram showing an example of arrangement of distance sensors and weight sensors arranged in individual areas. FIG. 6B is a diagram showing the processing flow of the distance sensor and the weight sensor when specifying the product that was taken out and the user A who took out the product. FIG. 6C is a diagram showing a processing flow of the server when specifying the product that was taken out and the user A who took out the product. FIG. 6D is a diagram showing a processing flow of store entry confirmation when step authentication is performed. FIG. 6E is a diagram showing an example of a detailed flow of processing for photographing a portion of the feet in S643 of FIG. 6D. FIG. 6F is a diagram showing a processing flow of the server when identifying the product taken out and the user A who took out the product when step authentication is performed. FIG. 7 is a diagram showing the state of confirmation of leaving the store in S5 of FIG. FIG. 8 is a diagram showing another example of a shelf for displaying products.

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1A is an overall view of a product movement detection device according to one embodiment.

Reference numeral 1 denotes a store 1 equipped with a product movement detection device according to one embodiment.

100 is the entrance gate of the store 1 . When a visitor to the store 1 enters the store 1, the visitor confirms that he or she is a legitimate user of the store 1. - 特許庁If the visitor is determined to be a legitimate user as a result of confirmation, the visitor can enter the store 1 through the entrance gate 100.例文帳に追加The entrance gate 100 is equipped with a speaker (not shown) for emitting a notification sound and an LED (Light Emitting Diode) for lighting a notification lamp, for example, as a means for notifying the visitor to be permitted to enter the store or not to be permitted to enter the store. may

Reference numeral 110 denotes an entry confirmation operation panel for confirming that the visitor is an authorized user of the shop 1. FIG. As shown in FIG. 1B, the entry confirmation operation panel 110 includes a reader 110-1 for reading a membership number issued to a valid user of the shop 1 and a camera 110-2 for photographing the face of the visitor. Prepare. The member number read by reader 110-1 and the image of the face captured by camera 110-2 are notified to server 200 via network 201, so that the visitor to store 1 is a valid user. Confirmation is done.

The membership number may be displayed in characters, or may be displayed as a QR code (registered trademark), for example. The store entry confirmation operation panel 110 may also include a help button 110-3 for calling the clerk of the store 1 when the operation method of the store entry confirmation operation panel 110 is unknown or when trouble occurs in the operation. good.

When it is determined that the visitor to the store 1 is a valid user as a result of the store entry confirmation by face authentication of the member number and the face part, the visitor passes through the entrance gate 100 and enters the store 1. can do. In order to become a valid user of the store 1, prior registration is required. The pre-registration procedure will be described with reference to FIG.

Reference numeral 120 denotes a shelf having dimensions in the width, depth, and height directions for displaying products of the store 1 .

Reference numeral 121 denotes a plate-like display area for displaying each product, which the shelf 120 has. The display area 121 may be a single level or multiple stacked levels. The example of the shelf 120 in FIG. -4 and 5th display areas 121-5, which are five stacked display areas.

Reference numeral 122 denotes a partition plate that partitions the display area 121 in the height direction of the shelf 120 with an appropriate width. The interval between the adjacent partition plates 122 is desirably a necessary and sufficient interval for one user to see and pick up the products displayed in the display area 121 . The interval is, for example, about 1 meter.

A rail 123 is arranged on the display area 121 that spreads out like a plate. This rail 123 has a height perpendicular to the plate-shaped display area 121 and extends in the depth direction of the shelf 120 along the partition plate 122 when viewed from a user standing in front of the shelf 120. are arranged above the display area 121 as shown in FIG. The rails 123 divide the display area 121 into a plurality of individual areas 124 . An example of the layout of the rails 123 arranged above the display area 121 will be described with reference to FIG. 1C.

124 is an individual area configured by dividing the display area 121 by the rail 123 . The individual area 124 is an area for arranging products in a row. In one individual area 124, one type of merchandise is displayed in the same direction. Therefore, the interval between adjacent rails 123 that constitute one individual area 124 can be changed to match the size of the products to be displayed. The display area 121 is thus composed of one or more individual areas 124 . Each individual area 124 is uniquely assigned an individual area identification number. By specifying the individual area identification number, the location of the individual area 124 to which the number is assigned can be specified.

One distance sensor 131 and one weight sensor 132 are arranged in one individual area 124 . Distance sensor 131 and weight sensor 132 can communicate with server 200 via network 201 .

Reference numeral 111 denotes a store exit confirmation operation panel for confirming that the user of the store 1 has completed payment for the products registered in the basket shown in FIG. The store exit confirmation operation panel 111 has a reader 111-1 for reading the settlement completion notification. The configuration of the store exit confirmation operation panel 111 will be described with reference to FIG. 1D.

The settlement completion notification is a notification sent from the server 200 of the store 1 to the mobile terminal of the user when the settlement is made to purchase the products registered in the basket described with reference to FIG. Payment processing will be described with reference to FIGS. 2 and 7. FIG.

101 is the exit gate of the store 1 . The user causes the reader 111-1 of the exit confirmation operation panel 111 to read the payment completion notification, and when the confirmation of the completion of the payment for the purchased product is completed, the user passes through the exit gate 101 and exits the store. You can exit from 1. The exit gate 101 is equipped with a speaker (not shown) that emits a notification sound and an LED (not shown) that lights a notification lamp as a means for notifying the user that he/she is permitted to leave the store or not permitted to enter the store. good too.

130 is a camera installed on the shelf 120 for identifying the user. At least one camera 130 is installed at an arbitrary position between the adjacent partition plates 122, and the installed camera 130 photographs the face of the user standing facing the shelf 120. It has a sufficient angle of view for The camera 130 is also connected to the server 200 via the network 201 and sequentially sends captured images to the server 200 .

200 is the server of the store 1 . Server 200 can communicate with store entry confirmation operation panel 110 , store exit confirmation operation panel 111 , distance sensor 131 , weight sensor 132 , and camera 130 via network 201 .

Server 200 determines that the product has been removed from display area 121 based on the detection results sent from distance sensor 131 and weight sensor 132 . The server 200 also identifies the user standing in front of the shelf 120 by comparing the image sent from the camera 130 with the face image registered in advance.

Also, when confirming entering the store by the entering confirmation operation panel 110 and when confirming leaving the store by the exiting confirmation operation panel 111, processing for confirming entering the store and confirming leaving the store is performed based on the sent data.

Furthermore, the server 200 also communicates with a specific application installed in the mobile terminal of the user of the store 1 who has performed the pre-registration described in FIG. The server 200 also stores the details of pre-registration performed using the specific application.

The server 200 may also be called a control unit. Also, the place where the server 200 is placed is not limited. The server 200 may be installed for each shelf 120 or for each display area 121, for example. Alternatively, the server 200 may be located at a location other than the store 1 connected via the network 201 .

FIG. 1B is a diagram showing an example of an entry confirmation operation panel 110. As shown in FIG. The store entry confirmation operation panel 110 has a reader 110-1 for reading the membership number issued to the authorized user of the store 1, and the membership number read by the reader 110-1 after photographing the face of the visitor. A camera 110-2 is provided for determining the identity of the user by face authentication. The entering-store confirmation operation panel 110 may also include a help button 110-3 for calling a store clerk at store 1 when the user does not know how to operate or when trouble occurs during operation.

FIG. 1C is a diagram showing an example of an arrangement of rails 123 arranged on the display area 121. As shown in FIG. The rail 123 has a height that is perpendicular to the plate-shaped display area 121 and extends in the depth direction of the shelf 120 along the partition plate 122 when viewed from a user standing in front of the shelf 120 . is placed above the display area 121 at the top of the display area 121 . The example of FIG. 1C is an example in which the width from rail 123-1 to rail 123-2 and the width from rail 123-2 to rail 123-3 are different. Goods should be displayed in a line with the same orientation. The rail 123 may be linear or have any shape, but is preferably linear.

FIG. 1D is a diagram showing an example of the exit confirmation operation panel 111. As shown in FIG. The store exit confirmation operation panel 111 has a reader 111-1 for reading the settlement completion notification. The exit confirmation operation panel 111 may also include a display screen 111-2 so that the details of the completed payment can be confirmed again. The store exit confirmation operation panel 111 may also include a help button 111-3 for calling a clerk at store 1 when the user does not know how to operate or when trouble occurs during operation. The display screen 111-2 may not only display the details of the completed payment, but may also display future special sale information of the store 1 and the like. Alternatively, based on the user's past purchase history, a list of products recommended for purchase the next time the user visits the store may be displayed.

FIG. 2 shows behavior patterns of users of the store 1 equipped with the product movement detection device of the present embodiment, from pre-registration for using the store 1 to actually purchasing products and leaving the store. is a diagram showing the relationship with the devices required for

A person who wishes to use the store 1 equipped with the product movement detection device of the present embodiment needs to perform pre-registration (S1) in order to use the store 1. FIG.

Pre-registration is a registration work for a person who wishes to use the store 1 to obtain a qualification to use the store 1 . Pre-registration is performed by a person wishing to use the store 1 who installs a specific application provided by the store 1 on a portable terminal and uses the installed specific application. A membership number is issued to a person who wishes to use the store 1 by pre-registration. Also, in the server 200 of the store 1, a set of a member number issued to a person who wishes to use the service and an image of the face is registered as individual member information. A pre-registration procedure will be described with reference to FIG.

A visitor to the store 1 confirms entering the store when visiting the store 1 (S2). The store entry confirmation is a confirmation work performed by a visitor to the store 1 to confirm that the person is a valid user of the store 1 .

The visitor causes the reader 110-1 of the store entry confirmation operation panel 110 to read the membership number, and has the camera 110-2 photograph the face of the visitor. The member number read by the reader 110-1 and the image of the face taken by the camera 110-2 are sent to the server 200 of the store 1. FIG. When the server 200 receives the member number and face image sent from the store entry confirmation operation panel 110, the server 200 compares it with the individual member information registered in advance registration (S1) to determine whether the visitor is a valid user. make a judgment as to whether

The image of the face registered in the individual member information may be successively updated with the image of the face captured by the camera 110-2 at the time of confirmation of entry into the store.

The flow of store entry confirmation will be described with reference to FIG.

A visitor to the store 1 can enter the store 1 through the entrance gate 100 when it is confirmed that the visitor is the legitimate user A of the store 1 as a result of the store entry confirmation (S2). .

User A who has entered the store 1 can freely view and select products displayed in the display area 121 of the shelf 120 (S3). When the user A takes out the product A from the display area 121 in order to purchase the product displayed on the shelf 120, the distance sensor 131 and the weight sensor 132 arranged in the individual area 124 where the product A is taken out , that item A has been taken out. The face of the user A who has selected the product A is photographed by the camera 130 installed on the shelf 120 . The photographed image of the face of user A is sent to the server 200 via the network 201 and compared with the image of the face registered as member individual information registered in advance registration (S1). The person who selected the product A is identified as the user A by performing authentication with the user A.

In this way, distance sensor 131 and weight sensor 132 arranged on shelf 120 identify which product has been selected, and images taken by camera 130 arranged on shelf 120 identify who selected it. be. The information on the specified user A and the product A taken out by the user A from the individual area 124 is stored in the server 200 . The server 200 registers the information of the user A and the product A which the user A has taken out from the individual area 124 in the basket A of the user A. FIG.

It should be noted that the method of specifying the user by video is not limited to using the video of the face part, but the part of the user's appearance that seems to change little during the period from entering the store 1 to leaving the store. You can add a video of

An image of a portion of the user's appearance that is expected to change little during the period from entering the store to leaving the store is, for example, an image of the user's feet including the shoes and sandals worn by the user.

A basket is a storage area for each user for registering in the server 200 the information on the product A that the user A took out from the individual area 124 in S3. The basket is managed as one member's individual information created at the time of pre-registration. Each time the user A takes out a product from the individual area 124 in S3, the information of the taken product is additionally registered in the basket A of the user A.

The user can always refer to the information of the products registered in the basket by using the specific application installed on his/her mobile terminal.

Also, when the user purchases a product registered in the basket, the user can make a payment using a specific application. When User A uses a specific application to pay for the products registered in Basket A (S4), server 200 of store 1 notifies User A's portable terminal of payment completion notification.

The settlement completion notice includes information such as the date and time of purchase, a list of purchased products, the amount of payment, and change.

When user A completes payment for the products registered in basket A (S4), he can operate the exit confirmation operation panel 111 to confirm exit (S5).

Confirmation work for exiting the store is performed by causing the reader 111-1 of the exiting confirmation operation panel 111 to read the settlement completion notification sent to the user's mobile terminal. When the reader 111 - 1 reads the settlement completion notification, the settlement completion notification read by the reader 111 - 1 is notified to the server 200 via the network 201 . The server 200 that has received the notification confirms whether or not the payment for the user A has been completed. If the payment has been completed, the server 200 notifies the exit confirmation operation panel 111 to that effect. open the

FIG. 3 is a diagram showing the pre-registration procedure of S1 in FIG. A person wishing to use the store 1 needs to install a specific application required for pre-registration to use the store 1 on his or her portable terminal 301 . The mobile terminal 301 may be any communicable terminal that can be carried by the user, such as a smart phone, tablet, or mobile PC.

FIG. 3A is an example of input information necessary for a prospective user of store 1 to perform pre-registration according to instructions of a specific application displayed on display screen 310 of portable terminal 301 .

A person who wishes to use the store 1, according to the instructions of the specific application, sends a picture 310-1 including his/her face, a name 310-2, a gender 310-3, and a destination address 310-4 for notification from the store 1. input.

It is desirable to register a plurality of photos 310-1 including the face of the person who wishes to use the service. Moreover, it is desirable that the photograph 310-1 including each of the plurality of faces should have different shooting angles, facial expressions, poses, and the like. The example in FIG. 3 is an example in which photographs 310-1 containing five types of faces are registered.

The destination address may be, for example, an e-mail address, an SNS (Social Networking Service) address, or any other address of the communication means used by the store 1 . Also, a plurality of destination addresses may be input.

After completing the input of the information required for pre-registration, the person wishing to use store 1 presses button 310-5 managed by the specific application. When the specific application recognizes that the registration button 310-5 has been pressed, it transmits the input information necessary for pre-registration to the server 200. FIG.

When the server 200 receives the information required for pre-registration sent from the mobile terminal 301 , the server 200 stores the information as individual member information in the storage area of the server 200 , issues a member number, and transmits it to the mobile terminal 301 . . The issued member number is also stored in the server 200 as one of the individual member information.

FIG. 3B is an example of display when the member number sent from the server 200 is displayed on the display screen 310 of the portable terminal 301. As shown in FIG.

The specific application of the mobile terminal 301 that has received the notification of the member number from the server 200 displays the notified member number on the display screen 310 . The member number may be displayed by, for example, numerical values, or may be displayed by converting the numerical member number into a QR code. The example of FIG. 3B is an example in which the numeric value of the membership number is converted into a QR code 311 and displayed on the display screen 310 . In this way, a person who wishes to use the store 1 can obtain a membership number that is necessary when using the store 1 .

FIG. 4A is a diagram showing how the store entry confirmation in S2 of FIG. 2 is performed. A visitor 10 at the store 1 stands in front of the store entry confirmation operation panel 110 and performs store entry confirmation for entering the store 1. - 特許庁

FIG. 4B is a diagram showing a processing flow of store entry confirmation.

When the visitor 10 holds up the reader 110-1 of the store entry confirmation operation panel 110 with, for example, a membership card on which the membership number is displayed or the display screen of a portable terminal, the store entry confirmation operation panel 110 performs an entry confirmation process. Start (S400). The reader 110-1 of the store entry confirmation operation panel 110 reads the membership number from the held membership card or the display screen of the portable terminal (S401).

When the reading of the membership number is completed, the entry confirmation operation panel 110 uses the camera 110-2 to photograph the face of the visitor 10 (S402). The store entry confirmation operation panel 110 sends the member number read by the reader 110-1 and the image of the face captured by the camera 110-2 to the server 200 (S403).

The server 200 compares the individual member information registered at the time of pre-registration with the member number sent from the entry confirmation operation panel 110 and the image information of the face portion (S404).

As a result of the comparison (S405), if there is member individual information that matches the member number sent from the store entry confirmation operation panel 110 and the face image information (Yes in S405), the server 200: The visitor 10 is judged to be a legitimate user of the store 1 (S406).

The server 200 notifies the store entry confirmation operation panel 110 that the visitor 10 is permitted to enter the store (S407). The store entry confirmation operation panel 110 having received the permission to enter the shop 10 performs a process of permitting the visitor 10 to enter the shop 1 (S408), and ends the shop entry confirmation process (S409). ).

The process of permitting entry is, for example, the process of opening the entrance gate 100, announcing "Please enter the store" by voice, and lighting the notification lamp of the entrance gate 100 in blue.

As a result of the comparison (S405), if there is no individual member information that matches the member number sent from the store entry confirmation operation panel 110 and the face image information (No in S405), the server 200: The visitor 10 is determined to be an unauthorized user of the store 1 (S410).

The server 200 notifies the store entry confirmation operation panel 110 that the visitor 10 is not permitted to enter the store (S411).

The store entry confirmation operation panel 110, having received the notification that the visitor 10 is not permitted to enter the store, performs a process of not permitting the visitor 10 to enter the store 1 (S412), and terminates the store entry confirmation process (S409). ).

The process of not permitting entry is, for example, the process of keeping the door of the entrance gate 100 closed and announcing "cannot enter the store" by voice, or lighting the notification lamp of the entrance gate 100 in red. The process of not permitting entry may include a process of announcing, for example, "press the help button and talk to the staff" in order to guide users who are not permitted to enter the store.

FIG. 5A is a diagram showing the state of product selection in S3 of FIG.

A visitor 10 can enter the store 1 through an entrance gate 100 when determined to be an authorized user A of the store 1.例文帳に追加

User A can freely view and select products displayed in the display area 121 of the shelf 120 .

FIG. 5B is a top view of user A freely viewing and selecting products displayed in the display area 121 of the shelf 120 . As shown in FIG. 5B(A), the partition plate 122 forming the shelf 120 preferably protrudes from the display area 121 toward the position where the user stands. As shown in FIG. 5B(B), a partition 122-P that is separate from the partition plate 122 may be arranged.

As described above, the partition plate 122 of the shelf 120 is protruded from the display area 121 in the direction of the position where the user stands, so that the number of users standing in front of the display area 121 can be reduced. This makes it easier to identify the user who takes out the merchandise displayed in the individual area 124 by means of.

FIG. 6A is a diagram showing an example of arrangement of the distance sensor 131 and the weight sensor 132 arranged in the individual area 124. FIG. As shown in FIG. 6A, one distance sensor 131 and one weight sensor 132 are arranged in one individual area 124 .

The distance sensor 131 is desirably arranged at the edge of the individual area 124 that faces the products displayed in the individual area 124 and is closer to the position where the user stands.

Weight sensor 132 is preferably positioned below the items displayed in discrete area 124 .

The example of FIG. 6A is an example in which a box-shaped product made of cardboard whose exterior is not easily deformed and a bag-shaped product whose exterior is easily deformed are displayed in the individual area 124 . For products whose exterior is easily deformed, for example, if the product is specified using only the image captured by the camera 130, it may be difficult to specify due to the deformation of the exterior. On the other hand, the product movement detection device of the present embodiment can reliably identify the product by combining the distance sensor 131 and the weight sensor 132 even for an object whose exterior is easily deformed.

6B and 6C show the detection result of the distance sensor 131, the detection result of the weight sensor 132, and the camera when the user A takes out the product displayed in the individual area 124 from the individual area 124 in S3 of FIG. FIG. 13 is a diagram showing a processing flow for identifying the product that has been taken out and the user A who took out the product using the video shot by 130;

FIG. 6B is a diagram showing the processing flow of the distance sensor 131 and the weight sensor 132 when identifying the product that has been taken out and the user who took out the product. The distance sensor 131 and the weight sensor 132 arranged for each individual area 124 sequentially perform the sensing process (S602) after starting the specific process (S600).

If the distance sensor 131 detects a change in distance as a result of the sensing process (S602) (Yes in S603), the distance sensor 131 notifies the server 200 of the change in distance and the amount of change in distance (S604). ).

Similarly, when the weight sensor 132 detects a change in weight as a result of the sensing process (S602) (Yes in S605), the weight sensor 132 notifies the server 200 of the change in weight along with the amount of change in weight. (S606). Each notification includes the individual area identification number of the individual area 124 in which the distance sensor 131 and the weight sensor 132 are arranged.

Note that the timing of the sensing process in S602 can be arbitrarily adjusted. The timing of sensing can also be linked with the result of image capturing by the camera 130, for example. That is, only when the user exists nearby, only the distance sensor 131 and the weight sensor 132 arranged in the individual area 124 in the vicinity may perform the sensing process (S602). In this way, it is possible to control the timing of performing the sensing process (S602) according to various conditions such as the position of the user at the store 1 or the size of the store, thereby efficiently operating the store 1. can do. The timing of the sensing process in S602 may be managed and controlled by the server 200, for example.

Detection of change in distance in the process of S603 and detection of change in weight in the process of S605 are determined based on whether or not a preset threshold is exceeded. This threshold can be freely changed for each individual area 124 . Specifically, for example, the threshold for detecting a change in distance by the distance sensor 131 placed in the individual area A is 1 cm, the threshold for detecting a change in distance by the distance sensor 131 placed in the other individual area 124 is 2 cm, Thus, the threshold set in the distance sensor 131 may be changed for each individual area 124 .

Similarly, for example, the threshold for detecting a change in weight by the weight sensor 132 placed in the individual area B is 2 g, and the threshold for detecting a change in weight by the weight sensor 132 placed in the other individual area 124 is 10 g. The threshold value set for the weight sensor 132 may be changed for each individual area 124 as shown.

Alternatively, the detection accuracy of distance change detection in the processing of S603 and the detection accuracy of weight change in the processing of S605 can be freely changed for each individual area 124 . Also, the detection accuracy can be freely changed according to the size of the threshold value described above. For example, when relatively small products are displayed in the individual area A, the distance detection accuracy of the distance sensor 131 arranged in the individual area A is set higher than the detection accuracy of the distance sensor 131 arranged in the other individual area 124. You can set it so that Specifically, for example, the detection accuracy with which the distance sensor 131 arranged in the individual area A detects the distance is 0.5 cm, and the detection accuracy with which the distance sensor 132 arranged in the other individual area 124 detects the distance is 1 cm. The detection accuracy set to the distance sensor 131 , such as a unit, may be changed for each individual area 124 , or distance sensors 131 with different detection accuracy may be arranged for each individual area 124 .

Similarly, for example, when light products are displayed in the individual area B, the weight detection accuracy of the weight sensor 132 arranged in the individual area B is the same as the detection accuracy of the weight sensor 132 arranged in the other individual area 124. may be set to be higher than Specifically, the weight detection accuracy of the weight sensor 132 arranged in the individual area B is 1 g unit, and the weight detection accuracy of the weight sensor 132 arranged in the other individual area 124 is 5 g unit. The detection accuracy set for the weight sensor 132 may be changed for each individual area 124 , or weight sensors 132 with different detection accuracy may be arranged for each individual area 124 .

If the threshold and detection accuracy can be set, the server 200 may manage and control the threshold and detection accuracy.

For example, when small-sized products are displayed in the individual area 124, there is a method of identifying the product using only the image captured by the camera 130, but the size, shape, weight, and other characteristics of the product may be used. Since it is difficult to change the performance of the camera, it may be difficult to identify each product using only the captured images. On the other hand, the product movement detection device of this embodiment uses the distance sensor 131 and the weight sensor 132 to detect the movement of the product. By changing it accordingly, the product can be specified with certainty.

As described above, the product movement detection device of the present embodiment can detect the threshold value and accuracy for detecting that a product has been taken out according to the characteristics of each product displayed in the individual area 124, such as the size, weight, and shape of the product. can be freely changed, it is possible to reliably detect product removal.

FIG. 6C is a diagram showing a processing flow of the server 200 when specifying the product that was taken out and the user A who took out the product.

After starting the identification process (S620), the server 200 performs a reception process each time there is a notification from the distance sensor 131 or the weight sensor 132 (S622). The server 200 analyzes the contents notified from the distance sensor 131 or the weight sensor 132, and confirms whether or not both the notification from the distance sensor 131 and the notification from the weight sensor 132 having the same individual area identification number have been received. (S623). As a result of confirmation, when both the notification from the distance sensor 131 and the notification from the weight sensor 132 having the same individual area identification number are received (Yes in S623), the server 200 detects the change in distance included in both notifications. It is determined whether the product has been removed from the individual area 124 based on the change in volume and weight (S624).

As a result of the determination (S624), if it has been removed (Yes in S625), of the video data of the face portion sequentially sent from the camera 130, the notification from the distance sensor 131 or the notification from the weight sensor 132 is received in S622. The user who removed the product is specified as the user A by comparing the image data near the timing of receiving the member's personal information with the image of the face part registered as the member's individual information for authentication (S626).

The server 200 then registers the information of the product determined to have been removed in S624 in the identified user A's basket A (S627).

As a result of the determination (S625), if it has not been removed (No in S625), the server 200 waits for the next notification without performing any processing.

Each time user A takes out a product displayed in individual area 124, distance sensor 131, weight sensor 132, and server 200 identify the product that has been taken out, identify the user who takes out the product, and use the specified product. Repeat the process of registering the information of the product identified as having been taken out to the customer's basket. As a result, all the products taken out by the user A from the individual area 124 are registered in the basket A of the user A.

In the processing of S626, when identifying the user who removed the product using the image data sequentially sent from the camera 130, the image of the part other than the face may be added. When an image of a portion other than the face is used, it is desirable that the image be an image of a portion of the appearance of the user that is expected to change little between entering and leaving the store.

As explained earlier, the image of the part of the user's appearance that seems to change little from the time they enter the store to the time they leave is, for example, the image of the user's feet, including the shoes and sandals they are wearing. It is a video. The foot portion preferably includes the shoes, sandals, etc. worn by the visitor, and further includes the height corresponding to the ankles and shins. A method of identifying a user who has removed a product displayed in the individual area 124 using an image of a portion of the user's feet may be referred to as foot authentication.

When step authentication is performed, it is necessary to photograph the user's feet with a camera when confirming entry into the store in S2 of FIG. When doing so, it is necessary to photograph not only the face portion but also the foot portion with the camera. The server 200 also stores characteristic image pattern data required for foot authentication, such as characteristic points for each type of shoe such as sandals, sneakers, and boots, and characteristic points of human feet such as toes and heels. are accumulating.

A camera 110-4 (not shown) that captures the user's feet when confirming entry into the store in S2 may be installed below the entry confirmation operation panel 110. FIG. Also, the camera 130 for photographing the user's feet when selecting the product in S3 may be installed below the shelf 120 .

FIG. 6D is a diagram showing a processing flow of store entry confirmation when step authentication is performed. The difference from the process flow of FIG. 4B is that a process (S643) of photographing the feet with the camera 110-4 is added. Other processing from S630 to S642 is the same as the processing from S400 to S412 in FIG.

FIG. 6E is a diagram showing an example of a detailed flow of processing for photographing a portion of the feet in S643 of FIG. 6D.

The store entry confirmation operation panel 110 compares the image input from the camera 110-4 with the characteristic image pattern required for step authentication, which is acquired from the server 200 and stored inside, and determines the angle of the camera 110-4. is adjusted to identify the user's feet (S651). Thereby, the store entry confirmation operation panel 110 recognizes the shoes worn by the user, the shape of the foot, and the like necessary for foot authentication.

When the angle adjustment is completed, the entry confirmation operation panel 110 takes a picture in this state. As a result, the store entry confirmation operation panel 110 acquires image information such as the shape of the user's shoes and feet (S652).

In addition, the store entry confirmation operation panel 110 further uses the angle of the camera 110-4 to acquire images up to the height of the user's shins, such as the socks worn by the user and the hem of trousers and skirts. is further adjusted (S653).

When the angle adjustment is completed, the entry confirmation operation panel 110 takes a picture in this state. As a result, the entry confirmation operation panel 110 acquires not only the shape of the shoes and feet worn by the user, but also the appearance image information up to the height of the shin (S654).

The images acquired by the store entry confirmation operation panel 110 with the camera 110-4 are sequentially sent to the server 200 (S655) and registered in the individual member information together with the image of the face.

FIG. 6F is a diagram showing a processing flow of the server 200 when identifying the product that has been taken out and the user A who took out the product when step authentication is performed. The difference from the processing flow of FIG. 6C is that the user who removed the product is specified (S666) using the image of the face portion and the image of the foot portion sent from the camera. is. Other processes from S660 to S665 and from S667 are the same as the processes from S620 to S625 and from S627 in FIG. 6C. The server 200 compares the image data of the foot portion in addition to the image data of the face portion sent from the camera with the image data of the foot portion registered as the individual member information and authenticates them, thereby removing the product. identified as user A (S666)
When a user is identified by an image, identification accuracy can be improved by using an image of a portion other than the face in addition to the image of the face.

Also, some users may be reluctant to photograph the face. In the sense of considering the user's emotions, it is possible to identify the user using the image of the face part and the user identification using the image of the part other than the face depending on the location etc. where the camera is installed. You can use them properly. In this way, in addition to specifying the user using the image of the face part, specifying the user using the image of the part other than the face is very effective from the viewpoint of improving the satisfaction level of the user of the store 1. target.

When the server 200 registers the information of the product determined to be removed in the process of S624 in the basket A of the user A, the server 200 registers the specific area identification number of the individual area 124 where the removed product was displayed. Register it by replacing it with a generic product name.

For example, assume that a product A is displayed in an individual area A having an individual area identification number A. When the user A takes out the product A from the individual area A, in the process of S624, there is only information that the user took out the product A displayed in the individual area A having the individual area identification number A. are doing. If the product displayed in the individual area A with the individual area identification number A is, for example, a snack A, the information that the product displayed in the individual area A with the individual area identification number A is displayed as a snack A that the user can understand. It is necessary to convert it into information of a specific product name.

The server 200 has a conversion table and a conversion function for converting the information of the product displayed in the individual area 124 having the individual area identification number into a specific product name that the user can understand. In the process of S627, the server 200 uses this conversion table and conversion function to convert the product information that the user A has taken out from the individual area 124 into a specific product name that the user can understand when registering it in the basket A. Convert and register.

Note that the notification sent from the distance sensor 131 and the weight sensor 132 may include time information. In addition, time information may also be attached to captured images sequentially sent from the camera 130 . In this case, the server 200 processes the information sent from the distance sensor 131, the weight sensor 132, and the camera 130, including the time information included in the information sent from the distance sensor 131, the weight sensor 132, and the camera 130, for example, in the processing of S623 and the processing of S627, thereby making the specific processing more efficient. It is possible to

Furthermore, in S627, the server 200 not only registers the information of the product A determined to be removed in S624 in the basket A of the user A, but also notifies the user A that the product A has been registered in the basket A. You may do so.

Notifying user A that product A has been registered in basket A means, for example, installing a speaker (not shown) that emits a notification sound or an LED (not shown) that emits a notification lamp for each individual area 124. , a notification sound is emitted from a speaker installed in the individual area A where the product A is displayed, or a notification lamp is illuminated. Notifying the user that product A has been registered in basket A means that product A has been registered in the basket to the destination address of user A registered in the pre-registration (S1) shown in FIG. It may be a notice to that effect.

The user A can recognize that the product A has been taken out from the individual area 124 and that the taken-out product A has been registered in the basket A by the notification sound, notification lamp, mail, or the like. As a result, the user A can immediately notice that the product taken out by another person from the individual area 124 has been erroneously registered in his/her own basket A, and can quickly solve the problem of product selection. Become.

FIG. 7 is a diagram showing a processing flow for confirmation of leaving the store in S5 of FIG.

When user A holds the display screen of the mobile terminal on which the payment completion notice is displayed over the reader 111-1 of the exit confirmation operation panel 111 in order to exit the exit confirmation operation panel 111, the exit confirmation operation panel 111 displays the exit confirmation operation panel 111. Confirmation processing is started (S700). The reader 111-1 of the store exit confirmation operation panel 111 reads the payment completion notice from the display screen of the portable terminal held up (S701). The exit confirmation operation panel 111 sends the settlement completion notification read by the reader 110-1 to the server 200 (S702).

When the server 200 receives the payment completion notification sent from the exit confirmation operation panel 111, it compares the payment status with the payment status of the products registered in the basket stored in the individual member information (S703).

As a result of the comparison, if the payment has been correctly completed (Yes in S704), the server 200 notifies the exit confirmation operation panel 111 that User A is permitted to exit the store (S705). The store exit confirmation operation panel 111 that has received the permission to leave the store for the user A performs processing for permitting the user A to leave the store 1 (S706), and ends the store exit confirmation processing (S707). ).

The process of permitting leaving the store is, for example, the process of opening the door of the exit gate 101, announcing "thank you for using" by voice, or lighting the notification lamp of the exit gate 101 in blue.

As a result of the comparison, if the payment has not been correctly completed (No in S704), the server 200 notifies the exit confirmation operation panel 111 that User A is not permitted to exit the store (S708). The store exit confirmation operation panel 111, having received the fact that User A is not permitted to leave the store, performs processing not to permit User A to leave store 1 (S709), and terminates the store exit confirmation process (S707). ).

The process of not permitting leaving the store is, for example, the process of keeping the door of the store exit gate 100 closed and announcing "cannot leave the store" by voice, or lighting the notification lamp of the store exit gate 101 in red. The process of not permitting leaving the store may include a process of announcing, for example, "Press the help button and talk to the staff" in order to guide users who are not permitted to leave the store.

Furthermore, the exit confirmation operation panel 111 may have a function of issuing a sticker to be affixed to the paid product to the user who has completed the payment for the product registered in the basket.

The seal to be stuck on the settled commodity is a settlement display sticker that the user of the store 1 puts on the commodity for which the settlement has been completed by himself, and for example, the seal is torn apart when the user tries to peel off the stuck seal.

The sticker to be affixed to the product for which payment has been completed is issued from the sticker issuing machine connected to the exit confirmation operation panel 111 as one of the processes for permitting exit in S706. The issued sticker may include the product name of the product for which payment has been completed. Also, the size of the seal to be issued may be arbitrarily changed according to the size of the product for which payment has been completed.

The product selection processing described above is for the case where the user takes out the product displayed in the individual area 124 by himself/herself, but it may be difficult for some users to take out the product by themselves.

FIG. 8 is a diagram showing an example of another form of the shelf 120 for displaying products.

A monitor 135 , for example, may be arranged on the shelf 120 assuming users who find it difficult to take out products by themselves. This monitor 135 is connected, for example, to a service corner 140 where a clerk of the store 1 is stationed, via the network 201, so that the user can interact with the clerk or give an instruction to reserve the product to be purchased at the service corner 140. It is possible to issue

While several embodiments of the invention have been described, these embodiments have been presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof. Furthermore, in each constituent element of the claims, even if the constituent element is divided and expressed, a plurality of constituent elements are expressed together, or a combination of these is expressed, it is within the scope of the present invention. Moreover, a plurality of embodiments may be combined, and examples configured by such combinations are also within the scope of the invention.

In addition, in this specification and each figure, the same reference numerals are given to components that exhibit the same or similar functions as those described above with respect to the previous figures, and duplicate detailed description may be omitted as appropriate. . In addition, when the claims are expressed as control logic, when expressed as a program including instructions to be executed by a computer, and when expressed as a computer-readable recording medium in which the instructions are written, the apparatus of the present invention is applied. be. Also, the names and terms used are not limited, and other expressions are included in the present invention as long as they have substantially the same content and the same meaning.

100... Entrance gate 101... Exit gate 110... Entrance confirmation operation panel 111... Exit confirmation operation panel 120... Shelf 121... Display area 122 Partition plate 123 Rail 124 Individual area 130 Camera 131 Distance sensor 132 Weight sensor

Claims (6)

A server that manages the products displayed in each of the individual areas that divide the display area of a store that employs an unmanned cash register into one or more and the users of the store;
a first sensor that detects a change in weight at an arrangement position of the product displayed in the individual area and notifies the server as a first notification;
a second sensor that detects a change in distance from the product displayed in the individual area and notifies the server as a second notification;
a third sensor that captures an image including the face and feet of the user present in the individual area and notifies the server of the captured image including the face and feet as a third notification;
an assistance request means for requesting assistance from the server for the operation of the user entering and exiting the store;
with
The server detects that the product is taken out from the individual area when both the change in weight and the change in distance are detected by the first notification and the second notification from the first sensor and the second sensor. By comparing the video including the face and feet notified as the third notification with the pre-registered video of the face and feet, the user was identified and determined to have been extracted. Perform registration processing for the user identified as the product, perform settlement processing by the unmanned cash register based on the result of the registration processing, and perform operations for entering and exiting the store when a request is made by the assistance request means. to assist in the processing of
Product management system. Furthermore, an entry confirmation means for confirming the entry of the user of the shop is provided,
The store entry confirmation means includes a reader that reads a membership number issued to an authorized user of the store at the time of entering the store and notifies it to the server, and photographs the face and feet of the user entering the store. A camera that notifies the server of the image,
The server confirms that the user entering the store is an authorized user based on the member number notified from the store entry confirmation means and the video.
The product management system according to claim 1. Furthermore, it has a store exit confirmation means for confirming the legitimacy when the user of the store leaves the store,
The exit confirmation means permits the user of the store to exit the store when receiving a payment completion notification for the product from the server.
The commodity management system according to claim 2. When the server identifies the user who has moved the product, the server notifies the user that the user has moved the product.
The commodity management system according to claim 1. The server presents the user with information on future bargain products at the store or recommended product information based on the purchase history of the user.
The commodity management system according to claim 1. The server manages the products displayed in each of the individual areas in which the display area of the store adopting an unmanned register is divided into one or more and the users of the store,
A first sensor detects a change in weight at an arrangement position of the product displayed in the individual area and notifies the server as a first notification;
A second sensor detects a change in distance from the product displayed in the individual area and notifies the server as a second notification;
A third sensor captures an image including the face and feet of the user present in the individual area, and notifies the server of the captured image including the face and feet as a third notification,
receiving a request for assistance in operation of the user entering and exiting the store and notifying the server;
The server detects that the product is taken out from the individual area when both the change in weight and the change in distance are detected by the first notification and the second notification from the first sensor and the second sensor. The user is identified by comparing the video including the face and feet notified as the third notification with the pre-registered video of the face and feet, and the product determined to have been taken out. performs registration processing for the user identified as such, performs settlement processing at the unmanned cash register based on the result of the registration processing, and performs assistance processing for entering and exiting the store when the assistance request is received. to do
Merchandise management method.

Images