WO2020075837A1 - Information processing system - Google Patents

Abstract

The present invention enables the automatic payment of the price for a product and improves the accuracy of specifying the product, when a buyer purchases the product. An object recognition unit (233) recognizes an object from an image including the object as a subject. A product specification unit (235) specifies the object as a product, by using a method such as image recognition. A visual inspection result acquisition unit (239) transmits the image to a visual inspection terminal (Q), and acquires a specific result of the product by means of visual inspection transmitted from the visual inspection terminal (Q). A settlement unit (237) performs a settlement process on the product specified on the basis of the result by means of the product specification unit (235) and the visual inspection result acquisition unit (239).

Description

Information processing system

The present invention relates to an information processing system.

Conventionally, in shops such as convenience stores, supermarkets, shopping centers, and various mass retailers, buyers take products from product shelves, put them in a shopping cart or shopping cart, and carry them to a cashier located near the exit of the store, The price of the product is settled by a self cash register (for example, refer to Patent Document 1).

Japanese Patent Laid-Open No. 2001-76261

However, when paying for the products in the shopping cart or shopping cart at the cash register, the barcodes for each of the products will be read even at the self-checkout counter, so it is not possible to eliminate the waiting line at the cash register. If not, the purchaser will have to wait for a long time at the cashier.
Further, when a purchaser purchases a desired product at a store, if there are many other purchasers, the purchaser may give up shopping.
Therefore, in consideration of the above circumstances, when the purchaser purchases the product displayed in the store, it is possible to automate the settlement of the price of the product and shorten the time required for the settlement of the price of the product. The system is required.
Further, in the conventional store, fraud such as shoplifting by shoppers and cashiers is also a problem, and a system capable of preventing such fraud is also required.

The present invention has been made in view of such a situation, and when the purchaser purchases a product, it is possible to automate the settlement of the price of the product, and to improve the identification accuracy of the product. To do.

In order to achieve the above object, an information processing system according to one embodiment of the present invention includes:
First identifying means for obtaining a result of attempting to identify the object as a product by using a first method in which an eye examiner visually confirms an image including the object as a subject;
Settlement means for performing settlement processing for the product identified based on the result of the first identifying means,
Is provided.

According to the present invention, when a purchaser purchases a product, it is possible to provide an information processing system capable of automating the payment of the price of the product and increasing the accuracy of identifying the product.

FIG. 1 is a diagram showing a list of essential points of Embodiments 1 to 4 of an information processing system according to the present invention. It is a schematic diagram showing the outline of the system flow in Embodiments 1 and 2. It is a schematic diagram showing the outline of the system flow in Embodiments 3 and 4. FIG. 3 is a diagram showing a layout example of a convenience store that employs the product recognition system according to the first embodiment. 1 is a schematic perspective view showing an example of the external configuration of a cashier terminal used in Embodiment 1. FIG. It is a block diagram which shows the structure of the goods recognition system as Embodiment 1 of the information processing system of this invention. It is a block diagram which shows the hardware constitutions of a server in the goods recognition system of FIG. It is a block diagram which shows the hardware constitutions of the cashier terminal in the goods recognition system of FIG. It is a functional block diagram which shows an example of a functional structure with the server of FIG. 7 and the cashier terminal of FIG. The example of the imaging screen of the object placed in the cashier terminal is shown. It is a figure which shows an example of the truth table for calculating the number of the objects put on the cashier terminal. 10 is a flowchart illustrating an automatic settlement process executed by the server and the cashier terminal of FIG. 9. 10 is a flowchart illustrating processing of a trade-restricted product in the automatic settlement processing executed by the server and the cashier terminal of FIG. 9. FIG. 9 is a diagram showing a layout example of a bookstore that employs the product recognition system according to the second embodiment. FIG. 9 is a schematic perspective view showing an example in which a book is automatically settled by a cashier terminal adopted in the second embodiment. It is a block diagram which shows the structure of the goods recognition system as Embodiment 2 of the information processing system of this invention. It is a block diagram which shows the hardware constitutions of a sales floor apparatus in the goods recognition system of FIG. FIG. 18 is a functional block diagram showing an example of a functional configuration of the server of FIG. 7, the cashier terminal of FIG. 8, and the sales floor device of FIG. 17. 19 is a functional block diagram showing a detailed functional configuration example of a moving object tracking unit provided in the sales floor device of FIG. 18. FIG. FIG. 19 is a functional block diagram showing a detailed functional configuration example of a position information management unit provided in the sales floor device of FIG. 18. 19 is a functional block diagram showing a detailed functional configuration example of a book number counting unit provided in the sales floor device of FIG. 18. FIG. 19 is a flowchart illustrating an automatic settlement process executed by the server, the cashier terminal, and the sales floor device of FIG. 18. 23 is a flowchart for verifying the number-of-volumes information of the product and the number of volumes to be settled in step S210 of FIG. It is a figure which shows the example of a layout of the supermarket which employs the goods recognition system in Embodiment 3. It is a block diagram which shows the structure of the goods recognition system as Embodiment 3. It is a block diagram which shows the hardware constitutions of a sales floor apparatus in the goods recognition system of FIG. It is a functional block diagram which shows an example of a functional structure of the server of FIG. 7, a settlement machine, and the sales floor apparatus of FIG. FIG. 28 is a functional block diagram showing a detailed functional configuration example of a moving object tracking unit provided in the sales floor device of FIG. 27. It is a functional block diagram which shows the detailed functional structural example of the shelf goods recognition part with which the sales floor apparatus of FIG. 27 was equipped. FIG. 28 is a functional block diagram showing a detailed functional configuration example of a basket product recognition unit provided in the sales floor device of FIG. 27. It is a flowchart explaining the basic flow of the automatic settlement processing which the server of FIG. 27, a sales floor apparatus, and a settlement machine perform. It is a flowchart explaining the process which recognizes the goods in a basket of the automatic settlement process which the server sales floor apparatus of FIG. 27 and a settlement machine perform. It is a figure which shows the example of a layout of the supermarket which employs the goods recognition system in Embodiment 4. It is a block diagram which shows the structure of the goods recognition system as Embodiment 4 of the information processing system of this invention. It is a figure which shows the hardware constitutions of the settlement gate in the goods recognition system of FIG. It is a functional block diagram which shows an example of a functional structure of the server of FIG. 7, the sales floor apparatus of FIG. 26, and the settlement gate of FIG. FIG. 13 is a functional block diagram showing a detailed functional configuration example of a moving object tracking unit included in a sales floor device according to a fourth embodiment. It is a functional block diagram which shows the detailed functional structural example of the shelf goods recognition part with which the sales floor apparatus in Embodiment 4 was equipped. It is a flowchart explaining the automatic settlement process which the server 1, the sales floor apparatus, and the settlement gate of FIG. 36 perform. It is a flowchart explaining the automatic settlement process which the server 1, the sales floor apparatus, and the settlement gate of FIG. 36 perform.

〔Overview〕
The outline of the embodiment of the information processing system of the present invention will be described below.
The information processing system of the present invention is applied as a product recognition system that automatically pays for products. Hereinafter, Embodiments 1 and 2 of a product recognition system including a cashier terminal for specifying a product and Embodiments 3 and 4 of a product recognition system that does not include a cashier terminal for specifying a product will be described respectively.

FIG. 1 is a diagram showing a list of the main points of the first to fourth embodiments of the information processing system of the present invention.
In the “implementation store” column in FIG. 1, each implementation store of Embodiments 1 to 4 is described. For example, the first embodiment is an information processing system that is mainly intended for application in convenience stores.
However, this shop is merely an example, and the application destinations of the first to fourth embodiments are not particularly limited. For example, the implementation store of Embodiment 1 may be a retail store such as a supermarket, a cafeteria, or a store where settlement is performed.
In the “payment place” column in FIG. 1, the place where the shopper pays is described in each of the first to fourth embodiments. The checkout terminal in Embodiments 1 and 2 exemplifies a cashier terminal. The cashier terminal has a function of identifying an object as a product and paying the product. The settlement place in the third embodiment is a cashier table. The cash register has a function to settle a product already specified as a product from the object. The settlement place in the fourth embodiment exemplifies a settlement gate. The checkout gate has a function of paying out a product that has already been specified as a product from an object without placing it on the cash register.

In the "Outline" column in FIG. 1, an outline of each of the first to fourth embodiments is described. For example, in summary, the information processing system according to the first embodiment is an information processing system that performs automatic settlement of a product placed on a cashier terminal.
Details of each embodiment are described in the “Details” column in FIG. 1. That is, for example, the information processing system according to the first embodiment recognizes a hand-held product placed on the cashier terminal by the cashier camera installed on the cashier terminal, and automatically pays the product.

Here, a product that has not been identified as a product sold in the store is called an “object”. In addition, although it is not a product, for example, personal items brought by a shopper are also “objects”. A product in which the "object" is specified as a product is called a "product". Therefore, the "commodity" means a product sold at a store.
In addition, the product recognition system of Embodiments 1 to 4 includes one or more sensing devices that image an object. As the sensing device, various devices such as a temperature sensor and a distance sensor can be adopted in addition to the image sensor (camera or the like).

An image captured by an image sensor such as a camera will be referred to as “captured image” below. Furthermore, a captured image including an object as a subject is hereinafter referred to as an “object captured image”. On the other hand, a captured image including a product as a subject is hereinafter referred to as a “product captured image”.
Further, in the present specification, when various kinds of image processing are performed on the product imaged image and the object imaged image, they are actually handled in the form of data, but in the following, the data will be omitted for the sake of convenience of description.
Here, as the object imaged image and the product imaged image, for example, an imaged image of the entire area where the object of the cashier terminal is placed, an image obtained by individually cutting out the product, a product logo image, a product barcode image, a product image The image of the label, the image of the entire shelf of the product, the image of the inside of the store taken from the ceiling or the like can be used.
Further, in the information processing system of the present invention, when an object is specified as a product, it is estimated which product the subject of the object captured image is, by using various recognition methods for the object captured image, A list of products for which the subject of the image matches and the matching degree thereof (hereinafter, also referred to as “product candidate list”) is generated.

Further, in the information processing systems of the first to fourth embodiments, when the product is settled, the object imaged image or the product imaged image of the object that cannot be specified as the product or the product whose sale is restricted is displayed on the terminal for eye examination. The information is transmitted, and the eye examination terminal identifies the product and determines the trade restriction (determines whether or not to cancel the trade restriction). The sale restriction of the sale-restricted product is released either by the salesclerk responding on the spot in response to the result of the determination or by adopting the result of the visual inspection as a system. Then, the result of the eye inspection by the eye inspector is appropriately referred to, and the product is settled.

FIG. 2 is a schematic diagram showing an outline of the system flow in the first and second embodiments.
As shown in FIG. 2, in the first and second embodiments, (1) an object-captured image is captured and a product is specified by a cashier terminal, and (2) an object or trade-restricted product in which the product cannot be specified. If it is determined that (3), relevant information regarding these objects or products is transmitted to the inspection terminal. As the related information, for example, an object captured image, a product captured image, a recognition result log, a product candidate list, or the like can be adopted. At this time, a configuration may be adopted in which the object captured images of a plurality of frames are transmitted. Then, (4) the eye examination request (reception of related information regarding the eye examination object) is notified by the eye examination terminal, and (5) the eye examiner executes the eye examination. This makes it possible to uniquely identify an object for which a product cannot be identified at the cashier terminal as a product, or to release the trade restriction of the trade-restricted product. Further, (6) the eye inspection result by the eye examiner (the result of specifying the product or the determination result of the trade restriction) is transmitted to the cashier terminal, and (7) the eye inspection result is referred to by the cashier terminal. At this time, it is possible to inform the cashier terminal that the product is a trade-restricted product.
When the result that the product is unspecified is returned even after the eye examination, or when the imaged image of the object (purchaser's face, identification card, etc.) used to judge the release of the trade restriction of the trade-restricted product is unclear, etc. The purchaser may be informed by a screen display or the like at the cashier terminal, or may be instructed by voice. Specific cases that can be considered are (A) when the product is unspecified, the purchaser is requested to replace the products (separate the products from each other, do not stack the products), and (B) determine whether to cancel the trading restriction. When the captured image of the purchaser's face or identification card used for the above is unclear, a request is made to retake the image, (C) the purchaser is guided to use the cashier terminal, and the like.
When providing voice guidance, prerecorded voice may be played. Further, the purchaser and the eye examiner may make a call via a microphone and a speaker provided in the cashier terminal.

Further, FIG. 3 is a schematic diagram showing an outline of the system flow in the third and fourth embodiments.
As shown in FIG. 3, in the third and fourth embodiments, (1) an object imaged image is captured and a product is specified in the store (in the sales floor), and (2) an object or a trade that cannot specify the product. When it is determined that the product is a restricted product, (3) related information about these objects or products is transmitted to the inspection terminal. As the related information, for example, an object captured image, a product captured image, a recognition result log, a product candidate list, position information where the object captured image is captured, information for tracking a product or a shopper in a store, and the like are adopted. be able to. At this time, the object captured images of a plurality of frames may be transmitted. Then, (4) the eye examination request (reception of related information regarding the eye examination object) is notified by the eye examination terminal, and (5) the eye examiner executes the eye examination. This makes it possible to uniquely identify, as a product, an object for which the product could not be identified in the store, or release the trade restriction of the trade-restricted product. Further, (6) the eye inspection result by the eye examiner (the result of specifying the product or the judgment result of the trade restriction) is transmitted to the sales floor device installed in the sales floor, and (7) the eye inspection result is referred to in the sales floor device. To be done. At this time, it is also possible to inform the sales floor device that the product is a trade-restricted product.
When the result that the product is unspecified is returned even after the eye examination, or when the imaged image of the object (purchaser's face, identification card, etc.) used to judge the release of the trade restriction of the trade-restricted product is unclear, etc. The configuration may be such that a purchaser is informed by a screen display or the like in a sales floor device or the like, or a configuration in which voice guidance is provided. Specific possible cases include (A) requesting the purchaser to retake the product (removing it from the shelf, putting it back into the basket) when the product is unspecified, and (B) determining whether to cancel the trading restrictions. When the captured image of the purchaser's face, identification card, or the like used for the above is unclear, a request is made to retake the image, and (C) the purchaser is guided to use the sales floor device.
When providing voice guidance, prerecorded voice may be played. Further, the purchaser and the eye examiner may make a call via a microphone and a speaker provided in the sales floor device or the like.
The first to fourth embodiments of the information processing system in which such a system flow is executed can be specifically realized as the following product recognition system.

The product recognition system according to the first embodiment recognizes the presence of an object based on an object captured image including an object placed on a cashier terminal as a subject.
Specifically, for example, in Embodiment 1, the cashier terminal includes one or more cashier cameras as an example of a sensing device. The cashier camera images a predetermined area of the cashier terminal. Specifically, the cash register camera images a predetermined area before an object is placed. Further, the cash register camera images the predetermined area after the object is placed in the predetermined area. Therefore, in the commodity recognition system of the first embodiment, the cashier terminal compares the captured image before the object is placed in the predetermined area of the cashier terminal with the object captured image after the object is placed in the predetermined area of the cashier terminal. By doing so, the existence of the object is recognized.
In addition, by using a segmentation technique in image recognition other than comparing the captured image before the object is placed in the predetermined area of the cashier terminal with the captured image of the object after the object is placed in the predetermined area of the cashier terminal, The presence of the object may be recognized only from the object captured image after being placed in a predetermined area of the cashier terminal.
In the first embodiment, the cashier terminal specifies which product each of the recognized objects is by an object recognition method by image recognition. Here, for example, a method of identifying a product with high accuracy by creating a product candidate by deep learning and then activating the verification function is adopted.
Further, in the first embodiment, the eye examiner confirms the object placed on the cashier terminal at the eye examination terminal connected to the cashier terminal via the network according to the set conditions. This confirmation result (result of visual inspection) is transmitted to the cashier terminal. The “eye check” means that a person visually confirms an object and makes some conclusion, for example, specifying what kind of product the object is. Hereinafter, such a conclusion will be referred to as a “confirmation result” or an “eye examination result”.
When such a confirmation result (visual inspection result) is transmitted to the cashier terminal, the cashier terminal identifies the final product based on the identification result by the object recognition method based on the image recognition and the confirmation result. I do. It should be noted that the final identification of the product may be performed not by the cashier terminal but by another device (not shown) or a natural person.
Then, in the first embodiment, the cashier terminal recognizes the quantity of the next specified product. In the first embodiment, the product specified next is settled.

The product recognition system according to the second embodiment is applied to a store such as a bookstore. Specifically, the product recognition system according to the second embodiment is described as a space between shelves installed at a sales floor in a bookstore or a table such as a wagon (hereinafter, referred to as “inside the shelf” including the table). ) Object is recognized as the number of books, and when this book is taken, the shopper is tracked until it is placed in the cashier terminal, and when the book is placed in the cashier terminal, the number of books placed is recognized. Then, by identifying the book, the book is recognized as a product and the book is automatically settled. At this time, in the second embodiment, the eye checker confirms the book placed on the cashier terminal at the eye examination terminal connected to the cashier terminal via the network according to the set conditions (the book is sold as Specified as). Therefore, the product recognition system according to the second embodiment may specify the product by further considering this confirmation result (visual inspection result).
The eye checker may visually check the shopper before visually checking the “book”. As a result, even if the tracking by the product recognition system fails, the product can be identified.

The product recognition system of the third embodiment is applied to retail stores such as supermarkets. Specifically, the product recognition system of the third embodiment recognizes baskets (shopping baskets and carts) placed in the sales floors of retail stores such as supermarkets, and tracks baskets moving in the sales floors. In the product recognition system according to the third embodiment, when an object is taken from the shelf, the object is recognized and specified as a product, and the baskets are placed on the cash register so that the list of the products placed in the baskets is displayed. It is read and the item is automatically settled. At this time, in the third embodiment, according to the set conditions, the object taken from the shelf is confirmed by the eye examiner at the eye inspection terminal connected to the camera or the like installed in the sales floor via the network. It Therefore, the product recognition system according to the third embodiment may specify the product by further considering the confirmation result (visual inspection result).

The product recognition system of Embodiment 4 is applied to retail stores such as supermarkets. Specifically, not only the shopper, the shopping cart and the cart placed in the supermarket, but also the baskets including the shopper's my bag and the shopping bag and the shopper are recognized and tracked as moving objects. Then, in the product recognition system of the fourth embodiment, the product is recognized and specified when the object is taken from the shelf, and the product can be automatically settled at the cash register even if the product is not placed in the cashier terminal. At this time, in the fourth embodiment, according to the set conditions, the object taken from the shelf is also confirmed by the eye examiner at the eye inspection terminal connected to the camera or the like installed in the sales floor via the network. To be done. Therefore, the product recognition system according to the fourth embodiment may specify the product by further considering this confirmation result (visual inspection result).

Hereinafter, Embodiments 1 to 4 will be described with reference to the drawings.

[Embodiment 1]
The information processing system of the first embodiment is a product recognition system having a cashier terminal 2 as shown in FIG. 5 which is adopted in a store such as a convenience store as shown in FIG. The information processing system according to the first embodiment enables automatic payment by placing a product on a cashier terminal.

FIG. 4 is a diagram illustrating a layout example when the store that employs the information processing system according to the first embodiment is a convenience store.
A cashier counter 12 is installed near the doorway 11 in the store 10. An unmanned cashier terminal 2 is installed on the cashier counter 12 for automatically paying the merchandise. A manned register 13 is installed next to the cashier terminal 2.
In the store 10, a plurality of shelf racks 14 for displaying products are installed, and the facing shelf racks 14 serve as passages 15 through which shoppers move.
The goods in the shelf are picked up by the shopper who has moved through the aisle 15 and placed in a predetermined area of the cashier terminal 2 (a predetermined area A in FIG. 5 described later). With respect to the products placed in the predetermined area, a plurality of products are collectively specified by the cashier terminal 2 and triggered by a predetermined operation of the shopper on the cashier terminal 2 for automatic payment.
In the manned register 13, as in the conventional case, the clerk recognizes the products one by one by the barcode and makes the payment.

An example of the external configuration of the unmanned cashier terminal 2 will be described with reference to FIG.
FIG. 5 is a schematic perspective view showing an example of the external configuration of the cashier terminal 2.
The cashier terminal 2 includes a surrounding portion 270 that surrounds a predetermined area A in which an object is placed. The surrounding portion 270 includes a top plate portion 271, a bottom plate portion 272, and a pair of side plate portions 273.

A cash register camera 211 that images the predetermined area A is fixed to each of the top plate portion 271 and the pair of side plate portions 273. The cashier camera 211 images an object placed in the predetermined area A.
Although only three cash register cameras 211 are depicted in FIG. 5, five cashier cameras may be provided as described later, and it is sufficient that at least one cashier camera is present, and the number of cashier cameras is not limited. The cashier terminal 2 also includes an external camera 212 that images the face, hands, etc. of the shopper.

The housing portion 275 is installed on the bottom plate portion 272. A front surface of the housing 275 is provided with a receipt output unit and a display unit (not shown in FIG. 5) (a receipt output unit R and a display unit D of the output unit 206 of FIG. 8 described later).
A semitransparent plate 276 on which an object is placed is installed on the housing 275. The plate surface on the upper surface of the plate 276 is set as the predetermined area A. The board surface of the plate 276 is formed in a wave shape. The wave shape is not limited to a sine wave shape, and may be a rectangular wave. Furthermore, the pitch and amplitude may be not only uniform but also uneven.
The plate 276 is configured such that the predetermined area A is formed by repeatedly forming the concave portion and the convex portion in this manner, so that at least a part of a columnar or spherical object is sandwiched between the convex portion and the convex portion, You can prevent it from rolling.

An illumination unit 221 that illuminates a predetermined area A is provided inside the plate 276 and the top plate portion 271 of the surrounding portion 270. The lighting unit 221 may be included in the side plate portion 273 of the surrounding portion 270.
The illumination unit 221 emits not only white light but also various colors such as blue and red, which are not limited. The illumination unit 221 emits light so that the shadow of an object placed in the predetermined area A is not generated or reduced in the predetermined area A.

The surrounding section 270 determines the state of the cashier terminal 2, for example, the state of normal standby, the state of settlement, the state of operation by a store clerk, and the occurrence of an abnormal situation. The color of the presentation unit 210 can be changed so that the color can be visually recognized.
At least the top plate portion 271 and the side plate portion 273 of the surrounding portion 270 may be configured with an instantaneous light control sheet so that the transparent state becomes transparent and the opaque state becomes opaque.
In that case, the visibility of the predetermined area A can be ensured by making the surrounding portion 270 transparent. Since the surrounding portion 270 is in the opaque state, it is possible to acquire the captured image of the object while suppressing the influence of external light during shooting.

Such a cashier terminal 2 is incorporated in the product recognition system as the first embodiment of the information processing system.
FIG. 6 is a configuration diagram showing a configuration of a product recognition system as the first embodiment of the information processing system of the present invention.
The product recognition system according to the first embodiment includes a server 1, n (n is an arbitrary integer value of 1 or more) cashier terminals 2-1 to 2-n, and a visual inspection terminal Q. .
The server 1, the n cashier terminals 2-1 to 2-n, and the inspection terminal Q are connected to each other via a network N such as the Internet. The inspection terminal Q can be possessed by a store clerk distant from the cashier terminal 2, provided in a backyard of the store, or provided in a call center remote from the store.

Note that, for convenience of explanation, only one server 1 is depicted in FIG. 6, but in reality, there may be a plurality of servers.
Further, hereinafter, when it is not necessary to individually distinguish the cashier terminals 2-1 to 2-n, they are collectively referred to as a “cash register terminal 2”.

The server 1 executes necessary processing so that the respective operations of the plurality of cashier terminals 2 are performed in harmony. The cashier terminal 2 is placed on the cashier counter 12 shown in FIG. At the cashier terminal 2, the number of objects placed by the shopper in the predetermined area A of the cashier terminal 2 is specified, and then the merchandise is specified and automatically settled.
However, the cashier terminal 2 does not necessarily need to have the server 1 and can function independently. In this case, some or all of the functions of the server 1 are possessed by another information processing device (for example, the cashier terminal 2).

FIG. 7 is a block diagram showing the hardware configuration of the server 1 in the information processing system according to the first embodiment shown in FIG.
The server 1 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a bus 104, an input / output interface 105, an output unit 106, and an input unit 107. The storage unit 108, the communication unit 109, and the drive 110 are provided.

The CPU 101 executes various processes according to a program stored in the ROM 102 or a program loaded from the storage unit 108 into the RAM 103.
The RAM 103 also stores data necessary for the CPU 101 to execute various processes.

The CPU 101, the ROM 102, and the RAM 103 are connected to each other via a bus 104. An input / output interface 105 is also connected to the bus 104. An output unit 106, an input unit 107, a storage unit 108, a communication unit 109, and a drive 110 are connected to the input / output interface 105.

The output unit 106 includes a display, a speaker, and the like, and outputs various kinds of information as an image or sound.
The input unit 107 includes a touch panel, a keyboard, a mouse, a microphone, etc., and inputs various information.

The storage unit 108 includes a hard disk, a DRAM (Dynamic Random Access Memory), and the like, and stores various data.
As illustrated in FIG. 6, the communication unit 109 communicates with the cashier terminal 2 via the network N including the Internet.

A removable medium 120 including a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted on the drive 110. The program read from the removable medium 120 by the drive 110 is installed in the storage unit 108 as needed.
In addition, the removable medium 120 can also store various data stored in the storage unit 108, similarly to the storage unit 108.

FIG. 8 is a block diagram showing a hardware configuration of the cashier terminal 2 in the information processing system according to the first embodiment shown in FIG.
The cashier terminal 2 includes a CPU 201, a ROM 202, a RAM 203, a bus 204, an input / output interface 205, an output unit 206, an input unit 207, an illumination unit 221, a light shielding unit 209, a presentation unit 210, and a cash register. The camera 211, the storage unit 208, the communication unit 213, and the drive 214 are provided.
The removable medium 220 is appropriately attached to the drive 214.

The CPU 201, the ROM 202, the RAM 203, the bus 204, the input / output interface 205, the storage unit 208, the communication unit 213, the drive 214, and the removable medium 220 of the cashier terminal 2 are configured similarly to those of the server 1.

The output unit 206 is provided in the housing unit 275 shown in FIG. The output unit 206 includes a display unit D that displays information about products and information about settlement, a receipt output unit R that outputs a receipt, and a speaker S that outputs a voice.
The input unit 207 is provided in the housing unit 275 shown in FIG. The input unit 207 includes a touch panel (not shown), a card reader unit C, and a microphone M. Further, a bar code reader (not shown) may be provided.

The light shielding unit 209 switches the surrounding unit 270 shown in FIG. 5 between a transparent state and an opaque state when the surrounding unit 270 is formed of an instant light control sheet.
The presenting unit 210 can recognize whether the cashier terminal 2 is in a normal standby state, a settlement state, a store clerk in operation, an abnormal situation, or the like. Then, the presentation unit 210 shown in FIG. 5 is switched to emit light in a different color. The presentation unit 210 is provided not only on the front surface but also on the back surface.
The cash register camera 211 images an object placed in the predetermined area A, and outputs one or more captured images obtained as a result as an object captured image.

Since the hardware configuration of the inspection terminal Q is the same as the hardware configuration of the server 1, for the hardware configuration of the inspection terminal Q, refer to FIG. The description is omitted.

FIG. 9 is a functional block diagram showing an example of the functional configuration of the server 1, the cashier terminal 2, and the eye examination terminal Q.
In the CPU 101 of the server 1, the DB management unit 141 that manages personal information, product information, and the like functions. The DB management unit 141 may be provided in each cashier terminal 2.
A product DB 131 is provided in an area of the storage unit 108 of the server 1. The product DB 131 is a DB (Data Base) that stores product information.

In the CPU 201 of the cashier terminal 2, as shown in FIG. 9, a light emission control unit 228, a light shielding control unit 229, a presentation control unit 230, a personal authentication unit 231, an image acquisition unit 232, and an object recognition unit 233. The object quantity recognition unit 234, the product identification unit 235, the trade restricted product determination unit 236, the settlement unit 237, the display control unit 238, and the visual inspection result acquisition unit 239 function.
The cashier terminal 2 includes a DB information holding unit 241 that holds personal information, product information, and the like.
In the CPU 101 of the eye examination terminal Q, the image display control unit 411 and the eye examination result transmission unit 412 function.

The light emission control unit 228 of the CPU 201 of the cashier terminal 2 performs control for switching between a state in which the illumination unit 221 emits light at the timing of capturing an object and a state in which the illumination unit 221 does not emit at the timing of not capturing an object, or in the predetermined area A. The control for switching the emission color of the illumination unit 221 is executed according to the situation of recognizing an object to be placed.
The light-shielding control unit 229 switches the surrounding unit 270 between an opaque state and a transparent state when the surrounding unit 270 is formed of an instantaneous light control sheet. That is, the light-shielding control unit 229 controls the light-shielding unit 209 provided in the surrounding unit 270 from one of the opaque state at the timing of capturing an image of an object placed in the predetermined area A and the transparent state at the timing of not capturing the image. Execute control to switch to.
The presentation control unit 230 executes control so that the presentation unit 210 changes the emission color for presenting the state of the cashier terminal 2.

The personal authentication unit 231 performs personal authentication of the shopper by referring to the personal information managed by the DB management unit 141 during the settlement process. Specifically, the personal authentication unit 231 performs authentication processing using authentication methods such as face authentication, card authentication, fingerprint authentication, vein authentication, various biometric authentication such as iris authentication before image recognition of the product. . The personal information managed by the DB management unit 141 includes information such as age, allergy, and halal. Therefore, the personal information acquired by the personal authentication unit 231 is utilized by the trading restricted product determination unit 236.
The image acquisition unit 232 acquires the object captured image captured by the cashier camera 211 with the object placed in the predetermined area A as shown in FIG. 10.

FIG. 10 shows an example of an image pickup screen of an object placed on the cashier terminal 2. Here, FIG. 10 is a diagram showing an example of an object imaged image obtained as a result of the objects X, Y, Z, and Z ′ placed in the predetermined area A being imaged by each of the three cash register cameras 211.
FIG. 10A illustrates an example of an object imaged image obtained as a result of imaging the objects X, Y, Z, and Z ′ by the cashier camera 211 fixed to the top plate portion 271 of the cashier terminal 2. This object captured image contains six objects. In this object-captured image, there are no objects X, Y, Z, and Z ′ that are shadows of the object, so all the objects X, Y, Z, and Z ′ placed in the predetermined area A are captured.
FIG. 10B shows an example of an object imaged image obtained as a result of imaging the objects X, Y, Z, and Z ′ by the cashier camera 211 fixed to one side plate portion 273 of the cashier terminal 2. This object captured image includes two objects. In this object captured image, the objects X, Y, Z, and Z ′ on the cash register camera 211 side hide the objects on the back side.
FIG. 10C shows a captured image obtained as a result of capturing the objects X, Y, Z, and Z ′ by the cashier camera 211 fixed to the other side plate portion 273 of the cashier terminal 2. This object captured image includes two objects. In this object captured image, the objects X, Y, Z, and Z ′ on the cash register camera 211 side hide the objects on the back side.
Of the two objects Z included in the respective object captured images of FIGS. 10B and 10C, one is the same object Z ′, but the other is a different object.

The object recognition unit 233 recognizes the presence of an object placed in the predetermined area A from the object captured image acquired by the image acquisition unit 232 using the above-described predetermined image recognition method. That is, the object recognizing unit 233 compares the background image before the object is placed in the predetermined area A of the cashier terminal with the imaged image of the object after the object is placed, and performs the background subtraction process on each object region for each object. The existence of an object is recognized by defining (specifying).
The object recognizing unit 233 does not compare the background image before the object is placed in the predetermined area A with the object captured image after the object is placed using a method other than the background difference processing, and only the object captured image is obtained. The existence of an object may be recognized by defining the object area from.

The object quantity recognition unit 234 recognizes the quantity of objects placed in the predetermined area A by comparing the object recognition quantity with the settlement quantity. The object recognition quantity is the quantity of objects recognized by the object recognition unit 233 from the object captured images captured by the plurality of cash register cameras 211 of the cashier terminal 2.
Settlement quantity is the quantity of the product to be settled.
The object recognition quantity may vary depending on the cashier camera 211, as shown in FIG. That is, in FIG. 10A, six objects are imaged, but in FIGS. 10B and 10C, two objects are imaged.
In such a case, the object quantity recognition unit 234 recognizes the quantity of objects by taking the logical sum as shown in FIG.

FIG. 11 is a diagram showing an example of a truth table for calculating the number of objects placed on the cashier terminal 2.
FIG. 11 shows images of the objects X, Y, Z, and Z ′ shown in FIG. 10 taken by the first to fifth cash register cameras 211 provided in the cashier terminal 2 (FIG. 5). , "0" indicates that the image could not be captured. It should be noted that the object Z and the object Z ′ are the same object, but since the cashier cameras 211 that have imaged them are different, they represent that they are imaged in different states.
In FIG. 11, the first cash register camera 211 is represented by “cash table camera 1”, and similarly, the second to fifth cash register cameras 211 are represented by “cash table camera 2” to “cash table camera 5”. ing.
In the example of FIG. 11, the object X is imaged by the first, fourth, and fifth cashier cameras 211. The object Y is imaged by the second, fourth and fifth cashier cameras 211. The object Z is imaged by the second, third and fifth cashier cameras 211. The object Z ′ is imaged by the first and third cashier cameras 211.

In the example of FIG. 11, there is no product imaged by all the first to fifth cash register cameras 211. Further, although the object Z and the object Z ′ are the same product, they are imaged as different objects because the cashier cameras 211 that image the images are different. It is considered that this is because the cashier cameras 211 for imaging have different angles.
Based on this, the object quantity recognition unit 234 recognizes the quantity of the product by the method using the logical sum. That is, even if objects are overlapped and placed in the predetermined area A, or even if one object is placed in the predetermined area A so as to be a shadow of another object, the logical sum of the objects imaged by one of the cash register cameras 211 is obtained. It is recognized that the object is placed in the predetermined area A by using.

In this way, when the object recognition quantity recognized using the truth table from a plurality of object captured images and the settlement quantity are not the same, the object quantity recognition unit 234 informs the display control unit 238 that the quantities are different. Output.

Returning to FIG. 9, the product identification unit 235 matches the object whose existence is recognized by the object recognition unit 233 with the product information held in the DB information holding unit 241. That is, the product identification unit 235 first lists up product candidates by image recognition methods such as specific object recognition, general object recognition, and deep learning. The listed product candidates are called “product candidate list S”. After that, the product specifying unit 235 causes the verification function to be performed and specifies the product with high accuracy.

The verification function is a function of listing the "commodity candidate list P" by an algorithm different from the method of listing the product candidates described above. The results of the product candidate lists S and P are matched with each other, and a product is specified when the result exceeds a predetermined threshold.
As a method of listing the “commodity candidate list”, for example, it is realized by a method of matching the image information of the object obtained from the object whose existence is recognized with the image information held in the DB information holding unit 241 or the memory. May be. That is, when the feature information of both images matches (exceeds a threshold value), the object whose existence is recognized by the object recognition unit 233 is a product registered in the DB information holding unit 241, and thus the product specification unit 235 , And specifies that the product is registered in the DB information holding unit 241.

Here, a specific example in which the product specification unit 235 outputs the result as product non-specification when the product cannot be specified by the image processing method will be described. First, the product identification unit 235 compares the product image stored in the DB information storage unit 241 with the object captured image captured in the predetermined area A, and the feature points (similar feature points) and the features that are similar to each other in the images. Calculate the amount. Next, the product identification unit 235 reads the feature points and the feature amounts of the images of the products included in the product candidate list from the product DB 131 of the server 1. Next, the product identification unit 235 compares the feature amount for each feature point of the product included in the read product candidate list with the feature amount for each feature point of the recognized object, and saves it in the DB information holding unit 241. The similar feature points of the displayed product image and the object captured image captured in the predetermined area A are matched.

Next, the product identifying unit 235 compares the positional relationships using the corresponding point coordinates of each set of similar characteristic points, and does not correctly correspond (change in positional relationship) in changes due to rotation and translation. Is removed and the number of remaining similar feature points is calculated. The product specifying unit 235 determines that the product is unspecified when the number of similar feature points is less than the threshold value.

In such a method of making products unspecified, products such as lunch boxes that are not packaged products may be unspecified products. The bento may have a slightly different side dish position, and in such a case, it may be an unspecified product. In such a case, the product specification unit 235 detects various codes such as a multi-dimensional code including a bar code and the like written on a label attached to the product and characters such as a product name, and the DB information holding unit 241 or The product name or product code may be read based on the product information stored in the product DB 131 of the server 1 to specify the product.

Furthermore, the product identification unit 235 verifies similar products and related products (hereinafter referred to as “group products”) using the product information stored in the DB information storage unit 241 and the product DB 131 of the server 1. In the product specification unit 235, for example, for group products of series having different sizes, colors, etc., thresholds are compared using characteristics such as size and color of products. The product specifying unit 235 determines that the product is unspecified when the threshold is less than a predetermined threshold.

In the present embodiment, a mode in which the eye inspection terminal Q performs eye inspection for a specific object (hereinafter, referred to as “specific object eye inspection mode”) and all the objects are inspected by the eye inspection terminal Q. The mode (hereinafter, referred to as “all-object visual inspection mode”) in which is performed can be set (the same applies to each of the following embodiments).
When the specific object visual inspection mode is set, the product identification unit 235 transmits the object captured image of the object that is not specified as the product and the related information to the visual inspection terminal Q via the visual inspection result acquisition unit 239. Send to. On the other hand, when the all-object visual inspection mode is set, the product identification unit 235 determines the product identification result and the product captured image of the identified product, the object captured image of the object not identified as the product, and related information. And the like are transmitted to the inspection terminal Q via the inspection result acquisition unit 239.
Then, the product identification unit 235 identifies the product based on the eye inspection result acquired by the eye inspection result acquisition unit 239 from the eye inspection terminal Q. Specifically, for the product identified by the product identification unit 235, when the identification result is approved or corrected by the inspection result, the product identification unit 235 identifies the product according to the content indicated by the inspection result. . In addition, for an object for which the product specifying unit 235 has determined that the product is unspecified, the product specifying unit 235 specifies the object as a product indicated by the visual inspection result.

The trade-restricted product determination unit 236 determines whether or not the product identified by the product identification unit 235 corresponds to the trade-restricted product, based on the determination information.
In addition, the trade-restricted products include (A) products such as cigarettes and alcoholic beverages that cannot be purchased until a certain age is reached, (B) products whose expiry date and expiration date have expired, and (C) allergic ingredients According to the above, products that should not be ingested, products that are restricted by religion, such as (D) products other than halal foods, correspond.
In the present embodiment, the trade-restricted product determination unit 236 transmits the product imaged image of the product corresponding to the trade-restricted product to the eye inspection terminal Q via the eye inspection result acquisition unit 239. At this time, the trade-restricted product determination unit 236 determines an image (face image, identification card image, etc.) relating to the shopper according to the type of the trade-restricted product (above (A) to (D), etc.). The information is appropriately transmitted to the inspection terminal Q via the inspection result acquisition unit 239. At this time, when the image relating to the shopper is unclear, the purchaser may be informed by screen display or voice guidance at the cashier terminal 2 and the image retake may be requested. Further, the purchaser and the eye examiner may make a call via the microphone M and the speaker S provided in the cashier terminal 2.
Then, the trade-restricted product determination unit 236 determines whether or not the sale of the trade-restricted product is permitted based on the eye inspection result acquired by the eye-inspection result acquisition unit 239 from the inspection terminal Q (the sale-restriction is released). Whether or not) is determined.

The settlement unit 237 calculates the total amount of money for all the products identified by the product identification unit 235. At this time, when the trading restricted products determination unit 236 determines that there are trading restricted products, it is necessary that the restrictions are released for all of the trading restricted products. Therefore, the settlement section 237 reads out the price of the product placed in the predetermined area A from the DB information holding section 241, and displays it on the display section D (FIG. 8).

The display control unit 238 controls the output unit 206 to warn the shopper or the clerk to confirm the quantity of the product when the object quantity recognition unit 234 does not match the object recognition quantity with the settlement quantity. To do. The display control unit 238, when the product is not specified by the product specifying unit 235, or when the product is not specified by the visual inspection in the inspection terminal Q, alerts the shopper or the clerk (object). The output unit 206 is controlled so as to output (replacement instruction, etc.). Specifically, the purchaser may be notified by a screen display or voice guidance, and the purchaser may be requested to replace the product. Further, the purchaser and the eye examiner may make a call via the microphone M and the speaker S provided in the cashier terminal 2. The display control unit 238 controls the output unit 206 to warn the shopper or the clerk that the object is a trade-restricted product when the object is determined to be a trade-restricted product by the trade-restricted product determination unit 236. To do. Further, when the total amount is calculated by the settlement unit 237, the output unit 206 is controlled to display the product name, price, etc. of the product to the shopper and the clerk.

The eye inspection result acquisition unit 239 requests the identification of the product by the eye inspection by transmitting the object captured image, or the product identification result and the product captured image of the identified product to the eye inspection terminal Q. The eye inspection result transmitted from the eye inspection terminal Q is acquired in response to the request. The visual inspection result acquisition unit 239 outputs the acquired visual inspection result (commercial product identification result) to the commercial product identification unit 235. Further, the eye inspection result acquisition unit 239 requests the determination of the sale restricted product by the eye inspection by transmitting the product imaged image of the product corresponding to the sale restricted product to the eye inspection terminal Q, and the eye inspection terminal Q. Get the result of the eye inspection sent from. At this time, the visual inspection result acquisition unit 239 appropriately transmits an image (a face image, an image of an identification card, etc.) regarding the shopper to the visual inspection terminal Q according to the type of the restricted sale product. The eye inspection result acquisition unit 239 outputs the acquired eye inspection result (determination result of the sale-restricted product) to the sale-restricted product determination unit 236.
For the products for which the trade restriction product determination unit 236 determines that the object is a trade restriction product, and the trade restriction is not released even by the result of the visual inspection, a warning to that effect is presented from the presentation unit 210 and the output unit 206. The clerk who notices this presentation collects the trade-restricted products so as not to sell the trade-restricted products.
The product recognition system may be provided with a remote operation unit for instructing cancellation of the trading restriction for the trading restricted product, in addition to the inspection terminal Q. The remote control unit can be possessed by a clerk distant from the cashier terminal 2 or provided in the backyard of the store. A single remote control unit can also remotely control a plurality of cashier terminals 2.

In the eye inspection terminal Q, the image display control unit 411 displays images such as the object imaged image or the product imaged image transmitted from the cashier terminal 2 and these images when the cashier terminal 2 requests the eye inspection. Various information (commodity candidate list, etc.) transmitted in association with is output to the output unit 106.
The eye inspection result transmission unit 412 receives the object inspection image or the product captured image output to the output unit 106 from the eye inspection result input by the eye examiner through the input unit 107 (determination of product identification result or trade restricted product). (Result etc.) is transmitted to the cashier terminal 2 which requested the eye examination.

The product recognition system applied as the information processing system of the present specification includes a product registration system for imaging the appearance of a product sold at the store 10 and registering it together with product information such as the price of the product. . The product registration may be performed in the store 10 or may be performed outside the store such as the manufacturer or wholesaler of the product, and the location is not limited.
The product registration system includes a registration image generation unit (not shown), a product master registration unit, a captured image acquisition unit, and a recognition unit.

The registration image generation unit generates an image of a product that can be placed in the predetermined area A of the cashier terminal 2 as a product registration image.
The product master registration unit associates and registers the product registration image generated by the registration image generation unit and the product identifier uniquely assigned to the product included in the product registration image as a subject.
The captured image acquisition unit acquires a captured image of an object placed in a predetermined area A of the cashier terminal 2 as an object captured image. Based on the object captured image and the product registration image, the product specifying unit 235 specifies which product is the object whose existence is recognized.
The object recognition unit 233 recognizes the presence of an object placed in the predetermined area A based on the acquired object captured image.
This product recognition system matches the object captured image of the object whose existence is recognized with the image of the product held in the DB information holding unit 241 or the storage unit 108 of the server 1 as described above, and thereby the product specifying unit. 235 identifies which product.

In the store 10 in which such a product recognition system is installed, a product registration image is generated, and a product identifier is given to a product included in the product registration image, whereby master registration by the product identifier uniquely given to the product is performed. It will be possible. Furthermore, the product recognition system provided with this product registration system can manage the products for which the barcode sticker cannot be attached.

Here, a product settlement method in the product recognition system according to the first embodiment will be described with reference to FIG.
FIG. 12 is a flowchart illustrating a product settlement process executed by the server 1, the cashier terminal 2, and the eye examination terminal Q in FIG. 9.
Prior to the automatic payment processing of the product, an image of the predetermined area A captured in advance in a state where no object is placed in the predetermined area A of the cashier terminal 2 is stored in the image acquisition unit 232.
The image of the predetermined area A captured in advance is updated at a predetermined timing, is not updated every time the shopper uses the cashier terminal 2, and may be shared even if the shopper changes.
Then, when the shopper presses the operation button of the cashier terminal, the automatic settlement process is started.

In step S101, the cashier camera 211 of the cashier terminal 2 captures an image after an object is placed on the predetermined area A of the cashier terminal 2. The image is input to the image acquisition unit 232 as a captured image of an object as illustrated in FIG. 10, for example.
In the object-captured image, the inside of the cashier terminal 2 is illuminated by the illumination unit 221, so that the shadow of the object is not generated or reduced.
The object captured image may be input to the display control unit 238 and output from the output unit 206.
In addition, the external camera 212 of the cashier terminal 2 images the shopper, and the personal authentication unit 231 personally authenticates the shopper.

In step S102, the object recognition unit 233 of the cashier terminal 2 recognizes the presence of an object placed in the predetermined area A from the object captured image acquired by the image acquisition unit 232 using the above-described predetermined image recognition method. To do. That is, the object recognizing unit 233 compares the background image before the object is placed in the predetermined area A of the cashier terminal 2 with the captured image of the object after the object is placed, and performs the background subtraction process on each object region for each object. The existence of an object is recognized by defining (specifying).

In step S103, the product specifying unit 235 of the cashier terminal 2 determines whether or not there is an object whose product cannot be specified among the objects placed in the predetermined area A of the cashier terminal 2.

In step S104, the product specification unit 235 of the cashier terminal 2 determines whether the specific object inspection mode or the all object inspection mode is set.
If the specific object visual inspection mode is set, the process proceeds to step S106.
If the all-object visual inspection mode is set, the process proceeds to step S105.

In step S <b> 105, the eye inspection result acquisition unit 239 of the cashier terminal 2 requests the eye inspection terminal Q to perform the eye inspection (specification of the product) for the target object or product, and acquires the eye inspection result.
After step S105, the process proceeds to step S107.

In step S105, when there is an object whose product is not specified even by the visual inspection at the visual inspection terminal Q, the presentation unit 210 of the cashier terminal 2 notifies the error state by color, sound, or the like.
When an error occurs due to unspecified goods, the object may be replaced and a predetermined operation may be performed on the cashier terminal 2 to return to step S101 and image the object again. The replacement may be performed by a store clerk who has found the error state or may be performed by the shopper himself. The purchaser and the eye examiner may make a call via the microphone M and the speaker S provided in the cashier terminal 2.
It should be noted that the error state is that when the system processing of the cashier terminal 2 becomes abnormal, the characteristic part of the object cannot be imaged, the objects overlap, or become a shadow, the trade-restricted product is This is the case when a product or personal property is left behind in the cashier terminal 2 when it is placed in the terminal 2. When the objects are overlapped with each other or have a shadow, the objects can be replaced to pick up which product the object is and specify the product.

If there is an object whose product identification unit 235 cannot identify which product (YES in step S105), the process proceeds to step S105.

If there is no object whose product specifying unit 235 cannot specify which product in step S106 (NO in step S103), the product specifying unit 235 stores the product held in the DB information holding unit 241 or the storage unit of the server 1. The product is specified by including information such as name, price, and restricted product. As a result, the process proceeds to step S107. The identified product information may be output to the display control unit 238.

In step S107, the trade-restricted product determination unit 236 executes the process of the trade-restricted product. The processing of the trade-restricted product is a process of determining whether the specified product is a trade-restricted product. Specifically, the trade-restricted product determination unit 236 determines, for example, whether the specified product is an age-restricted product, whether it is a product that does not correspond to the halal or a product that contains an allergic component, or whether it is an expired product or an expired product. To do. Details of the processing of the trade-restricted product will be described later with reference to FIG.
Note that the trade-restricted product determination unit 236 can execute the process of the trade-restricted product for each shopper by acquiring the personal authentication in advance at any timing up to step S105 (including before step S101). it can.

In step S108, the settlement unit 237 settlements the products placed in the predetermined area A.
Specifically, the settlement unit 237 obtains the prices of the products specified in step S103 individually and totals them to settle all the products placed in the predetermined area A.
The product information such as the product name and price of the adjusted product is output from the display control unit 238 to the output unit 206, displayed on the display unit D of the output unit 206, and printed on the receipt from the receipt output unit R and output. It
Further, since the cashier terminal 2 is connected to the server 1 via the communication unit 213, it can be utilized as a POS (Point Of Sale). That is, the cashier terminal 2 links the payment item purchase information and the estimated age / sex information to the POS system.
In step S108, when the quantity of the object specified by the object quantity recognition unit 234 and the quantity of the product specified by the product specification unit 235 are different from the settlement quantity, the presenting unit 210 of the cashier terminal 2 indicates that fact. May be presented. The cashier terminal 2 has a cancel function, so that the checkout process can be stopped.

Next, the processing of the trade-restricted product in step S107 will be described in detail with reference to FIG.
FIG. 13 is a flowchart for explaining the processing of the trade-restricted product executed by the server 1, the cashier terminal 2, and the inspection terminal Q of FIG. The sale restriction of the sale-restricted product is released by either the store clerk on the spot in response to the result of the determination or the result of the visual inspection is adopted as a system. In FIG. 13, a case where the result of the visual inspection is adopted as the system will be described.

In step S111, the trade-restricted product determination unit 236 determines whether or not the product identified by the product identification unit 235 is an alcoholic beverage or other product that requires age confirmation.

If it is determined in step S111 that the product identified by the product identification unit 235 is a product that requires age confirmation, that is, if YES, the process proceeds to step S112.
In step S112, the cashier terminal 2 uses the external camera 212 to acquire an image of the purchaser's face and age confirmation document. However, if the personal information of the shopper is acquired and it is not necessary to confirm the age here, step S112 is skipped and the process proceeds to step S116.
In step S113, the trading restricted product determination unit 236 of the cashier terminal 2 transmits the acquired image to the eye inspection terminal Q via the eye inspection result acquisition unit 239 to request the eye inspection. The eye examiner who receives this image confirms the age of the purchaser based on the image.
At this time, if there is a problem with the purchaser's age authentication, such as when the purchaser's face or an image of the age confirmation document is unclear, the purchaser is notified by a screen display or voice guidance at the cashier terminal 2. Alternatively, the purchaser may be requested to retake the image. Further, the purchaser and the eye examiner may make a call via the microphone M and the speaker S provided in the cashier terminal 2.
In step S114, the trade-restricted product determination unit 236 determines whether or not an instruction to release the trade restriction has been received from the inspection terminal Q via the inspection result acquisition unit 239.

When it is determined in step S114 that the instruction to cancel the trading restriction has not been received, the process proceeds to step S115.
If it is determined in step S114 that the cancel instruction for canceling the trading restriction has been accepted, that is, if YES, the process proceeds to step S116.
In step S115, the visual inspection result transmission unit 412 of the visual inspection terminal Q transmits a warning that the trading restriction has not been released even by the visual inspection result. By receiving this warning, for example, the display control unit 238 of the cashier terminal 2 presents, via the output unit 206, a warning that the trading restriction has not been canceled even by the result of the visual inspection. After step S115, the trading restriction process ends.
In step S116, the trade restriction product determination unit 236 cancels the trade restriction.
In this way, when step S116 is completed, or when it is determined in step S111 that the product is not age-restricted (NO is determined), the process proceeds to step S117.

In step S117, the trade restriction product determination unit 236 determines whether the product identified by the product identification unit 235 is a product other than a halal (permitted) food product or an allergic product. To do.
When the trade-restricted product determination unit 236 determines that the product is a product other than a halal product (a product that is not allowed) or an allergy product, the process proceeds to step S118.

In step S118, the display control unit 238 causes the cashier display unit D to display that the product is a product other than a halal product or an allergic product. However, when the personal information of the shopper is acquired and it is not necessary to determine here whether the product is a non-halal product or an allergy product, step S118 is skipped and the process proceeds to step S121.

In step S119, the trading restriction product determination unit 236 determines whether or not an instruction to cancel the trading restriction has been received.
If it is determined that the instruction to release the trading restriction is not received (NO is determined), the process proceeds to step S115.
If it is determined that the instruction to cancel the trading restriction has been received (YES is determined), the process proceeds to step S120.

In step S120, the trade restriction product determination unit 236 cancels the trade restriction.
In this way, when step S120 is completed, or when the product is not a product other than a halal product or is not an allergic product (when it is determined to be NO) in step S117, the process proceeds to step S121.

In step S121, the trade-restricted product determination unit 236 determines whether the product identified by the product identification unit 235 is a product whose expiration date has expired.
When the trade-restricted product determination unit 236 determines that the product includes an expired product, the determination in step S121 is YES, and the process proceeds to step S122.

In step S122, the display control unit 238 causes the display unit D of the cashier terminal 2 to display that there is a possibility that the expired product is included.
If it is determined in step S123 that the instruction to cancel the trading restriction has not been received, NO is determined and the process proceeds to step S115.

If it is determined in step S123 that the instruction to cancel the trading restriction has been accepted, YES is determined and the process proceeds to step S124.
In step 124, the trade restriction product determination unit 236 cancels the trade restriction.
When the trade-restricted product determination unit 236 determines that the product whose expiration date has expired is not included, the determination is NO in step S121, and the process ends.

As a result, the trading restriction process ends. Then, the process proceeds to the settlement at the cashier terminal in step S106.
In this way, the information processing system can recognize the product placed on the cashier terminal and automatically make a payment. Further, at this time, regarding the objects and products to be subjected to the eye inspection (for example, objects for which the product cannot be specified at the cashier terminal 2, products corresponding to the sale-restricted products, etc.), Inspection is performed, and automatic settlement is performed according to the result of the visual inspection (the result of identifying the product, the result of determining the trade-restricted product, etc.).
Therefore, according to the present information processing system, when the shopper purchases a product, it is possible to automate the payment of the price of the product and to improve the accuracy of specifying the product.

The first embodiment is not limited to the above-described embodiments, and modifications, improvements, etc. within a range in which the object of the present invention can be achieved are included in the present invention.
For example, the appearance configuration of the cashier terminal 2 shown in FIG. 5 is an example, and the appearance is not limited to this appearance. The cashier terminal 2 only needs to include at least the predetermined area A, an image capturing unit such as the cashier camera 211, and the output unit 206, and other components may be added.
Further, the product recognition system according to the first embodiment may include a transport mechanism (for example, a belt conveyor) that transports one or more objects from the upstream side to the downstream side. A predetermined area having an imaging unit is arranged on the upstream side. A settlement area is arranged on the downstream side.
In the predetermined area, the number of objects imaged by the imaging unit is counted. This product recognition system detects an error when the number of counted objects is different from the number of products settled in the settlement area.

[Embodiment 2]
The information processing system according to the second embodiment is a product recognition system including a cashier terminal 2 as shown in FIGS. 5 and 15 in a store 20 such as a bookstore as shown in FIG. In the information processing system according to the second embodiment, a book, which is a product, is placed on a cash register to enable automatic settlement.
FIG. 14 is a diagram illustrating a layout example of a bookstore that employs the information processing system according to the second embodiment.

A store 20 such as a bookstore has a doorway 22 having a gate 21. Depending on the scale of the store 20, the entrances / outlets 22 are not limited to the two locations as illustrated, and may be one location or three or more locations. The gate 21 is not limited to the type having the opening / closing member as shown in the figure, but also the type having a function of notifying the occurrence of an abnormal situation by sound or light like a speaker or a lamp, which is not shown, is dealt with in case of misconduct. It is made possible.
The store 20 has a plurality of shelf racks 23 for displaying books. The shelf rack 23 displays books by arranging a plurality of shelves in the vertical direction at intervals. A passage 24 is formed between the rack racks 23 facing each other in the horizontal direction.

A plurality of ceiling cameras 310 (only one is shown in the drawing) are installed on the ceiling of the passage 24. The ceiling camera 310 always captures an image of a shopper who has entered the store without blind spots in the inside of the store.
In addition, a plurality of (only one in the drawing) shelf cameras 311 that constantly image the inside of each shelf rack 23 may be installed in each shelf rack 23. A plurality of shelf cameras 311 are arranged so that the inside of the shelf can be imaged without blind spots, and even a shopper standing in front of the shelf rack 43 can also image.

A cashier counter 25 is installed near the doorway 22 in the store 20. On the cashier counter 25, a plurality of unmanned cashier terminals 2 for automatic payment are installed. A manned register 26 is installed next to the cashier terminal 2. A shopping cart (not shown) for inserting books may be placed near the doorway 22 or in the passage 24.
A clerk Mt is working in the passage 24, the cashier counter 25, or the like. The store clerk Mt owns the information terminal 9. The information terminal 9 is also installed in the backyard of the store 20 or the general headquarters outside the store 20.

In the information processing system, the ceiling camera 310 captures the action of the shopper taking out or returning one or more books from the shelf of the shelf rack 23, and after grasping the number of books taken by the shopper. As shown in FIG. 15, information such as the price of a book placed on the unmanned cashier terminal 2 is acquired and automatically settled.
In the second embodiment, the shopper will be described as a “moving object Mo”.
FIG. 15 is a schematic perspective view showing the general configuration of the cashier terminal 2 used in the second embodiment, showing a state in which a book (without numbering) is being paid.

The unmanned cashier terminal 2 according to the second embodiment employs the same external configuration as the cashier terminal 2 according to the first embodiment shown in FIG.
Therefore, the cashier terminal 2 includes the surrounding portion 270 that surrounds the predetermined area A in which the book is placed. The surrounding portion 270 includes a top plate portion 271, a bottom plate portion 272, and a pair of side plate portions 273. The surrounding portion 270 has the same structure as the surrounding portion 270 of the first embodiment shown in FIG.

Therefore, the cash register camera 211 that images the predetermined area A is fixed to the top plate portion 271 and the pair of side plate portions 273. As shown in FIG. 15, at least one cash register camera 211 images at least the spine of a book placed in the predetermined area A. Note that, in FIG. 15, the book is placed so that the spine cover faces one side plate portion 273 and the spine cover faces the other side plate portion 273, but the spine cover is attached to the cash register camera 211 installed in the top plate portion 271. May be directed, and there are no restrictions on how to place books.

Such a cashier terminal 2 is incorporated in a product recognition system as one embodiment of the information processing system.
FIG. 16 is a configuration diagram showing the configuration of the product recognition system as the second embodiment of the information processing system of the present invention.
The product recognition system includes a server 1, cash register terminals 2-1 to 2-n, a sales floor device 3, and an inspection terminal Q. The sales floor device 3 has a function of recognizing the number of books from a captured image of the book captured by the ceiling camera 310.
The server 1 is installed in the backyard of the store 20 or outside the store in order to manage the cashier terminals 2-1 to 2-n, the sales floor device 3, and the inspection terminal Q. Further, the sales floor device 3 controls the ceiling camera 310 installed in the store 20 in order to discover and track the moving object Mo in the store 20 shown in FIG.
The server 1, the cashier terminals 2-1 to 2-n, the sales floor device 3, and the inspection terminal Q are connected to each other via a network N such as an Internet line. The inspection terminal Q can be possessed by a store clerk distant from the cashier terminal 2, provided in a backyard of the store, or provided in a call center remote from the store.

Note that, for convenience of explanation, only one server 1 in FIG. 16 is drawn, but in reality, there may be a plurality of servers.
Further, hereinafter, when it is not necessary to individually distinguish the cashier terminals 2-1 to 2-n, they are collectively referred to as a “cash register terminal 2”.

The server 1 executes each process in order to manage each operation of the cashier terminal 2 and the sales floor device 3.
The server 1 includes a CPU 101, a ROM 102, a RAM 103, a bus 104, an input / output interface 105, an output unit 106, an input unit 107, a storage unit 108, a communication unit 109, and a drive 110. There is. These are configured similarly to the server 1 described in the first embodiment shown in FIG.

The sales floor device 3 is connected to the server 1 via the network N.
FIG. 17 is a block diagram showing the hardware configuration of the sales floor device 3 in the product recognition system of FIG.

The sales floor device 3 includes a CPU 301, a ROM 302, a RAM 303, a bus 304, an input / output interface 305, a ceiling camera 310, a shelf camera 311, a communication unit 315, and an information terminal 9.

The CPU 301, the ROM 302, the RAM 303, the bus 304, the input / output interface 305, and the communication unit 315 of the sales floor device are configured similarly to those of the server 1 shown in FIG. 7.

The ceiling camera 310 is connected to the network by a USB (Universal Serial Bus) cable.
The shelf camera 311 may employ a camera capable of capturing a wide angle such as a fisheye camera.
Further, the shelf camera 311 is connected to the network by a USB cable.

The information terminal 9 is an information device such as a smartphone or a tablet including a remote operation unit 390, a display unit 391, and the like. The remote control unit 390 has a function of remotely correcting an error state such as a system processing abnormality. The display unit has a screen for displaying an error state, a moving object Mo, and the like. The information terminal 9 also includes a voice generation unit (not shown) that notifies the error state.
The error state in the store includes, for example, the case where the ceiling camera 310 cannot recognize the number of books taken from the shelf, the case where an unaccounted book is about to be taken out of the store, and the like. In addition, as an error state in the cashier terminal, when the book cannot be identified or the number of books cannot be recognized in the cashier terminal 2, when the age-restricted product is about to be settled, the book is left in the cashier terminal 2. When it is.
The server 1 also includes an error display unit 151 that displays such an error and an error canceling unit 152 that cancels the error state.
In the present embodiment, when the cashier terminal 2 cannot identify the book or cannot recognize the number of books, or when the age-restricted product is about to be settled, the eye examination terminal Q requests the eye examination. By doing so, it is possible to eliminate the error state.

The cashier terminal 2 is connected to the sales floor device 3 through the network N.
The cashier terminal 2 is configured similarly to the cashier terminal of the first embodiment shown in FIG.
Therefore, the cashier terminal 2 according to the second embodiment includes the CPU 201, the ROM 202, the RAM 203, the bus 204, the input / output interface 205, the output unit 206, the input unit 207, the illumination unit 221, and the light shielding unit 209. The presentation unit 210, a cash register camera 211, a storage unit 208, a communication unit 213, and a drive 214 are provided.

The cash register camera 211 images a book placed in a predetermined area A, and outputs a captured image obtained as a result to the image acquisition unit 232 in the CPU 201 as an object captured image. If the cashier camera 211 can image in a wide angle like a fish-eye camera, or if it specializes in imaging only the spine of a book, only one cashier camera 211 may be provided.

Since the hardware configuration of the inspection terminal Q is the same as the hardware configuration of the server 1, for the hardware configuration of the inspection terminal Q, refer to FIG. The description is omitted.

FIG. 18 is a functional block diagram showing an example of the functional configuration of the server 1, the cashier terminal 2, the sales floor device 3, and the inspection terminal Q.
The CPU 101 of the server 1 includes an error determination unit 150.
A product DB 131 and a position information management DB 132 are provided in one area of the storage unit 108 of the server 1.
The product DB 131 is a DB (Data Base) that stores information such as book titles, prices, author names, publishers, and the like. The position information management DB 132 manages the position of the moving object Mo.

In the CPU 201 of the cashier terminal 2, as shown in FIG. 18, a light emission control unit 228, a light shielding control unit 229, a presentation control unit 230, an image acquisition unit 232, an object recognition unit 233, and a product identification unit 235. The trading restricted product determination unit 236, the settlement unit 237, the display control unit 238, and the visual inspection result acquisition unit 239 function.
In the CPU 101 of the eye examination terminal Q, the image display control unit 411 and the eye examination result transmission unit 412 function.

The light emission control unit 228 responds to control for switching between a state in which the illumination unit 221 emits light at the timing of capturing an image of a book and a state of not emitting light at the timing for not capturing an image of the book, a situation of recognizing a book placed in the predetermined area A, and the like. Control to switch the emission color.
The light-shielding control unit 229 controls the light-shielding unit 209 provided in the surrounding unit 270 to switch between an opaque state at the timing of photographing a book placed in the predetermined area A and a transparent state at the timing of not photographing the book.
The presentation control unit 230 executes control so that the presentation unit 210 changes the emission color for presenting the state of the cashier terminal 2.

The image acquisition unit 232 acquires the data of the object image captured by the cashier camera 211 with the object placed in the predetermined area A.
The object recognition unit 233 recognizes the presence of an object placed in the predetermined area A using the above-described predetermined image recognition method.

The product identification unit 235 lists up product candidates for the object whose existence is recognized by the object recognition unit 233, by an image processing method such as specific object recognition, general object recognition, character recognition, or deep learning. The listed product candidates are called “product candidate list S”. After that, the product specifying unit 235 causes the verification function to be performed and specifies the product with high accuracy. The “image recognition method” described below means a method of recognizing what is made using some image. For example, specific object recognition, general object recognition, character recognition, and deep learning are examples of image recognition methods.
The verification function lists up the "commodity candidate list P" by an algorithm different from the method of listing up the product candidates described above. The results of the product candidate lists S and P are matched with each other, and a product is specified when the result exceeds a predetermined threshold.
As a method of listing the “commodity candidate list”, for example, it is realized by a method of matching the image information of the object obtained from the object whose existence is recognized with the image information held in the DB information holding unit 241 or the memory. May be. That is, when the feature information of both images matches (exceeds a threshold value), the object whose existence is recognized by the object recognition unit 233 is a product registered in the DB information holding unit 241, and thus the product specification unit 235 , And specifies that the product is registered in the DB information holding unit 241.
Also in the present embodiment, as in the first embodiment, when the product cannot be specified at the cashier terminal 2, the result is output as the product not specified.
Then, as in the first embodiment, when the specific object visual inspection mode is set, the product identification unit 235 outputs the object captured image of the object not specified as the product and the related information to the visual inspection result acquisition unit. It transmits to the terminal Q for eye examinations via 239. On the other hand, when the all-object visual inspection mode is set, the product identification unit 235 determines the product identification result and the product captured image of the identified product, the object captured image of the object not identified as the product, and related information. And the like are transmitted to the inspection terminal Q via the inspection result acquisition unit 239.
Further, in the present embodiment, even when the number of products (books) placed on the cashier terminal 2 and the number of books counted on the sales floor device 3 do not match, the result is the number of products (number of books). It is output as a mismatch. In this case, the product specifying unit 235 transmits the object captured image and the number of books counted in the sales floor device 3 to the inspection terminal Q via the inspection result acquisition unit 239.
Furthermore, the product identification unit 235 identifies the product based on the eye inspection result acquired by the eye inspection result acquisition unit 239 from the eye inspection terminal Q. Specifically, for the product identified by the product identification unit 235, when the identification result is approved or corrected by the inspection result, the product identification unit 235 identifies the product according to the content indicated by the inspection result. . In addition, for an object for which the product specifying unit 235 has determined that the product is unspecified, the product specifying unit 235 specifies the object as a product indicated by the visual inspection result. If the number of books is not confirmed by visual inspection (disagreement of the number of books is not resolved), an error message is displayed and settlement is stopped.

The trade-restricted product determination unit 236 determines whether or not the product identified by the product identification unit 235 is a trade-restricted product, based on the determination information, and presents it. The trade-restricted product is, for example, a book whose sale is restricted to shoppers under a certain age. When a clerk sells a book, the clerk who sees the shopper can confirm the age of the shopper and determine whether or not to sell the book.
However, this system, which employs automatic settlement, rather than face-to-face sales, requires a mechanism that allows shop assistants to confirm the age of shoppers.
In the present embodiment, the trade-restricted product determination unit 236 of the cashier terminal 2 that has identified the trade-restricted product receives the product image of the product corresponding to the trade-restricted product from the eye-checking terminal Q via the eye-check result acquisition unit 239. Send to. At this time, the trade-restricted product determination unit 236 appropriately transmits the image (face image, identification image, etc.) regarding the shopper to the inspection terminal Q via the inspection result acquisition unit 239. At this time, when the image relating to the shopper is unclear, the purchaser may be informed by screen display or voice guidance at the cashier terminal 2 and the image retake may be requested. Further, the purchaser and the eye examiner may make a call via the microphone M and the speaker S provided in the cashier terminal 2.
Then, the trade-restricted product determination unit 236 determines whether or not the sale of the trade-restricted product is permitted based on the eye inspection result acquired by the eye-inspection result acquisition unit 239 from the inspection terminal Q (the sale-restriction is released). Whether or not) is determined.
If the sale of the trade-restricted product is permitted according to the result of the visual inspection, the payment of the product is continued. On the other hand, if the sale of the trade-restricted product is not permitted according to the result of the visual inspection, the fact that the product is a trade-restricted product is presented and the settlement process is interrupted. In this case, the salesclerk who received the error state may confirm the age of the shopper on the error display unit 151, operate the cashier terminal 2, and release the restricted state. Thereby, the settlement process is restarted. In addition, the cashier terminal 2 includes a cashier camera 211 that captures the face, hands, and the like of the shopper, and estimates the age of the shopper, thereby restricting the trade to a shopper who is determined not to have reached a predetermined age. May not be sold.

Therefore, the trade-restricted product determination unit 236 identifies a trade-restricted product such as an age-restricted book from the information of the DB management unit 141 of the server 1.
The trade-restricted product determination unit 236 may restrict the trade by associating with the shopper information obtained by the individual authentication. When it is determined that the book is a trade-restricted product, the display unit D indicates that the book is a trade-restricted product.

The settlement unit 237 calculates the total price of the books specified by the product specifying unit 235 and which can be sold by the trade restriction product judging unit 236. For example, the settlement unit 237 reads out the prices of the books placed in the predetermined area A from the DB information holding unit 241, adds the prices, displays the prices on the display unit D (FIG. 8), and performs the settlement.

The display control unit 238 executes control to display the title and price of the book imaged by the cashier camera 211 of the cashier terminal 2 and settled by the settlement unit 237 to the purchaser and the clerk. The display control unit 238, when the product is not specified by the product specifying unit 235, or when the product is not specified by the visual inspection in the inspection terminal Q, alerts the shopper or the clerk (object). The output unit 206 is controlled so as to output (replacement instruction, etc.). Specifically, the purchaser may be notified by a screen display or voice guidance, and the purchaser may be requested to replace the product. Further, the purchaser and the eye examiner may make a call via the microphone M and the speaker S provided in the cashier terminal 2.

The eye inspection result acquisition unit 239 requests the identification of the product by the eye inspection by transmitting the object captured image or the product identification result and the product captured image of the identified product to the eye inspection terminal Q, and The eye inspection result transmitted from the eye inspection terminal Q is acquired in response to the request. The visual inspection result acquisition unit 239 outputs the acquired visual inspection result (commercial product identification result) to the commercial product identification unit 235. Further, the eye inspection result acquisition unit 239 requests the determination of the sale restricted product by the eye inspection by transmitting the product imaged image of the product corresponding to the sale restricted product to the eye inspection terminal Q, and the eye inspection terminal Q. Get the result of the eye inspection sent from. At this time, the visual inspection result acquisition unit 239 appropriately transmits an image of the shopper (face image, image of identification card, etc.) to the visual inspection terminal Q. The eye inspection result acquisition unit 239 outputs the acquired eye inspection result (determination result of the sale-restricted product) to the sale-restricted product determination unit 236.

In the eye inspection terminal Q, the image display control unit 411 displays images such as the object imaged image or the product imaged image transmitted from the cashier terminal 2 and these images when the cashier terminal 2 requests the eye inspection. Various information (commodity candidate list, etc.) transmitted in association with is output to the output unit 106.
The eye inspection result transmission unit 412 receives the object inspection image or the product captured image output to the output unit 106 from the eye inspection result input by the eye examiner through the input unit 107 (determination of product identification result or trade restricted product). (Result etc.) is transmitted to the cashier terminal 2 which requested the eye examination.

In the CPU 301 of the sales floor device 3, as shown in FIG. 18, a personal authentication unit 320, a moving object tracking unit 330, a position information management unit 340, and a volume number counting unit 350 function.

The personal authentication unit 320 includes a personal information acquisition unit 321.
The personal authentication unit 320 personally authenticates who the shopper registered in the DB management unit 141 of the server 1 is based on the personal information of the shopper acquired by the personal information acquisition unit 321. The personal authentication unit 320 and the personal information acquisition unit 321 may be provided in the cashier terminal 2 as in the first embodiment.

The personal information here includes, for example, personal information such as name, sex, date of birth, address, and telephone number, as well as biometric information such as fingerprints, veins, and iris, and financial information such as credit card numbers and bank account numbers. Information about privacy, such as information about
The personal information acquisition unit 321 is provided in, for example, the gate 21 installed at the doorway 22. As the personal information acquisition unit 321, a reading device that touches a portable information terminal such as an IC card of a shopper, a smartphone, or a tablet, a reading device that reads biometric information such as a fingerprint, a vein, or an iris is used.

When the personal authentication cannot be performed when entering the store, the personal authentication may be performed from the image of the shopper captured by the ceiling camera 310 during shopping.
The acquired personal information is used for trading restrictions (including cancellation) and purchase analysis.

FIG. 19 is a functional block diagram showing a detailed functional configuration example of the moving object tracking unit 330 provided in the sales floor device 3 of FIG.
As shown in FIG. 19, the moving object tracking unit 330 discovers the moving object Mo from the image captured by the ceiling camera 310 and tracks the moving moving object Mo. The ceiling camera includes a moving object area definition unit 3304 and a ceiling camera moving object area tracking unit 3305.

Moreover, the moving object tracking unit 330 using the ceiling camera is connected to the ceiling camera 310 via a USB cable, the network N, or the like. Therefore, the ceiling camera 310 is linked with another ceiling camera 310, a personal computer, or the like.

The moving object finding unit 3302 with the ceiling camera estimates the state of the moving object Mo using a state space model (Bayesian filter or the like) based on the captured image taken by the ceiling camera 310, and finds the moving object Mo, A uniquely identifiable ID is assigned.

By the way, in the image from the ceiling camera 310, since the peripheral area of the moving object Mo that is not directly facing the ceiling camera 310 is imaged at an angle (from an oblique direction), the position information of the moving object Mo may not be accurately acquired. is there. Therefore, it is conceivable that the captured image is corrected so that the image is directly faced. However, even if such a correction is applied, it may not be possible to accurately acquire the position information.
Therefore, the ceiling camera 310 may acquire highly accurate position information by acquiring height information of the moving object Mo using a distance sensor or the like.

The moving object area definition unit 3304 using the ceiling camera updates the position information of the area of the moving object Mo after moving. Since the moving object Mo continues to move, the moving area changes within the range captured by one ceiling camera 310, and also moves within the range captured by another ceiling camera 310. Each time the moving object Mo moves, the moving object area is defined, and the position information of each moving object area such as the position information management DB 132 that manages the position information and the memory is updated.

The moving object area tracking unit 3305 estimates the position of the moving object area and continues to track the moving object area of the moving object Mo.

FIG. 20 is a functional block diagram showing a detailed functional configuration example of the position information management unit 340 provided in the sales floor device 3 in FIG.
The position information management unit 340 includes an inter-camera information transfer unit 341, a position definition unit 342 of each camera, and a moving object display unit 343.
The inter-camera information transfer unit 341 shares the image information captured by each ceiling camera 310 with the image information captured by another ceiling camera 310, so that the captured image of the ceiling camera 310 with the moving object Mo. Even when the image is captured by another ceiling camera 310, the moving object area can be continuously tracked.

The camera-to-camera information transfer unit 341 exchanges information between the ceiling cameras 310 on the storage unit 108, including the product DB 131, through the server 1 that controls the information obtained by imaging the ceiling cameras 310.
As another example, in view of the large number of ceiling cameras 310, the inter-camera information passing unit 341 receives the images captured by the ceiling cameras 310 without passing through the server 1 between the ceiling cameras 310 by, for example, P2P. You may hand it over.

The position definition unit 342 of each camera defines the position information of where in the store each ceiling camera 310 is reflected. That is, the position definition unit 342 of each camera grasps where in the store the moving object imaged by the different ceiling cameras 310 by the inter-camera information transfer unit 341 is located.
The position definition unit 342 of each camera combines the captured images of the ceiling cameras 310 to create one store map. Further, the position definition unit 342 of each camera replaces the coordinates of each ceiling camera 310 and the shelf camera 311 with the coordinates on the store map. Further, the position definition unit 342 of each camera corrects the captured image captured by each ceiling camera 310 by perspective transformation so that the captured image directly faces the floor surface in the store.

The ceiling camera 310 is equipped with a distance sensor so that the height information is acquired, so that the positional information management unit 340 accurately corrects the distorted captured image and accurately recognizes the moving object Mo. can do.
The moving object display unit 343 displays the position information imaged by the position definition unit 342 of each camera for the moving object Mo in the store 30. The moving object display unit 343 may be adopted as the information terminal 9 held by the clerk Mt, the backyard screen of the store, or the like.

FIG. 21 is a functional block diagram showing a detailed functional configuration example of the book number counting unit 350 provided in the sales floor device 3 of FIG.
The number-of-volumes counting unit 350 includes a number-of-volumes recognizing unit 351, a moving object-number-of-volumes associating unit 352, a number-of-volumes unidentifying unit 353, a number-of-volumes managing unit 354 associated with a person, and a moving object area transfer recognizing unit using a ceiling camera. 355 and the number-of-books recognition unit 356 transferred by the ceiling camera.

The number-of-books recognition unit 351 recognizes the number of books taken by the moving object Mo from the shelf and the number of books returned to the shelf from the captured images captured by the ceiling camera 310. The number-of-books recognizing unit 351 detects the entry / exit of the moving object region into / from the shelf by providing the “object entry detection line” or the like in the captured image, and the moving object Mo indicates the shelf in the captured image at the time of detection. Define the area of the object taken from within and the object returned to the shelf. The number of the object areas is recognized as the number of books. When the ceiling camera 310 has a zoom function, even if the number-of-books recognition unit 351 recognizes the number of books after the ceiling camera zooms in consideration of the distance between the ceiling camera 310 and the moving object Mo. Good.

Further, the number-of-books recognizing unit 351 may recognize the number of books taken by the moving object Mo from the shelf and returned to the shelf by the shelf camera 311, and from the combination of the ceiling camera 310 and the shelf camera 311, the shopper can recognize the number of books. The number of books taken from the shelf and the number of books returned to the shelf may be recognized. The shelf camera 311 at that time may be a camera capable of capturing an image in a wide range.

The moving object and the number-of-books association unit 352 associates the number of books recognized by the number-of-books recognition unit 351 with the person who took the book.
When the number of books cannot be recognized, the unspecified number of books determining unit 353 associates the fact that the number of books cannot be recognized with the moving object Mo.

The number-of-volumes management unit 354 associated with a person continuously manages the number-of-volumes count list associated with the ID of the moving object Mo by utilizing the position information management DB 132 and the like. When a book is taken from the shelf, the number of books acquired is added. On the contrary, when the books are returned to the shelf, the number of the returned books is subtracted.

When a book is transferred between the moving object areas, the moving object area transfer recognizing unit 355 by the ceiling camera transfers the book number information associated with each moving object Mo to the transferred moving object Mo.
It should be noted that the moving object area transfer recognizing unit 355 by the ceiling camera may analyze the movement of a person by using an object recognition method such as deep learning and may recognize the transfer, and the moving object area transfer may be performed during the transfer. The hand may be recognized, or the overlap between the moving object regions (which may include the hand) may be recognized. The moving object area transfer recognition unit 355 using the ceiling camera may use the shelf camera 311 instead of the ceiling camera 310. The shelf camera 311 at that time may be a camera capable of capturing an image over a wide range.

The number-of-books recognition unit 356 delivered by the ceiling camera recognizes the number of books when the books are delivered between the moving object regions. For example, the number-of-books recognition unit 356 delivered by the ceiling camera recognizes the number of books from the captured image at the time when the delivery is recognized.
The number-of-books recognition unit 356 delivered by the ceiling camera may be equipped with a ceiling camera having a zoom function to zoom up the portion where the delivery is supposed to be performed and recognize the number of books.
The number-of-books recognition unit 356 delivered by the ceiling camera may recognize the number of books by using the shelf camera 311 capable of capturing a wide range instead of the ceiling camera 310.
Further, the number-of-books recognition unit 356 delivered by the ceiling camera associates each moving object Mo identified by the delivery-recognition unit between moving object regions 355 by the ceiling camera with the number of books recognized here, Update the volume list.

Next, a book settlement method in the product recognition system according to the second embodiment will be described mainly with reference to FIG. 22 and also with reference to FIG.
FIG. 22 is a flowchart illustrating book settlement processing in the second embodiment.

In step S201, when the shopper (moving object Mo) enters the store through the entrance / exit 22 of the store (FIG. 14), the ceiling camera 310 installed near the entrance / exit 22 starts imaging the shopper.
When the shopper advances along the aisle 24, the ceiling camera 310 on the back side images the shopper. In this way, the plurality of ceiling cameras 310 constantly image the shopper. The gate 21 may be provided near the entrance 22.
When the doorway 22 is provided with the gate 21, the gate 21 is always closed, but the gate 21 is opened at the timing when the shopper enters the store and is closed after entering the store. Note that the personal authentication unit 320 may perform personal authentication of the shopper and acquire the personal information of the shopper before step S201.

In step S202, the moving object detection unit 3302 using the ceiling camera defines as the moving object Mo by extracting only the area (moving object area) of the shopper imaged in step S201, and assigns the ID to the moving object Mo. A number is issued and the position information in the store associated with the ID is registered in the position information management DB 132 or the RAM 303 of the sales floor device 3.

In step S203, when the moving object area definition unit 3304 by the ceiling camera moves within the range imaged by the ceiling camera 310, the position of the area of the moving object Mo after the movement is newly defined. The position information is managed by the position information management DB 132 that manages the position of a person, a memory, and the like, and is updated for each area definition. This defined position is also recognized as a position imaged by another ceiling camera 310.

In step S204, the moving object area tracking unit 3305 using the ceiling camera estimates the position of the moving object area and keeps track of the moving object Mo based on the fact that the shopper moves through the passage 24 toward the book to be purchased. In step S201, the moving object Mo is constantly imaged, in step S202, the ID is assigned to the moving object Mo, in step S203, the positional information of the area of the moving object Mo after the movement is updated, and the inter-camera information passing unit is further updated. Since position information is exchanged by 341, even if the moving object Mo moves in the passage 24 and is imaged by a different ceiling camera 310, the moving object area tracking unit 3305 by the ceiling camera continues to track the moving object Mo. You can

In step S205, the number-of-books recognizing unit 351 recognizes the number of books taken from the shelf and returned to the shelf by the moving object Mo having the ID.
In step S205, when the number of books taken cannot be counted, an unillustrated number-of-books determination unit outputs an error.

In step S206, the moving object and the book number association unit 352 associates the number of books recognized (counted) in step S205 with the moving object Mo. Therefore, the number-of-books recognition unit 351 recognizes how many books the uniquely identified shopper has taken.

In step S207, the number-of-books management unit 363 associated with the person continues to manage the number of books associated with the person by utilizing the DB management unit 141 of the server 1. Therefore, even when the book picked up from the shelf by the shopper is returned to the shelf, the book quantity management unit 363 associated with the person recognizes the number of books owned by the shopper.

When the shopper finishes picking up the book, it goes to the cashier terminal 2 and puts the book in a predetermined area A of the cashier terminal 2. As shown in FIG. 15, the book is placed so that the spine cover is captured by at least one or more cameras 211 of the cashier terminal 2. Then, the shopper presses a button, which is the input unit 207 provided in the cashier terminal 2. The cashier terminal 2 uses the pressing of this button or the like as a trigger to specify the product. The product identification is performed by making full use of the image recognition method and identifying the title of the book from the cover or spine of the book.

In step S210, the product specification unit 235 of the cashier terminal 2 determines whether the number-of-volumes information associated with the moving object Mo recognized in step S207 matches the number-of-volumes information of the books placed in the predetermined area A of the cashier terminal 2. Verify whether. The specific flow of this verification will be described later with reference to FIG.
If the number-of-volumes information associated with the moving object Mo does not match the number-of-volumes information of the books placed in the predetermined area A of the cashier terminal 2, a visual inspection (confirmation of the number of volumes) at the visual inspection terminal Q is requested. If the number of books is not confirmed by visual inspection (disagreement of the number of books is not resolved), an error is displayed on the error display unit 151 and settlement is not possible. If an attempt is made to leave the store in that state, the error display unit 151 installed near the entrance 22 is warned by sound, light, or the like. If the gate 21 is installed, it remains closed.
In step S210, the trade-restricted product determination unit 236 also determines whether or not the book placed on the cashier terminal corresponds to the presence or absence of the restriction. By obtaining the personal information in advance at any timing up to step S207 (including before step S201), the trade-restricted product determination unit 236 allows the shopper who cannot purchase the restricted book to purchase the book. The error display unit 151 displays an error.

In step S211, the settlement unit 237 of the cashier terminal 2 settles the total amount of money of the books placed in the predetermined area A. That is, as described in the first embodiment, the product specification unit 235 obtains information such as the price of a book and calculates the total price. The shopper pays with a credit card, electronic money, or the like to complete the settlement.
When the settlement is completed, the DB management unit 141 updates the number-of-volumes count list associated with the person to the state of “settlement completed”.

Further, when the shopper is trying to bring out an unpaid book near the doorway 22, that is, when the book count count list associated with the moving object Mo is not “paid”, the take-out detection unit Detects it, and the error display section 151 near the entrance 22 and the presentation section 210 of the cashier terminal 2 present it with sound, light, or the like. Further, when the gate 21 is installed, the gate 21 remains closed and the shopper cannot leave the store. In addition to maintaining the gate 21 closed, the error display unit 151 or the like may give a warning by sound, light, or the like. Thereby, shoplifting can be prevented.
The error display unit 151 and the presentation unit 210 of the cashier terminal 2 may be made to emit light in different colors depending on the error state.

FIG. 23 is a flowchart for explaining the verification of the book volume information and the booked volume number in step S210 of FIG.
In step S221, when the reading start button of the input unit 207 of the cashier terminal 2 is pressed, the image acquisition unit 232 of the cashier terminal 2 acquires the object imaged image placed in the predetermined area A of the cashier terminal 2 and then exists. With respect to the object for which is recognized, the product specification unit 235 specifies which product, and also recognizes the number of products.
In step S222, the person of the volume number counting unit 350 of the sales floor device 3 and the volume number associating unit 362 make an inquiry from the ID of the moving object Mo about the volume number of the books placed on the cashier terminal 2.

In step S223, the product identification unit 235 determines whether the number of books placed on the cashier terminal 2 and the number of books counted by the number management unit associated with a person match.
If the number of books placed on the cashier terminal 2 and the number of books counted by the book number management unit associated with the person match (YES), the process proceeds to step S226.
If the number of books placed on the cashier terminal 2 does not match the number of books counted by the book number management unit associated with the person (NO), the process proceeds to step S224.
In step S224, the product specifying unit 235 transmits the object captured image and the number of books counted in the sales floor device 3 to the eye inspection terminal Q via the eye inspection result acquisition unit 239, and thus, the eye inspection terminal Q. Request a visual inspection (confirm the number of books) in and obtain the visual inspection result.
In step S225, the product identification unit 235 determines whether or not a visual inspection result indicating that the discrepancy in the number of books has been resolved is obtained.
When it is determined in step S225 that the eye inspection result indicating that the discrepancy in the number of books has been resolved is not acquired, the error display unit 151 of the cashier terminal 2 displays an error and issues a warning.
At this time, the purchaser and the eye examiner may talk with each other via the microphone M and the speaker S provided in the cashier terminal 2.
If an attempt is made to leave the store while the warning is given, the error display section 151 near the entrance 22 gives a warning by sound or light. When the gate 21 is installed, the gate 21 remains closed and the shopper cannot exit. Thereby, shoplifting can be prevented.
If it is determined in step S225 that the visual inspection result indicating that the discrepancy in the number of books has been resolved is acquired, the process proceeds to step S226.
In step S226, the product identification unit 235 determines whether the specific object inspection mode or the all object inspection mode is set.
If the specific object visual inspection mode is set, the process proceeds to step S227.
If the all-object visual inspection mode is set, the process proceeds to step S228.
In step S227, the eye inspection result acquisition unit 239 requests eye inspection (specification of the product) at the eye inspection terminal Q for the target book, and acquires the eye inspection result.
After step S227, the process proceeds to step S229.
In step S228, the product identification unit 235 determines whether or not it is not possible to identify which of the books placed in the predetermined area A of the cashier terminal 2. If there is a book that cannot be identified by product identification unit 235 (YES in step S228), the process proceeds to step S227.

If there is no book that the product specification unit 235 cannot identify in step S228 (NO in step S228), the product specification unit 235 stores the books held in the DB information holding unit 241 or the storage unit of the server 1. A book is specified by including information such as name, price, and restricted product. As a result, the process proceeds to step S229. Information on the specified book may be output to the display control unit 238.

In step S229, the trade-restricted product determination unit 236 determines whether the book identified by the product identification unit 235 is a book that needs age confirmation.

If it is determined in step S229 that the book identified by the product identification unit 235 is a book that requires age confirmation, that is, if YES, the process proceeds to step S230.
If it is determined in step S229 that the book identified by the product identification unit 235 is not a book that requires age confirmation, that is, if NO, the process returns to the book automatic settlement process.
In step S230, the display control unit 238 causes the display unit D of the cashier terminal 2 to display a screen for age confirmation. However, if the personal information of the shopper is acquired and it is not necessary to confirm the age here, step S230 is skipped and the process proceeds to step S234.
At this time, if there is a problem with the purchaser's age authentication, such as when the purchaser's face or an image of the age confirmation document is unclear, the purchaser is notified by a screen display or voice guidance at the cashier terminal 2. Alternatively, the purchaser may be requested to retake the image. Further, the purchaser and the eye examiner may make a call via the microphone M and the speaker S provided in the cashier terminal 2.
In step S231, the eye inspection result acquisition unit 239 requests the eye inspection terminal Q to perform eye inspection (determination of trade-restricted products) for the target book, and acquires the eye inspection result.
In step S232, the trade-restricted product determination unit 236 determines whether or not an instruction to cancel the trade restriction has been received.

If it is determined in step S232 that the instruction to cancel the trading restriction has not been received, the process proceeds to step S233.
If it is determined in step S232 that the cancel instruction for canceling the trading restriction has been received, that is, if YES, the process proceeds to step S234.
In step S233, the eye inspection result transmission unit 412 of the eye inspection terminal Q transmits a warning that the trading restriction has not been released even by the eye inspection result. By receiving this warning, for example, the display control unit 238 of the cashier terminal 2 presents, via the output unit 206, a warning that the trading restriction has not been canceled even by the result of the visual inspection. After step S234, the process ends and the settlement is stopped.
In step S234, the trade restriction product determination unit 236 cancels the trade restriction.
In this way, when step S234 is ended or it is determined in step S229 that the product is not age-restricted (NO is determined), the process returns to the book automatic settlement process.
In this way, the information processing system recognizes the number of books taken by the shopper in the store and determines whether the number matches with the number of books placed on the cashier terminal. In addition, at this time, in the case of being subject to the visual inspection (for example, when the number of volumes does not match, when the product cannot be specified at the cashier terminal 2 or when the book corresponds to the sale-restricted product), the visual inspection is performed. A visual inspection is performed at the terminal Q, and automatic settlement is performed according to a visual inspection result (a result of checking the number of books, a result of identifying a book, a result of determining a trade-restricted product, etc.).
Therefore, according to the present information processing system, when the shopper purchases a product such as a book, it is possible to automate the settlement of the price of the product and improve the product identification accuracy.

In the second embodiment, as an error expected in tracking the moving object Mo (shopper), (A) when the shopper enters from the entrance 22 of the store 20, the moving object finding unit 3302 using the ceiling camera moves. When the object cannot be detected, (B) when the moving object area tracking unit 3305 with the ceiling camera loses track of the moving object Mo being tracked, (C) it is associated with each of two or more different moving objects Mo. If the IDs are swapped during tracking, etc. are included (not limited to these, of course).

Depending on each error state, the system adopted in this embodiment takes various actions including the following examples.
(A) The image including the shopper captured by the ceiling camera 310 as an object is transmitted to the inspection terminal Q, and the detection of the shopper by the inspection is requested. When the eye checker detects a shopper, a new moving object Mo is defined, an ID is issued, and tracking by the moving object area tracking unit 3305 by the ceiling camera is started. If the eye checker cannot detect the shopper, the fact is notified to the clerk.
(B) When a shopper who is not associated with any ID registered in the positional information management DB 132 is recognized, the imaged image of the shopper and a list of ID candidates to be associated are displayed in the terminal Q for inspection. Sent to. The eye examiner associates the most appropriate ID with the shopper based on the past association information of the shopper and the ID, and starts re-tracking by the moving object area tracking unit 3305 using the ceiling camera. If the re-tracking cannot be started for some reason, the clerk is informed accordingly.
In addition, when an ID that has lost connection with the shopper is detected, the ID and a list of imaged images of shoppers that are not associated with any ID are transmitted to the inspection terminal Q. The eye examiner attempts to associate the shopper with the ID based on the association information of the shopper and the ID in the past, and when the association is successful, the moving object area tracking unit 3305 by the ceiling camera uses Start retracking. If the start of re-tracking fails for some reason, the clerk is informed accordingly.
(C) The image list of each shopper and the list of IDs that should be associated with each shopper are transmitted to the inspection terminal Q. The eye examiner tries to allocate the ID to the shopper most appropriately based on the past information of the shopper and the ID. When the IDs can be appropriately assigned to the shoppers, the re-tracking by the moving object area tracking unit 3305 by the ceiling camera is started. If the IDs cannot be properly allocated to the shoppers, the allocation is abandoned and the clerk is informed that the ties have been exchanged.

[Embodiment 3]
FIG. 24 is a diagram showing a layout example of a supermarket adopting the product recognition system of the third embodiment.
The product recognition system of the third embodiment is a system applied to a store 30 such as a supermarket as shown in FIG.

The store 30 has a sales floor from the entrance 31 to the exit 32. A plurality of shelf racks 33 for displaying products are installed in the sales floor. A passage 34 is formed between the two shelf racks 33 facing each other.

At the entrance 31, baskets such as shopping carts and shopping carts (not numbered) are placed. The settlement area 35 is located in front of the exit 32. A plurality of cash registers 36 are installed in the settlement area 35. The cashier table 36 is equipped with n settlement machines 4. The third embodiment does not include the cashier terminal 2 as employed in the first and second embodiments. Further, in the third embodiment, unlike the second embodiment in which a shopper is treated as a moving object Mo, baskets grasped (discovered / tracked) by the system are treated as a moving object Mo.

The shopper enters the store through the entrance 31, picks up the basket, and proceeds through the passage 34. The shopper takes the product in the shelf, puts it in the basket, and proceeds through the aisle 34. When all the desired products are purchased, the shopper proceeds to the settlement area 35 and makes the settlement with the settlement machine 4.
In the passage 34, the settlement area 35, etc., the clerk Mt is looking around. The store clerk Mt owns the information terminal 9a. The information terminal 9a is a portable information processing terminal such as a smartphone, and includes a screen for displaying the state of the store.
In FIG. 24, not the inside of the store 30 but the outside of the store 30 or the backyard of the store 30 is depicted inside the cloud-shaped diagram. The server 1 (FIG. 7) is installed outside the store 30. At a management center existing outside the store 30, a store 30 backyard, or the like, the store clerk Mt monitors the inside of the store 30 through the screen of a large monitor (not shown) or the screen of the information terminal 9b. Further, hereinafter, when it is not necessary to distinguish the information terminal 9a and the information terminal 9b, they are collectively referred to as "information terminal 9". Further, although not shown in FIG. 24, in the present embodiment as well, as in the first and second embodiments, the inspection terminal Q is installed.

A plurality of ceiling cameras 310 are separately installed on the ceiling above the passage 34, the shelf rack 33, and other arbitrary positions in the store 30. The ceiling camera 310 captures an image of the passage 34, the shelf rack 33, and a predetermined area therebelow. That is, when the moving object Mo enters, the ceiling camera 310 images the predetermined area including the moving object Mo as a subject.

A shelf camera 311 is installed as an example of a sensing device at each of a plurality of positions on each shelf in each shelf rack 33. The shelf camera 311 images the inside of the shelf, the products inside the shelf, and other predetermined areas. Further, the shelf camera 311 takes an image of the hand or the object in the shelf as an object imaged image when the shopper's hand or the like enters the predetermined area in the shelf or when an object is taken from the shelf. To do.
A basket camera 312 (not shown in FIG. 24) may be attached to each of the baskets as an example of a sensing device.
In that case, one or more basket cameras 312 of the sales floor device 3 are installed in the baskets and always image the objects put in the baskets. The basket camera 312 is installed so that the inside of baskets can be imaged without blind spots. The cage camera 312 captures an image of a characteristic portion of at least the front surface of an object placed in a basket.

Further, the basket camera 312 is linked with the ceiling camera 310 and the shelf camera 311 via the network N. By this cooperation, the sales floor device 3 can share both the captured image captured by the basket camera 312 and the captured image of the object captured by the ceiling camera 310 or the shelf camera 311. By this sharing, it is possible to improve the accuracy of identifying the product from the object imaged in the captured image.

The product recognition system according to the third embodiment has a function of continuously tracking the moving object Mo even if the moving object Mo captured by the ceiling camera 310 moves. The information processing system according to the third embodiment has a function of identifying which product the object picked up from the shelf rack 33 is from the captured image captured by the shelf camera 311.
The product recognition system of the third embodiment may have a function of identifying which product the object put in the basket is from the captured image captured by the basket camera 312.
The product recognition system according to the third embodiment has a function of identifying which product the object in the shelf is and further automatically adjusting the product in the checkout machine 4. At the time of this automatic settlement, the settlement machine 4 reads the information of the product associated with the moving object Mo, and the automatic settlement becomes possible.
The checkout machine 4 includes functions necessary for completing shopping, such as the total price of purchased commodities, points, display of details, display of trade-restricted commodity determination, and settlement function.

The ceiling camera 310, the shelf camera 311, and the basket camera 312 are incorporated in the sales floor device 3 as shown in FIGS. 25 and 26. The sales floor device 3 has a function of specifying a product from a captured image captured by the ceiling camera 310, the shelf camera 311 and the like, and a function of discovering and tracking a moving object Mo. This sales floor device 3 is incorporated in a product recognition system as shown in FIG.

FIG. 25 is a configuration diagram showing the configuration of the product recognition system of the third embodiment.
The product recognition system according to the third embodiment includes a server 1, a sales floor device 3, a settlement machine 4, and a visual inspection terminal Q. In the third embodiment, in order for the sales floor device 3 to specify which product the object is, a cashier 4 is provided instead of the cashier terminal 2 described in the first and second embodiments.
In FIG. 25, only one server 1 and the sales floor device 3 are drawn, but in reality, there may be a plurality of devices. Further, hereinafter, when it is not necessary to individually distinguish the settlement machines 4, these are collectively referred to as a “settlement machine 4”.
The server 1, the sales floor device 3, the settlement machine 4, and the inspection terminal Q are mutually connected via a network N such as the Internet.
The server 1 is configured similarly to the server 1 (FIG. 7) of the first embodiment. The eye examination terminal Q has the same configuration as the eye examination terminal Q of the first embodiment.

FIG. 26 is a block diagram showing the hardware configuration of the sales floor device 3 in the product recognition system of FIG.
The sales floor device 3 includes a CPU 301, a ROM 302, a RAM 303, a bus 304, an input / output interface 305, a ceiling camera 310, a shelf camera 311, a basket camera 312, a communication unit 315, and an information terminal 9. Prepare

The CPU 301, the ROM 302, the RAM 303, the bus 304, the input / output interface 305, and the communication unit 315 of the sales floor device 3 are configured similarly to those of the server 1 illustrated in FIG. 7.
The ceiling camera 310, the shelf camera 311, the communication unit 315, and the information terminal 9 of the sales floor device 3 are configured similarly to those of the sales floor device 3 (FIG. 17) described in the second embodiment.

FIG. 27 is a functional block diagram showing an example of a functional configuration of the server 1, the sales floor device 3, the settlement machine 4, and the inspection terminal Q.

The server 1 includes a CPU 101, a storage unit 108, a communication unit 109, an error display unit 151, and an error canceling unit 152.
These are configured similarly to the second embodiment shown in FIG.

In the terminal Q for eye inspection, the image display control unit 411, when a request for eye inspection is made from the sales floor device 3 or the settlement machine 4, picks up an object imaged image or a product image transmitted from the sales floor device 3 or the settlement machine 4. Images such as images and various information (commodity candidate list etc.) transmitted accompanying these images are output to the output unit 106.
The eye inspection result transmission unit 412 receives the object inspection image or the product captured image output to the output unit 106 from the eye inspection result input by the eye examiner through the input unit 107 (determination of product identification result or trade restricted product). The result, etc.) is transmitted to the sales floor device 3 or the settlement machine 4 that requested the inspection.

In the CPU 301 of the sales floor device 3, as shown in FIG. 27, a personal authentication unit 320, a moving object tracking unit 330, a position information management unit 340, a shelf product recognition unit 360, a basket product recognition unit 370, and trading. The restricted product determination unit 380 and the inspection result acquisition unit 301a function.

The personal authentication unit 320 includes a personal information acquisition unit 321.
The personal authentication unit 320 and the personal information acquisition unit 321 are configured similarly to the second embodiment shown in FIG.

28 is a functional block diagram showing a detailed functional configuration example of the moving object tracking unit 330 provided in the sales floor device 3 of FIG.
As shown in FIG. 28, the moving object tracking unit 330 includes a moving object finding unit 3302 using a ceiling camera, a basket finding unit 3303 using a ceiling camera, a basket area defining unit 3306 using a ceiling camera, and a basket area tracking unit using a ceiling camera. 3307, a grouping unit 3308 an inter-cage area transfer recognition unit 3310 by a ceiling camera, an object recognition unit 3312 transferred by a ceiling camera, and a product specifying unit 3313 transferred by a ceiling camera.
However, the moving object tracking unit 330 of the third embodiment handles baskets that are grasped (discovered / tracked) on the system as the moving object Mo, unlike the second embodiment that handles a shopper as a moving object.

The moving object tracking unit 330 is connected to the ceiling camera 310 via a USB cable, a network N, or the like. Therefore, the ceiling camera 310 is linked with another ceiling camera 310, a personal computer, or the like.

The moving object finding unit 3302 by the ceiling camera finds an object (shopper, basket, cart, etc.) moving in the store by using a state space model (Bayesian filter, etc.) based on the imaged image taken by the ceiling camera 310. .

The ceiling camera-based basket discovery unit 3303 discovers baskets (moving objects Mo) from the objects moving in the store 30 discovered by the moving object detection unit 3302, and assigns individual IDs to the moving objects Mo. . The ID of the moving object Mo is continuously used until a predetermined timing such as leaving the store or completing settlement.
As a method of discovering the moving object Mo from an object moving in the store 30, for example, one or more markers holding individual identifiable information are attached to each basket, and the basket finding unit 3303 by the ceiling camera is used. Detects the moving object Mo using this marker as a mark. The marker is not limited as long as it can identify the moving object Mo such as a two-dimensional code or a characteristic shape.
As another method for finding the moving object Mo from the object moving in the store 30, the basket finding unit 3303 using the ceiling camera may use information unique to the basket such as color information and shape information of the baskets. At that time, the basket finding unit 3303 using the ceiling camera can find the moving object Mo because the colors and shapes of the baskets can be distinguished from the floor surface and the like.

As a method of discovering the moving object Mo from an object moving in the store 30, thermography or the like may be used with the baskets kept at a low temperature. At that time, the basket finding unit 3303 using the ceiling camera finds the moving object Mo that has been cooled to a low temperature from the temperature difference between the basket and the region other than the baskets.
As a method of finding the moving object Mo from the object moving in the store 30, thermography or the like may be used by generating harmless gas from the baskets. At that time, the basket finding unit 3303 using the ceiling camera detects a temperature change associated with the generation of harmless gas generated from the baskets by thermography or the like, and discovers the moving object Mo.

As a method of discovering the moving object Mo from the object moving in the store 30, sound may be generated from the baskets regardless of whether or not the sound is audible. At that time, the basket finding unit 3303 using the ceiling camera finds the moving object Mo by detecting the sound generated from the baskets.
As a method of discovering the moving object Mo from an object moving in the store 30, a sensor that applies invisible paint to a basket and can recognize the invisible paint may be used. At that time, the basket finding unit 3303 using the ceiling camera finds the moving object Mo by the sensor recognizing the invisible paint applied to the baskets.
As a method of finding the moving object Mo from the object moving in the store 30, the baskets may be irradiated with visible light, infrared rays, or the like from the ceiling camera 310. At that time, the basket finding unit 3303 with the ceiling camera finds the moving object Mo by receiving the visible light or the reflected light of infrared rays with which the baskets are irradiated.

The ceiling camera basket area definition unit 3306 defines, as the area of the moving object Mo, a certain range within the captured image around the moving object Mo found by the ceiling camera basket finding unit 3303. To define the area of the moving object Mo, for example, when a marker is attached to the baskets as described above, a certain range from the attachment position where the marker is attached is defined as the area of the moving object Mo.
When a plurality of markers are attached to the baskets, even if the ceiling camera 310 cannot capture all the markers, the marker area definition unit 3306 of the ceiling camera does not allow each marker to position itself in the area of the moving object Mo. Since the information is held, the area of the moving object Mo may be defined by one or more markers.

The basket area definition unit 3306 using the ceiling camera may use a method that complements the above-described method for defining the basket area. As a complementary method, when the baskets are cooled to a low temperature as described above, the region of low temperature is defined as the region of the moving object Mo from thermography and image recognition.
As another complementary method, in the case of generating harmless gas from baskets as described above, the temperature change due to the generation of gas is detected by thermography and the like, and by making full use of image recognition technology, the temperature change The area where there is is defined as the area of the moving object Mo.

As yet another complementary method, the basket area definition unit 3306 by the ceiling camera estimates the edge of the basket from the coating position of the paint when the invisible paint is applied to the edge of the basket as described above, The area closed by the edge is defined as the area of the moving object Mo.
As yet another complementary method, when the ceiling camera 310 irradiates visible light or infrared rays from the ceiling camera 310 as described above, the basket area definition unit 3306 determines the area of the moving object Mo from the measurement result of the reflection. Define.

The basket area tracking unit 3307 using the ceiling camera estimates (defines) the position of the area of the moving object Mo. The ceiling camera basket area tracking unit 3307 tracks the moving object Mo with the same ID from the time when the moving object Mo is found to a predetermined time when the store is closed or the settlement is completed, and keeps track of the position information.

Therefore, the basket area tracking unit 3307 by the ceiling camera always keeps track of the basket area in the captured image, for example, in cooperation with a large number of ceiling cameras 310. In order to allow a large number of ceiling cameras 310 to cooperate with each other, the basket area tracking unit 3307 using the ceiling cameras may transfer the images captured by a certain ceiling camera 310 to the images captured by adjacent ceiling cameras. To do. The ceiling camera basket area tracking unit 3307 stores the position information of the moving object Mo being tracked in the position information management DB 132 of the server 1 or the storage unit 108.

As another method for the basket area tracking unit 3307 using the ceiling camera to track the moving object Mo, a marker having information identifying the moving object Mo is attached to each basket, and the ceiling camera 310 moves the moving object including the marker. Image Mo. The basket area tracking unit 3307 that uses the ceiling camera discovers the moving object Mo and acquires the position information by extracting the marker from the captured image.
Even if the moving object Mo moves, the basket area tracking unit 3307 using the ceiling camera can continue to track the moving object Mo by finding the marker from the captured image and acquiring the position information of the moving object Mo.

As another method for tracking the moving object Mo, the basket area tracking unit 3307 using the ceiling camera tracks the moving object Mo by using an object tracking technique in the image such as Bayesian filter, fast Fourier transform, and TLD. Good.
Alternatively, the basket area tracking unit 3307 using the ceiling camera estimates that the moving objects Mo that have acquired the same feature data are the same moving object Mo based on the feature data such as the colors and shapes of the baskets. The moving object Mo may be tracked. At that time, the basket area tracking unit 3307 using the ceiling camera continues to collect the characteristic data of the tracking target.

In any case, in the image from the ceiling camera 310, the moving object Mo that is not directly facing the ceiling camera 310 is captured at an angle (from an oblique direction), and thus the position information may not be accurately acquired. Therefore, it is conceivable that the captured image is corrected so that the image is directly faced. However, even if such a correction is applied, the position information of the moving object Mo may not be acquired with high accuracy.
Therefore, the basket area tracking unit 3307 using the ceiling camera acquires high-accuracy position information by acquiring height information of the moving object Mo using a distance sensor or the like. In this way, the basket area tracking unit 3307 using the ceiling camera may continue to track the moving object Mo.

The grouping unit 3308 may associate a plurality of baskets (moving objects Mo). By being linked in this way, the product list of each moving object Mo can be collectively settled in the settlement machine 4 installed in the settlement area 35.

The inter-cargo area passing / recognizing unit 3310 by the ceiling camera recognizes that the object is passed (moved in and out) between the moving objects Mo by the ceiling camera 310 and the like. The basket area transfer recognition unit 3310 using the ceiling camera may recognize the overlap between the areas of the moving object Mo to recognize the transfer. The inter-cargo area passing / recognizing unit 3310 by the ceiling camera identifies the moving object Mo to which the object is transferred, and reads the product list associated with the ID of each moving object Mo.

The object recognition unit 3312 delivered by the ceiling camera defines the area of the object from the captured image at the time when the delivery is recognized.
The object recognition unit 3312 delivered by the ceiling camera may make the ceiling camera 310 a zoomable camera, and may zoom up the location where the delivery is supposed to be performed to define the area of the object.

The product identification unit 3313 delivered by the ceiling camera identifies which of the products in the product list associated with the moving object Mo the delivered object is from the image after the object area definition, The moving object Mo specified by the inter-cargo area passing / recognizing unit 3310 by the ceiling camera is linked with the product specified by the transfer, and the list of products linked to each moving object Mo is updated.
The object recognition unit 3312 delivered by the ceiling camera and the product identification unit 3313 delivered by the ceiling camera may be realized by a shelf camera 311 or the like capable of capturing a wide range instead of the ceiling camera 310.

The position information management unit 340 is configured similarly to the position information management unit 340 described in the second embodiment.
That is, the position information management unit 340 includes an inter-camera information transfer unit 341, a position definition unit 342 of each camera, and a moving object display unit 343, as shown in FIG.

FIG. 29 is a functional block diagram showing a detailed functional configuration of the shelf product recognition unit 360 provided in the sales floor device 3 of FIG.
The shelf product recognition unit 360 includes an object recognition unit 3602 using a shelf camera, a product specifying unit 3603 using a shelf camera, a basket and product association unit 3605, a product list management unit 3607 associated with the basket, and an object input / output by the shelf camera. A detection unit 3608, a product unspecified determination unit 3609, a label recognition unit 3610, a discount sticker recognition unit 3611, and a basket entry / exit detection unit 3612 using a shelf camera or a ceiling camera are provided.
The shelf product recognition unit 360 is linked with another camera, a personal computer, or the like through the shelf camera 311 and a USB cable or a network.

The object recognition unit 3602 using the shelf camera compares the images before and after the image in which the object is taken from the shelf or the object is placed (returned) in the shelf, and the image area targeted for product identification is determined. Define (area definition).
That is, the object recognition unit 3602 by the shelf camera uses the object imaged image before the object is taken from the shelf or placed on the shelf and the object taken image after the object is taken from the shelf or placed on the shelf. The object captured images before and after the image change are compared, and the changed image area is specified. The object recognition unit 3602 by the shelf camera confirms the change in each of the RGB data when specifying the image area.

Further, the object recognition unit 3602 by the shelf camera uses the object entrance / exit detection unit as a trigger to capture an image of the object taken from the inside of the shelf or placed on the shelf, and the object taken from the shelf or the shelf. The area of the object may be defined from only one object-captured image by using a method different from the comparison before and after the change of the object-captured image after being placed on the.
Since the color data of the object is the same, the object recognition unit 3602 by the shelf camera also defines the area by using the shadow of the object so that it is not determined that there is no change even if there is a change such as entry and exit. You may.
The information on the recognized object area is passed to the product specifying unit 3603 by the shelf camera.

The shelf camera product identification unit 3603 identifies which product is the object in the shelf recognized by the shelf camera object recognition unit 3602. The shelf camera product identification unit 3603 lists up product candidates by image processing methods such as specific object recognition, general object recognition, and deep learning. The listed product candidates are called “product candidate list S”. After that, the product specification unit 3603 using the shelf camera specifies the product with high accuracy by performing the verification function.

The verification function lists up the "commodity candidate list P" by an algorithm different from the method of listing up the product candidates described above. The results of the product candidate lists S and P are matched with each other, and a product is specified when the result exceeds a predetermined threshold.
The method of listing the “commodity candidate list” may be realized by, for example, a method of matching image information of an object obtained from an object whose existence is recognized with image information held in the product DB 131 or the memory. . That is, when the feature information of both images matches (exceeds a threshold value), the object whose existence is recognized by the object recognition unit 3602 by the shelf camera is a product registered in the product DB 131, and thus the product identification by the shelf camera is performed. The section 3603 specifies that the product is registered in the product DB 131. A product candidate is created by deep learning, and then the verification function is exerted to specify the product with high accuracy.

The shelf camera-based product specification unit 3603 does not specify the product in one frame of the captured image captured by the shelf camera 311, but also uses the captured image captured by the ceiling camera 310 to identify the product over a plurality of captured images. May be. At that time, the product specifying unit 3603 using the shelf camera gives a percentage to the product candidate, adds the percentage based on information such as purchase history, time, place, and person's preference, and exceeds a certain threshold. Identify the product when

The basket and the product association unit 3605 associate the product information of the product identified by the product identification unit 3603 by the shelf camera with the moving object Mo. As a premise of this, if only one moving object Mo exists in a predetermined area imaged by one shelf camera 311, by specifying the ID of the moving object Mo, the moving object Mo and the product Information can be linked. At this time, the basket and the product association unit 3605 specify the ID of the moving object Mo using the position information attached to the marker or the like attached to the moving object Mo.

As another premise, if there are a plurality of moving objects Mo in a predetermined area imaged by one shelf camera 311, the basket entrance / exit detection unit by a shelf camera or a ceiling camera described later moves into the moving object Mo. As a result of detecting the entry / exit of the object, the moving object Mo containing the object taken from the shelf is identified, and the moving object Mo and the product information are associated with each other.
As yet another premise, if it is determined that no moving object Mo exists in the predetermined area imaged by one shelf camera 311, at least one shelf camera 311 or at least one shelf camera 311 is used. The car camera 312 detects entry into and exit from the moving object Mo, specifies the moving object Mo, and then associates the specified moving object Mo with the product information.

The product list management unit 3607 associated with the basket continues to manage the product list associated with the moving object Mo and the specified product until the settlement. That is, the product list management unit 3607 associated with the basket uses the position information management DB 132 and the like to constantly manage the list of products associated with the ID of the moving object Mo.
When the object is taken from the shelf, the product list management unit 3607 associated with the basket is added by the number of the acquired products. On the contrary, when the product is returned to the shelf, the product list management unit 3607 associated with the basket subtracts the number of the returned products.

The object entering / exiting detection unit 3608 by the shelf camera can act as a trigger for activating the object recognition unit 3602 by the shelf camera by detecting that an object has entered the shelf.
For example, the object entry / exit detection unit 3608 by the shelf camera detects the entry of the object into the shelf from the change in the image data in the “entry detection area” set in the captured image of each shelf camera 311. Further, the object entrance / exit detection unit of the shelf camera also detects that the object exits the area by tracking the incoming object in the image.

In order to track the object, the object entrance / exit detection unit 3608 by the shelf camera applies a particle filter to the entrance detection area in the shelf. In order to track a plurality of moving objects Mo at the same time as in the case where a plurality of shoppers reach their hands at the same time, after the first object enters, the particles are again spread over the “entry detection area”. , Prepare for the next entry.
However, if it is premised that a plurality of objects do not enter the same area, the object entering / exiting detection unit 3608 by the shelf camera does not sprinkle particles in the entry detection area that is present in the area where the object already exists.

The object entrance / exit detection unit 3608 by the shelf camera determines “object entrance” when the ratio of particles having a certain likelihood is equal to or more than a threshold value in a predetermined area, and when the particles having a certain likelihood is less than the threshold value. Is determined to be "object".
The object entry / exit detection unit of the shelf camera detects entry / exit of a product each time a product is entered so that the area of the object is estimated as small as possible once.
The images before and after the change are stored in the storage unit 108 of the server 1 so that they can be used in the estimation area.

The object entry / exit detection unit 3608 using the shelf camera may detect object entry / exit from spectrum (wavelength) data obtained from a captured image of the object, in addition to the above example. Further, the object entrance / exit detection unit using the shelf camera may detect the object entrance / exit using a method such as a weight / pressure sensor, an infrared sensor, or an ethylene gas sensor.

The product non-identification determination unit 3609 associates that the product could not be identified by the product identification unit 3603 by the shelf camera with the moving object Mo that picked up the object from the shelf.

Here, a case where the product cannot be specified by the verification function of the product specifying unit 3603 using the shelf camera will be described. First, the shelf camera product identification unit 3603 identifies the number of feature points at which the image of the object taken from the shelf and the image of the object placed on the shelf are similar. For example, the size of the object imaged in both images is specified and compared, the color difference is specified, and the colors are compared to determine whether the two objects are similar. If the number of characteristic points is small, the product cannot be specified, and it is possible to prevent accidental payment.

The label recognition unit 3610 recognizes the attached label according to the product specified by the product specification unit 3603 by the shelf camera. The label recognition unit 3610 makes full use of an image recognition method to read a multidimensional code including a character, a barcode, etc. written on the label and complement product identification.

The discount sticker recognizing unit 3611 recognizes the discount sticker attached according to the product specified by the product specifying unit 3603 by the shelf camera. The discount sticker recognizing unit 3611 makes full use of the image recognition method to identify the discount amount or the discount rate of the discount sticker attached to the product. The discount label recognition unit 3611 is executed during the processing of the product identification unit.

In the basket entry / exit detection unit 3612 using the shelf camera or the ceiling camera, the moving object Mo exists in a predetermined area of the shelf camera and the ceiling camera from a marker or the like to which at least one of the shelf camera 311 and the ceiling camera 310 is attached to the basket. Whether or not the moving object Mo is present or not is detected. The car entrance / exit detection unit 3612 using the shelf camera or the ceiling camera may further detect the entry of the moving object Mo defined by a marker or the like into the area. At that time, the moving object Mo enters the area by comparing the position information of the tracked product with the position information of the entrance detection line (the edge of the basket or the like) of the area of the moving object Mo.

The information processing system of the third embodiment may further include a basket product recognition unit 370.
FIG. 30 is a functional block diagram showing a detailed functional configuration example of the basket product recognition unit 370 provided in the sales floor device 3 of FIG.
The basket product recognition unit 370 includes a basket camera entrance / exit detection unit 372, a basket camera object recognition unit 373, a basket camera product identification unit 374, a basket product non-identification determination unit 375, and a basket product label recognition. A unit 376 and a basket product discount seal recognition unit 377 are provided.

When the object such as the hand of the shopper enters the basket (moving object Mo), the car entrance / exit detection unit 372 using the car camera detects the entrance, and thus the object recognition unit 373 using the car camera is activated. Can be the trigger of. The car entrance / exit detection unit 372 by the car camera detects the entrance of an object into the car from the change of the image in the “entry detection area” set in the image captured by each car camera 312.

The car entrance / exit detection unit 372 of the car camera also detects that the moving object Mo leaves the frame by tracking an object that has entered the frame of the captured image within the frame. The car entrance / exit detection unit 372 using the car camera tracks an object using a particle filter.
The car entrance / exit detection unit 372 of the car camera tracks a plurality of moving objects Mo at the same time, so that after the first object enters, particles are again applied to the entrance detection area to prepare for the next object to enter.
The car entrance / exit detection unit 372 using the car camera determines “object entrance” when the ratio of particles having a certain likelihood is equal to or more than a threshold value within a predetermined area, and “object exit” when the ratio is less than the threshold value. To determine. The car entrance / exit detection unit of the car camera detects entrance / exit every time an object enters / exits so as to estimate a region of the object as small as possible once.

The object recognition unit 373 by the basket camera compares an image when an object such as a hand of a shopper enters the baskets (moving object Mo) with an image after exiting from the baskets (moving object Mo). , An image area is defined in order to specify an object which is taken from the shelf and put in the basket (moving object Mo) (or returned from the basket (moving object Mo) to the shelf). The object recognition unit 373 by the car camera confirms the change of the area using the RGB data.
The object recognition unit 373 using the basket camera may recognize an object from spectrum (wavelength) data obtained from a captured image of the object, a weight sensor, a pressure sensor, an infrared sensor, a methylene gas sensor, or the like.

The product identification unit 374 by the basket camera identifies which product is the object put in the basket. The product identification unit 374 by the car camera lists up product candidates for the object recognized by the object recognition unit 373 by the car camera by image processing methods such as specific object recognition, general object recognition, and deep learning. The listed product candidates are called “product candidate list S”. After that, the product identification unit 374 by the car camera causes the verification function to be performed and identifies the product with high accuracy.

The verification function lists up the "commodity candidate list P" by an algorithm different from the above-mentioned method of listing up product candidates. The product identification unit 374 using the basket camera matches the product candidate lists S and P, and identifies a product when a predetermined threshold is exceeded.
As a method of listing product candidates, for example, a method of matching information of an object captured image obtained from an object whose existence is recognized with image information held in the product DB 131 or the storage unit 108 is realized. Good. That is, when the feature information of both images matches, that is, exceeds the threshold value, the object whose existence is recognized by the object recognition unit 373 by the car camera is specified as the product registered in the product DB 131.
The product identification unit 374 including the basket camera may specify the product from the captured images over a plurality of frames instead of specifying the product in one frame of the captured image. At that time, the product identification unit 374 by the car camera gives a percentage to the product candidates, adds the percentage based on information such as purchase history, time, place, and person's preference, and exceeds a certain threshold. Sometimes the product may be specified.
In the present embodiment, there are a specific object inspection mode in which the inspection terminal Q performs inspection for a specific object and an all-object inspection mode in which inspection is performed by the inspection terminal Q for all objects. It can be set.
When the specific object visual inspection mode is set, the product identification unit 374 by the basket camera visually examines the object captured image of the object that is not specified as the product and the related information via the eye inspection result acquisition unit 301a. To the terminal Q for use. On the other hand, when the all-object visual inspection mode is set, the product identification unit 374 using the basket camera determines the product identification result and the product captured image of the identified product, and the object captured image of the object that is not identified as the product. Related information and the like are transmitted to the inspection terminal Q via the inspection result acquisition unit 301a.
Then, the product identification unit 374 using the basket camera identifies the product based on the eye inspection result acquired by the eye inspection result acquisition unit 301a from the eye inspection terminal Q. Specifically, for the product identified by the product identification unit 374 by the car camera, when the identification result is approved or corrected by the result of the eye inspection, the product identification unit 374 by the car camera follows the contents indicated by the result of the eye inspection. , Specify the product. Further, for an object for which the product identification unit 374 for the car camera determines that the product is unspecified, the product specification unit 374 for the car camera identifies the object as a product indicated by the visual inspection result. When the product identification unit 374 using the basket camera requests the product identification by visual inspection, the request may be made after the product identification complement by the label recognition unit 376 for the basket product is completed. In this case, as a result of the product identification complement by the basket product label recognition unit 376, products whose products have been identified can be excluded, and products can be identified by visual inspection.

The basket product non-identification determination unit 375 associates the fact that the product placed in the basket cannot be recognized as a product by the product identification unit 374 of the basket camera with the ID of the moving object Mo.
The basket product label recognition unit 376 recognizes the attached label according to the product specified in the basket by the product specification unit 374 by the car camera. The label recognition unit of the basket product makes full use of the image recognition method to read the multidimensional code including the characters, the barcode, etc. written on the label to complement the product identification.
The basket product discount sticker recognizing unit 377 makes full use of the image recognition method to specify the discount amount of the discount sticker attached to the product specified by the product specifying unit 374 by the car camera and the discount rate. The discount sticker recognizing unit for the basket product identifies the discount amount or the like during the processing by the product identifying unit using the basket camera.

The information processing system also includes a trade-restricted product determination unit 380 and a remote operation unit 390.
The trade-restricted product determination unit 380 determines whether the specified product is a trade-restricted product, as described in the first embodiment. When the trade restricted product determination unit detects the trade restricted product, the trade restricted product determination unit displays the information on the error display unit 151.
In the present embodiment, the trade-restricted product determination unit 380 transmits the product captured image of the product corresponding to the trade-restricted product to the eye inspection terminal Q via the eye inspection result acquisition unit 301a. At this time, the trade-restricted product determination unit 380 appropriately sends information about the shopper (face image, personal information, etc.) to the inspection-use terminal Q via the inspection-result acquisition unit 301a according to the type of the sale-restricted product. Send.
Then, the trade-restricted product determination unit 380 determines whether or not the sale of the trade-restricted product is permitted based on the inspection result acquired by the inspection result acquisition unit 301a from the inspection terminal Q (the sale-restriction is released). Whether or not) is determined.
The remote operation unit 390 is provided in the information terminal 38 and the server 1, and has a function of canceling the error state when the error state is notified, for example.

The eye inspection result acquisition unit 301a requests the identification of the product by the eye inspection by transmitting the object captured image or the product identification result and the product captured image of the identified product to the eye inspection terminal Q. The eye inspection result transmitted from the eye inspection terminal Q is acquired in response to the request. The visual inspection result acquisition unit 301a outputs the acquired visual inspection result (product identification result) to the shelf product recognition unit 360. Further, the eye inspection result acquisition unit 301a requests the determination of the sale restricted product by the eye inspection by transmitting the product imaged image of the product corresponding to the sale restricted product to the eye inspection terminal Q, and the eye inspection terminal Q Get the result of the eye inspection sent from. At this time, the visual inspection result acquisition unit 301a appropriately transmits information (face image, personal information, etc.) regarding the shopper to the visual inspection terminal Q. The visual inspection result acquisition unit 301a outputs the acquired visual inspection result (determination result of the trade restricted product) to the trade restricted product determination unit 380.

The settlement machine 4 installed in the settlement area 35 calculates the total amount of money of all the commodities put in one or more baskets, and performs settlement or settlement. The settlement machine performs settlement based on the moving object Mo that has been continuously tracked and the product information associated with the moving object Mo.
Therefore, as shown in FIG. 27, the settlement machine 4 includes a CPU 401, an input unit 406, an output unit 407, and a communication unit 409.
In the CPU 401, the settlement unit 435, the input control unit 436, and the output control unit 437 function.

The input unit 406 includes a credit card, an electronic money reading unit, and the like.
The output unit 407 has a function of displaying a screen for displaying a payment item and a receipt.
The settlement unit 435 determines the settlement amount and the item to be settled.
The input control unit 436 inputs a signal from the input unit 406 and operates the CPU 401.
The output control unit 437 outputs the calculation result of the settlement unit 435 to the output unit 407.
The settlement machine 4 compares the positional information of the moving object Mo with the predetermined positional relation of the settlement area 35, and the positional information of the moving object Mo has entered the settlement area 35, or the moving object Mo is the settlement machine. Settled in 4 as a trigger.

Alternatively, the settlement machine 4 may be provided with a button for instructing to start the settlement, and the settlement may be triggered by a button press or the like. When the settlement machine 4 is provided with a weight sensor (not shown) and the moving object Mo is placed. It is also possible to recognize the change in the weight of the item and make the settlement by the settlement machine 4.
The settlement machine 4 can be used not only for cash, but also for gift certificates, cash vouchers, virtual currency, and the like.
In such a settlement machine 4, an error state may occur. The error state includes (A) system processing abnormality, (B) a product unspecified object is linked to the moving object Mo, (C) a trade-restricted product is linked to the moving object Mo, and the like. Examples include (but are not limited to, of course).

The system adopted by the present embodiment takes various actions according to each error state.
In the case of (A), the fact that the system processing is abnormal is presented to the settlement machine 4, the information terminal 38a, the information terminal 38b, and the server 1. This allows the store clerk to eliminate the error condition. The elimination of the error state may be performed by the remote operation unit 390.
In the case of (B), the fact that an unspecified item is linked to the moving object Mo is presented to the settlement machine 4, the information terminal 38a, the information terminal 38b, and the server 1. In the present embodiment, when the error state (B) occurs, a visual inspection is performed at the visual inspection terminal Q, and the settlement is continued or stopped depending on the visual inspection result. As a result, it is possible to increase the number of cases in which settlement can be performed automatically. When the error state of (B) occurs, the clerk may eliminate the error state. The error state can be resolved by the remote control unit 390.
In the case of (C), the fact that the trade-restricted product is linked to the moving object Mo is presented to the settlement machine 4, the information terminal 38a, the information terminal 38b, and the server 1. In the present embodiment, when the error state of (C) occurs, the visual inspection at the visual inspection terminal Q is performed, and the settlement is continued or stopped depending on the visual inspection result. As a result, it is possible to increase the number of cases in which settlement can be performed automatically. In addition, when the error state of (C) occurs, for example, in the case of a trade-restricted product due to age restriction, a clerk checks the age of the shopper, or in the case of a trade-restricted product due to a product whose consumption / expiration date has expired. If so, the store clerk may exchange the product, or if the product is a trade-restricted product other than allergy / halal food, the shopper may confirm the error by himself / herself to eliminate the error condition. The error state can be resolved by the remote control unit 390.

Here, a method of identifying a product from an object and paying for the product by the product recognition system according to the third embodiment will be described with reference to FIG.
FIG. 31 is a flowchart for explaining the basic flow of the automatic settlement processing executed by the server 1, the sales floor device 3, and the settlement machine 4 of FIG.

In step S301, a shopper (moving object) enters the store from the entrance 31 of the store 30 (FIG. 24), and the ceiling camera 310 installed near the entrance 31 starts imaging the shopper. When the shopper holds a basket or a card and proceeds through the aisle 34, the ceiling camera 310 at the back starts capturing an image of the shopper. In this way, the plurality of ceiling cameras 310 constantly capture images of the entire store 30 including shoppers, baskets, and carts. Note that the personal authentication unit 320 may perform personal authentication of the shopper and acquire personal information of the shopper before step S301.

In step S302, the moving object finding unit 3302 using the ceiling camera finds a moving object (not numbered) in the captured image using a state space model (Bayesian filter) or the like.
In the captured image, not only an object moving in the store 30 but also a stationary object is imaged. Therefore, the moving object finding unit 3302 by the ceiling camera finds only the object moving in the store 30 from which the stationary object is removed. .

In step S303, the ceiling camera basket finding unit 3303 discovers baskets (moving object Mo in the third embodiment) from the objects found by the ceiling camera moving object finding unit 3302, and obtains individual IDs. To turn. The ID is continuously used until a specific timing such as leaving the store or completing settlement.

In step S304, the ceiling camera basket area definition unit 3306 defines the position of the area of the moving object Mo discovered by the ceiling camera basket discovery unit 3303. Further, when the moving object Mo moves within the range imaged by the ceiling camera 310, the position of the area of the moving object Mo after the movement is defined again. The position information is managed in the position information management DB 132, a memory or the like in association with the ID of the moving object Mo, and is updated for each area definition. This defined position is also recognized as a position imaged by another ceiling camera 310.

In step S <b> 305, the ceiling camera basket area tracking unit 3307 estimates the position where the moving object Mo moves within the captured image captured by a certain ceiling camera 310. Further, the area of the moving object Mo is defined for the position where the basket area defining unit 3306 by the ceiling camera is estimated to have moved, and the position information of the moving object Mo stored in the position information management DB 132 or the memory is updated. To do.

In step S306, the object entering / exiting detection unit 3608 by the shelf camera detects whether an object such as the hand of the shopper has entered or exited the shelf. This detection triggers the object recognition unit 3602 by the shelf camera. The entry of the object into the shelf is detected by whether or not the image data in the entry detection area set in each shelf camera 311 has changed.
The object entrance / exit detection unit 3608 by the shelf camera detects that the object exits the shelf by keeping track of the object that has entered the shelf.

In step S307, the object recognition unit 3602 by the shelf camera compares the captured image before the object has entered the shelf with the captured image after the moving object Mo leaves the shelf, and the captured image is taken from the shelf. Recognize objects

In step S308, the shelf camera product identification unit 3603 identifies which product the detected moving object Mo is. The product specifying unit 3603 using the shelf camera lists up product candidates for the objects detected by the object recognizing unit 3602 by image processing methods such as specific object recognition, general object recognition, and deep learning. The listed product candidates are called “product candidate list S”. After that, the product specification unit 3603 by the shelf camera causes the verification function to be performed and specifies the product with high accuracy.

The verification function lists up the "commodity candidate list P" by an algorithm different from the above-mentioned method of listing up product candidates. The shelf camera-based product specification unit 3603 matches the product candidate lists S and P, and specifies a product when a predetermined threshold is exceeded.
As a method of listing product candidates, for example, a method of matching information of an object captured image obtained from an object whose existence is recognized with image information held in the product DB 131 or the storage unit 108 is realized. Good. That is, when the feature information of both images matches, that is, exceeds the threshold value, the object whose existence is recognized by the object recognition unit 3602 by the shelf camera is specified as the product held in the product DB 131.
The product non-identification determination unit 3609 transmits error information to the server 1 when the product cannot be identified. This error information is displayed on the error display unit 151 and the information terminal 38.

In step S309, the basket and product association unit 3605 associates the product identified by the product identification unit 3603 with the shelf camera with the ID of the moving object Mo. That is, one of the baskets (moving objects Mo) held by the shopper who picks up the specified product is specified.
Further, when the product non-identification determination unit 3609 fails to identify the product when the product or the object is taken from or returned from the shelf, the product non-identification information (error information) is associated with the ID of the moving object Mo.

In step S310, the product list management unit 357 associated with the moving object Mo continues to manage the list of products associated with the ID of the moving object Mo. This management is continued until a predetermined timing such as when the moving object Mo moves to the settlement area 35.

In step S311, when all the commodities of the moving object Mo are summed up in the settlement area 35 and settled or settled by the settlement machine 37, the product list associated with the moving object Mo is updated to the settled status, and the automatic settlement processing is performed. To finish. If the moving object Mo is associated with the product-unspecified information (error information), the error information is transmitted to the server 1 without ending the automatic settlement process. This error information is displayed on the error display unit 151 and the information terminal 38. That is, the clerk can check and cancel the error by calling the shopper.

When the moving object Mo that has not been settled passes through the exit 32, error information is transmitted to the server 1. This error information is displayed on the error display unit 151, the information terminal 38a, and the information terminal 38b. When an alarm device (not shown) is provided near the exit 32, the alarm device issues a warning by sound, light or the like.
Since the automatic settlement processing of the third embodiment specifies the product based on the image captured by the shelf camera, it is not necessary to specify the product for the object placed in the basket.
However, in order to verify whether the product has been mistakenly specified, it is preferable to specify the product for the object placed in the basket.

FIG. 32 is a flow chart for explaining a process of recognizing a product in the basket of the automatic settlement process executed by the server sales floor device and the settlement machine of FIG.

In step S321, the car camera entrance / exit detection unit 372 installs particles in the entrance detection region set in advance at a predetermined position in the captured image so that the likelihood other than the car region is low.
In step S322, the car camera entrance / exit detection unit 372 detects that an object (a shopper's hand or the like) has entered an entrance detection area in the car based on the likelihood of particles. In addition, in order to prepare for the entry of a plurality of objects, after the objects enter the entry detection area in the basket, the basket entry / exit detection unit 372 by the basket camera installs new particles.
In step S323, the basket camera 312 images the state of the inside of the basket at the time when an object (a shopper's hand or the like) enters the entrance detection area. The car camera 312 also images outside the approach detection area.
For example, it is assumed that three products are captured in the captured image (previous image). The product captured in the previous image is a product because it has already been identified from the object.

In step S324, the basket camera 312 images the state of the inside of the basket at the time when the object exits the entry detection area. For example, it is assumed that, in the image (post-image) captured at this time, one object is captured in addition to the above-described three products.

In step S325, the object recognition unit 373 by the car camera compares the front image and the rear image, and defines the image area for one added object.
In step S326, the product identification unit 374 by the car camera identifies which product the increased one object is. This product can be specified by using the same method as the specific method performed by the shelf product recognition unit 360.

In step S327, the basket camera product specification unit 374 determines whether the specific object visual inspection mode or the all object visual inspection mode is set.
If the specific object inspection mode is set, the process proceeds to step S328.
If the all-object visual inspection mode is set, the process proceeds to step S329.
In step S328, the visual inspection result acquisition unit 301a requests a visual inspection (specification of a product) at the visual inspection terminal Q for the target object, and acquires the visual inspection result.
After step S328, the process proceeds to step S329.
In step S329, the product identification unit 374 for the car camera determines whether it is not possible to identify which product the added one object is. If the product identification unit 374 by the car camera cannot identify which product (YES in step S329), the process proceeds to step S327.

When it is possible to identify which product the product identification unit 374 by the car camera is in step S329 (NO in step S329), the product identification unit 374 by the car camera determines the product name stored in the storage unit of the server 1 or A product is specified including information such as a price and a product that is restricted for sale. As a result, the process proceeds to step S330. The identified product information may be output to the output unit 407.
In addition, in step S329, when it is not possible to identify which product the product identification unit 374 by the car camera is, the screen display of a display unit (not shown) provided in the car camera, voice guidance by a speaker (not shown), or the like, The purchaser may be requested to reinsert the product into the basket. Further, the purchaser and the eye examiner may talk with each other via a microphone (not shown) provided in the car camera.

In step S330, the trade-restricted product determination unit 380 determines whether or not the product identified by the product identification unit 374 by the car camera is a product that requires age confirmation.

If it is determined in step S330 that the product identified by the product identification unit 374 by the car camera is a product for which age confirmation is required, that is, if YES, the process proceeds to step S331.
In step S330, if it is determined that the product identified by the product identification unit 374 by the car camera is not a product that requires age confirmation, that is, if NO, the recognition process of the product in the basket ends. Becomes
In step S331, the output control unit 437 causes the output unit 407 of the settlement machine 4 to display a screen for age confirmation. However, if the personal information of the shopper is acquired and it is not necessary to confirm the age here, step S330 is skipped and the process proceeds to step S335. Here, when there is a problem in the age verification of the shopper, the purchaser and the eye checker may make a call via a speaker (not shown) and a microphone (not shown) provided in the checkout machine 4.
In step S332, the visual inspection result acquisition unit 301a requests a visual inspection (determination of a trade-restricted commercial product) at the visual inspection terminal Q for the target product, and acquires the visual inspection result.
In step S333, the trade restriction product determination unit 380 determines whether or not an instruction to cancel the trade restriction has been received.

If it is determined in step S333 that the instruction to cancel the trading restriction has not been received, the process proceeds to step S334.
If it is determined in step S333 that the cancel instruction for canceling the trading restriction has been received, that is, if it is determined to be YES, the process proceeds to step S335.
In step S334, the visual inspection result transmission unit 412 of the visual inspection terminal Q transmits a warning that the trading restriction has not been released even by the visual inspection result. By receiving this warning, for example, the output control unit 437 of the settlement machine 4 presents, through the output unit 407, a warning that the trading restriction has not been canceled even by the result of the visual inspection. After step S334, the process ends and the settlement is stopped.
In step S335, the trade restriction product determination unit 380 cancels the trade restriction.
In this way, when step S335 is completed, or when it is determined in step S330 that the product is not age-restricted product (NO is determined), the recognition process of the in-car product is completed.

In this way, the information processing system identifies which product is the object put in the basket.
Then, the information processing system can verify whether or not the product specified in step S308 and the product specified in step S326 match. When the product is not specified, the error status is displayed on the error display unit 151, the information terminal 38a, and the information terminal 38b. Then, for the objects and products to be inspected (for example, objects for which the product cannot be specified at the cashier terminal 2 and products corresponding to restricted products), the inspection at the inspection terminal Q is performed. That is, the automatic settlement is performed according to the result of the visual inspection (the result of specifying the product, the result of determining the trade prohibited product, etc.).
Therefore, according to the present information processing system, when the shopper purchases a product, it is possible to automate the payment of the price of the product and to improve the accuracy of specifying the product.
In addition, when an object or a product for which settlement is not appropriate is subjected to settlement, it can be prevented from being settled by mistake. Further, when the fact that the product is unspecified is displayed on the information terminals 38a and 38b, the store clerk may go to the sales floor where the basket is located and verify the product, and the store clerk inside and outside the store can remotely check. The error state may be released by operation.

In the third embodiment, errors that can be expected in tracking the moving object Mo (a basket or a cart) include (A) when a shopper enters the store 31 through the entrance 31 and picks up the basket or the cart. When the moving object detection unit 3302 by the camera cannot detect the moving object Mo, (B) when the basket area tracking unit 3307 by the ceiling camera loses track of the moving object Mo, (C) two or more different movements When the IDs associated with the respective objects Mo are changed during the tracking, etc. are included (not limited to these, of course).

Depending on each error state, the system adopted in this embodiment takes various actions including the following examples.
(A) An image captured by the ceiling camera 310 and including a shopper who holds a basket or a cart as a subject is transmitted to the inspection terminal Q, and a detection of the moving object Mo is requested by the inspection. When the inspector detects the moving object Mo, a new moving object Mo corresponding to the basket or cart held by the shopper is defined, an ID is issued, and the basket area tracking unit 3307 by the ceiling camera is used. Start tracking by. When the eye examiner cannot detect the moving object Mo, the clerk is notified of that fact.
(B) When a basket or cart that is not associated with any ID registered in the location information management DB 132 is recognized, a captured image including a shopper who holds the basket or cart and an ID candidate to be associated. List is transmitted to the inspection terminal Q. The eye examiner associates the most appropriate ID with the basket or cart based on the past association information of the basket or cart and starts re-tracking by the basket area tracking unit 3307 with the ceiling camera. If the re-tracking cannot be started for some reason, the clerk is informed accordingly.
In addition, when an ID that loses the association with the basket or cart is detected, the ID and a list of imaged images of shoppers holding the basket or cart that is not associated with any ID are used for the visual inspection. It is transmitted to the terminal Q. The eye examiner attempts to associate the basket or cart with the ID based on the association information of the past basket or cart and the ID, and if the association is successful, the basket area tracking unit 3307 using the ceiling camera. Start re-tracking by. If the start of re-tracking fails for some reason, the clerk is informed accordingly.
(C) An image list of shoppers who hold their respective baskets and carts, and a list of IDs that should be associated with the respective baskets and carts are transmitted to the inspection terminal Q. The eye examiner tries to assign the ID to the basket or cart most appropriately based on the past association information of the basket or cart and the ID. When the IDs can be properly assigned to the basket and the cart, the re-tracking by the moving object area tracking unit 3305 by the ceiling camera is started. If the ID cannot be properly allocated to the basket or cart, the allocation is abandoned and the clerk is notified that the linking has been exchanged.

[Embodiment 4]
FIG. 33 is a diagram showing a layout example of a supermarket that adopts the product recognition system according to the fourth embodiment.
The information processing system of the fourth embodiment is a product recognition system applied to a store 40 such as a supermarket as shown in FIG. The information processing system according to the fourth embodiment is configured such that the merchandise can be automatically settled only by passing through the settlement gates 5-1 to 5-3 without being placed on the cash register.

At the entrance 40 of the store 40, a shopping cart, a shopping cart, etc. (without numbering) are placed. In front of the exit 42, the settlement gates 5-1 to 5-3 and the settlement register 6 operated by the clerk Mt are installed. Further, between the entrance 41 and the exit 42, a plurality of shelf racks 43 for displaying products are installed. In the shelf rack 43, a plurality of shelves are arranged in the vertical direction at a predetermined interval, and various products are displayed on each of the plurality of shelves. Hereinafter, the space between the shelves is also referred to as “inside the shelves”. A passage 44 is defined between the rack racks 43 facing each other in the horizontal direction.

The shopper enters the store through the entrance 41, picks up the shopping cart, pushes the shopping cart, or proceeds through the aisle 44 with his / her own bag brought. The shopper takes the product in the shelf, puts it in the basket, and proceeds through the aisle 44. When the shopper has taken all the desired products, the shopper proceeds to the settlement area 45 and makes the settlement.
In the passage 44, the settlement area 45, etc., the clerk Mt is looking around. The store clerk Mt owns the information terminal 9a. The information terminal 9a is a portable information processing terminal such as a smartphone, and has a function of displaying a state of the store, a function of displaying an error state occurring in the store, a remote control function, and the like.

In FIG. 33, not the inside of the store 40 but the outside of the store 40 or the backyard of the store 40 is drawn inside the cloud-shaped diagram. The server 1 (FIG. 7) is installed outside the store 40 or in the backyard of the store. Outside the store or in the backyard, the store clerk Mt can monitor the inside of the store 40 through the screen of a large monitor (not shown) or the screen of the information terminal 9b.
In addition, the shopping cart, the shopping cart, and the my bag are referred to as baskets. The passage 44 including the settlement gates 5-1 to 5-3 is a settlement area 45. In addition, baskets and the like and shoppers are referred to as a moving object Mo.

A plurality of ceiling cameras 310 are installed on the ceiling of the passage 44 between the entrance 41 and the exit 42. Further, a plurality of shelf cameras 311 are installed at a plurality of positions on each shelf in each shelf rack 43.
Note that in FIG. 33, the inside of the diagram drawn in a cloud shape does not show inside the store 40 but outside the store 40 or the backyard of the store 40. The server 1 (FIG. 7) is installed outside the store 40. In the backyard, the store clerk Mt monitors the inside of the store 30 through the screen of a large monitor (not shown) and the screen of the information terminal 9b.

FIG. 34 is a configuration diagram showing the configuration of a product recognition system as the fourth embodiment of the information processing system.
The product recognition system of the fourth embodiment has a configuration as shown in FIG.
The product recognition system includes a server 1, a sales floor device 3, and n (n is an arbitrary integer) settlement gates 5-1 to 5-n. The cashier terminal 2 adopted in the first and second embodiments is not adopted in the fourth embodiment.
The server 1, the sales floor device 3, and the settlement gates 5-1 to 5-n are connected to each other via a network N such as an Internet line.
Note that, for convenience of description, only one server 1 is depicted in FIG. 34, but in reality, there may be a plurality of servers.
Further, hereinafter, when it is not necessary to individually distinguish the settlement gates 5-1 to 5-n, they are collectively referred to as a “settlement gate 5”.

The server 1 executes each process in order to manage each operation of the sales floor device 3 and the settlement gate 5. The server 1 has the same hardware configuration as the server 1 of the first embodiment shown in FIG.

Therefore, the server 1 includes the CPU 101, the ROM 102, the RAM 103, the bus 104, the input / output interface 105, the output unit 106, the input unit 107, the storage unit 108, the communication unit 109, and the drive 110. I have it.

The sales floor device 3 is configured similarly to the hardware configuration of the third embodiment shown in FIG. 26, but does not include the illustrated basket camera 312.

Therefore, the sales floor device 3 includes a CPU 301, a ROM 302, a RAM 303, a bus 304, an input / output interface 305, a ceiling camera 310, a shelf camera 311, an information terminal 9, and a communication unit 315. .

However, in the error state in the fourth embodiment, when an abnormality occurs in the system processing or when the product cannot be specified in the shelf product recognition, an unspecified object or a moving object Mo associated with the trade-restricted product is This occurs under various circumstances, such as when the customer attempts to pass through the settlement gate 5 or when a shopper (moving object Mo) who has not made the settlement attempts to leave the store.

A settlement gate 5 is connected to such a sales floor device 3 via a network N.
The settlement gate 5 is divided into a settlement gate 5-1 that uses the settlement machine 5a, a settlement gate 5-2 that uses electronic money, and a settlement gate 5-3 that can be settled only by passing through.
The settlement gates 5-1 to 5-3 other than the manned settlement register 6 may be provided with an opening / closing member (not numbered) that is always closed.

Further, the settlement gate 5-1 using the settlement machine 5a is provided with a settlement button (not shown) to calculate the total amount of the purchased commodities, and the settlement machine 5a installed on the exit 42 side performs the settlement. The settlement machine 5a is installed closer to the exit 42 than the settlement gate 5-1. The settlement machine 5a is equipped with a settlement means capable of making payment by cash, credit card, electronic money, point payment, gift certificate / virtual currency, prepaid card, or the like.
In the settlement gate 5-1 using the settlement machine 5a, when the person who makes the settlement pushes down the settlement button, the pushing down triggers to read out the product information associated with the moving object area, which will be described later. The settlement amount is fixed, and it becomes possible to pass through the settlement gate 5-1. When the opening / closing member is provided, the opening / closing member opens. When the person who makes the payment makes the payment using the settlement machine 5a, the store can be closed.

The settlement gate 5-2, which uses electronic money or the like, performs settlement by holding electronic money or the like over the gate body. That is, a settlement gate using electronic money or the like includes a card reading unit (not shown), does not include a settlement button like the settlement gate 5-1 using the settlement machine 5a, and does not use the settlement machine 5a.
The electronic money and the like include not only IC cards that can be settled but also credit cards, so-called point cards, prepaid cards and other narrowly defined cards, as well as portable information terminals, but these will be described below as various cards.
When the person making the settlement enters the settlement area 45 of the settlement gate 5-2 and the various cards are read by the card reading unit, the product information associated with the moving object area, which will be described later, is read out, and the settlement and settlement are completed. , It becomes possible to pass through the settlement gate 5-2.

At the checkout gate 5-3, the shopper whose personal information has been acquired at the entrance 41, the passage 44, etc. only needs to pass through the checkout gate 5-3 to complete the checkout and settlement. That is, settlement and payment are completed without paying cash or reading a card.

At the manned checkout cash register 6, the clerk Mt individually inputs the price of the item for payment.
FIG. 35 is a block diagram showing the hardware configuration of the settlement gate 5 in the product recognition system of FIG. 34.
The settlement gate 5 includes a CPU 501, a ROM 502, a RAM 503, a bus 504, an input / output interface 505, an input unit 506, an output unit 507, a storage unit 508, and a communication unit 409.

The CPU 501, the ROM 502, the RAM 503, the bus 504, the input / output interface 505, the storage unit 508, and the communication unit 509 of the settlement gate 5 are configured similarly to those of the server 1.

The input unit 506 is a settlement button provided in the main body of the settlement gate 5-1 that uses the settlement machine 5a, and the settlement gate that uses electronic money and the settlement gate that can be settled only by passing through the settlement gate 5-2. Reference numeral 5-3 is an information reading unit for detecting information on various cards.

The output unit 507 outputs a signal for opening and closing the opening / closing member (without numbering) provided in the settlement gates 5-1 to 5-3. Further, the settlement gate 5-1 using the settlement machine 5a outputs the amount to be settled, the product name, etc. to the settlement machine 5a.

FIG. 36 is a functional block diagram showing an example of the functional configuration of the server 1 of FIG. 7, the sales floor device 3 of FIG. 26, and the settlement gate 5 of FIG. 35 described above.

The server 1 includes a CPU 101 similar to that of the third embodiment, a storage unit 108, an error display unit 151, and an error canceling unit 152.
The server 1 is configured similarly to the server 1 of the third embodiment.

FIG. 37 is a functional block diagram showing a detailed functional configuration example of the moving object tracking unit 330 provided in the sales floor device 3 in the third embodiment.
In the CPU 301 of the sales floor device 3, as shown in FIG. 37, a personal authentication unit 320, a moving object tracking unit 330, a position information management unit 340, a shelf product recognition unit 360, a basket product recognition unit 370, and trading. And a restricted product determination unit 380.
The personal authentication unit 320, the moving object tracking unit 330, the position information management unit 340, the basket product recognition unit 370, and the trade restricted product determination unit 380 are configured similarly to the third embodiment.

The personal authentication unit 320 includes the same personal information acquisition unit 321 as in the third embodiment. The personal authentication unit 320 acquires personal information from the personal information acquisition unit 321 or the DB management unit 141 of the server 1.

As shown in FIG. 37, the moving object tracking unit 330 includes a moving object finding unit 3302 using a ceiling camera, a moving object region defining unit 3304 using a ceiling camera, a moving object region tracking unit 3305 using a ceiling camera, and a grouping unit 3308. A blood relationship determination unit 3309, a moving object area transfer recognition unit 3311 by a ceiling camera, an object recognition unit 3312 transferred by a ceiling camera, and a product specification unit 3313 transferred by a ceiling camera. There is.
The moving object finding unit 3302 by the ceiling camera finds the moving object Mo (shopper, basket, cart, etc.) using a state space model (Bayesian filter, etc.) based on the captured image taken by the ceiling camera 310.
The moving object area definition unit 3304 using the ceiling camera defines the area of the moving object Mo found by the moving object finding unit 3302 using the ceiling camera as the moving object area. The moving object area definition unit 3304 using the ceiling camera defines a moving object area by continuously finding changed areas around the moving object Mo. That is, the moving object area defining unit 3304 using the ceiling camera defines the moving object Mo that has been found, the moving object Mo, and a certain range of the periphery of the moving object Mo as a moving object area.

Then, by applying a skeleton model when the shopper is viewed from above, an approximate pose is estimated, and the moving object Mo is clarified by comparing it with the moving object region in the actually obtained image. May be defined.
Here, when an area centered on a person is called a person area, the person area is a subordinate concept of the moving object area.
Also, when a region centering on baskets is called a basket region, the basket region is a subordinate concept of the moving object region.
When the area centering on the cart is called a cart area, the cart area is a subordinate concept of the moving object area.

The moving object area tracking unit 3305 using the ceiling camera tracks the movement of the moving object Mo. For example, the moving object region tracking unit 3305 using the ceiling camera also tracks the movement of the moving object Mo by collecting the characteristic data (color, shape, etc.) of the moving object Mo.
Alternatively, the moving object area tracking unit 3305 using the ceiling camera tracks the movement of the moving object Mo by using an object tracking technique in an image such as a Bayesian filter, a fast Fourier transform, and TLD (Tracking-Learning-Detection).

When a plurality of people, such as family members and friends, visit the grouping unit 3308, the grouping unit 3308 groups the plurality of people.
In addition, the grouping unit 3308 may group a plurality of people by using information such as a sense of distance (overlapping, sticking, etc.) between the moving objects Mo, a moving direction (vector), and the like.
The grouping unit 3308 may associate the human area with the basket area / cart area.
With the function of the grouping unit 3308 in this manner, one person can make a settlement at the settlement gate 5.

The blood relationship determination unit 3309 uses a face authentication method to distinguish the parent-child relationship, sibling relationship, and the like. The blood relationship determination unit assists the function of the grouping unit. The blood relationship determination unit may determine the degree of face similarity using a deep learning face recognition method and estimate the blood relationship.

When goods are transferred from the shopper (moving object Mo) to the shopper (moving object Mo), the moving object area transfer recognizing unit 3311 by the ceiling camera transfers the object from the moving object Mo to the moving object Mo. That is, the moving object Mo passed / passed is identified, and the product list associated with each moving object Mo is read.

The moving object area transfer recognizing unit 3311 by the ceiling camera recognizes that a product is transferred from the shopper (moving object Mo) to the shopper (moving object Mo), and is transferred to the transferred moving object Mo. The moving object Mo is identified and the product list associated with each moving object Mo is read. The object recognition unit 3312 delivered by the ceiling camera then defines the area of the object from the captured image at the time when the delivery is recognized.
Further, it is determined which of the products in the product list associated with the moving object Mo that has been transferred from the image after the object area is defined, the object transferred by the product specifying unit transferred by the ceiling camera. Each moving object Mo identified by the moving object area passing / recognizing unit 3311 by the ceiling camera is associated with the item for sale identified, and the list of each item for sale is updated.
The moving object area transfer recognizing unit 3111 using the ceiling camera may analyze the movement of the shopper (moving object Mo) by using an object recognition method such as deep learning, and may recognize the transfer. May recognize the hand inside the hand, and the recognition of passing may recognize the overlap between the human areas (which may include the hand).
The moving object area transfer recognizing unit 3311 by the ceiling camera may be realized by a shelf camera or the like capable of capturing a wide range instead of the ceiling camera.

After that, the object recognition unit 3312 delivered by the ceiling camera defines the area of the object from the captured image at the time when the delivery is recognized.

Further, the product identification unit 3313 delivered by the ceiling camera associates the object recognized by the object recognition unit 3312 delivered by the ceiling camera with the read moving object Mo (person who has delivered). It identifies which product in the product list, and links each moving object Mo specified by the moving object area passing / recognizing unit 3311 by the ceiling camera with the product specified by the selling camera specified by the ceiling camera 3313. Then, the list of products of each moving object Mo is updated.

The object recognition unit 3312 delivered by the ceiling camera may make the ceiling camera a zoomable camera, and may zoom up a portion where the delivery is estimated to be performed to define the area of the object.
The moving object area transfer recognizing unit 3311 by the ceiling camera, the object recognizing unit 3312 transferred by the ceiling camera, and the product specifying unit 3313 transferred by the ceiling camera are shelf cameras capable of capturing a wide range instead of the ceiling camera. And the like.

Similar to the second embodiment, the position information management unit 340 includes an inter-camera information transfer unit 341, a position definition unit 342 of each camera, and a moving object display unit 343, as shown in FIG.

FIG. 38 is a functional block diagram showing a detailed functional configuration example of the shelf product recognition unit 360 provided in the sales floor device 3 according to the fourth embodiment.
The shelf product recognition unit 360 includes an object recognition unit 3602 including a shelf camera, a product specification unit 3603 including a shelf camera, a moving object and product association unit 3604, and a product list management unit 3606 associated with the moving object, which are illustrated in FIG. 38. An object entry / exit detection unit 3608 by a shelf camera, a product non-identification determination unit 3609, a label recognition unit 3610, and a discount sticker recognition unit 3611 are provided.

Among these, the shelf camera product identification unit 3603 identifies which product is the object in the shelf recognized by the shelf camera object recognition unit 3602. The shelf camera product identification unit 3603 lists up product candidates by image processing methods such as specific object recognition, general object recognition, and deep learning. The listed product candidates are called “product candidate list S”. After that, the product specification unit 3603 using the shelf camera specifies the product with high accuracy by performing the verification function.

The verification function lists up the "commodity candidate list P" by an algorithm different from the method of listing up the product candidates described above. The results of the product candidate lists S and P are matched with each other, and a product is specified when the result exceeds a predetermined threshold.
The method of listing the “commodity candidate list” may be realized by, for example, a method of matching image information of an object obtained from an object whose existence is recognized with image information held in the product DB 131 or the memory. . That is, when the feature information of both images matches (exceeds a threshold value), the object whose existence is recognized by the object recognition unit 3602 by the shelf camera is a product registered in the product DB 131, and thus the product identification by the shelf camera is performed. The section 3603 specifies that the product is registered in the product DB 131. A product candidate is created by deep learning, and then the verification function is exerted to specify the product with high accuracy.

The shelf camera-based product specification unit 3603 does not specify the product in one frame of the captured image captured by the shelf camera 311, but also uses the captured image captured by the ceiling camera 310 to identify the product over a plurality of captured images. May be. At that time, the product specifying unit 3603 using the shelf camera gives a percentage to the product candidate, adds the percentage based on information such as purchase history, time, place, and person's preference, and exceeds a certain threshold. Specify the product when

In the present embodiment, there are a specific object inspection mode in which the inspection terminal Q performs inspection for a specific object and an all-object inspection mode in which inspection is performed by the inspection terminal Q for all objects. It can be set.
When the specific object visual inspection mode is set, the product identification unit 3603 by the shelf camera visually inspects the object captured image of the object that is not specified as the product and the related information via the eye inspection result acquisition unit 301a. To the terminal Q for use. On the other hand, when the all-object visual inspection mode is set, the product identification unit 3603 by the shelf camera determines the product identification result and the product captured image of the identified product, and the object captured image of the object that is not identified as the product and Related information and the like are transmitted to the inspection terminal Q via the inspection result acquisition unit 301a.
Then, the product identification unit 3603 using the shelf camera identifies the product based on the eye inspection result acquired by the eye inspection result acquisition unit 301a from the eye inspection terminal Q. Specifically, for the product identified by the product identification unit 3603 by the shelf camera, when the identification result is approved or corrected by the eye inspection result, the product identification unit 3603 by the shelf camera follows the contents indicated by the inspection result. , Specify the product. Further, for an object for which the product specification unit 3603 by the shelf camera determines that the product is unspecified, the product specification unit 3603 by the shelf camera specifies the object as the product indicated by the visual inspection result. When the product identification unit 3603 using the shelf camera requests the product identification by visual inspection, the request may be made after the product identification complement by the basket product label recognition unit 376 is completed. In this case, as a result of the supplement of the product identification by the label recognition unit 3610, the products whose products are identified can be excluded, and the products can be identified by visual inspection.
Note that the other functional units in the shelf product recognition unit 360 are the same as those in the second embodiment.

In the CPU 501 of the settlement gate 5, as shown in FIG. 35, a settlement unit 535, an input control unit 536, and an output control unit 537 function.

The settlement unit 535 receives product information (list of products) associated with one or more moving object regions from the server 1 via the communication unit 509 when the continuously tracked moving object region enters the settlement area 45. , Settle the payment amount and the item to be settled.
The input control unit 536 inputs a signal from the input unit 506 such as a payment button and an information reading unit provided in the gate body.
The output control unit 437 displays the settlement amount on the output unit 507, outputs information to the settlement machine 5a, and opens / closes the opening / closing member.

In addition, the settlement unit 535, based on the product information (list of products) associated with one or more moving objects Mo when the moving object Mo that has been continuously tracked, that is, a person who makes a payment enters the settlement area 45. , Settle the payment amount and the item to be settled.
Further, for example, when the father makes a payment with a wallet, the product information (a list of products) associated with the companion mother and the child may be grouped so that the father can make the payment.

The settlement gate 5-1 using the settlement machine 5a is a device for calculating and summing the total amount of goods and has a settlement button (not shown). The settlement machine 5a is installed closer to the exit 42 than the settlement gate 5. The settlement machine 5a is equipped with a settlement means capable of paying with cash, point payment, gift certificate, cash voucher, or the like.

In the settlement gate 5-1, when the person who makes the settlement pushes down the settlement button, the depression causes the merchandise information associated with the moving object Mo to be read out and the settlement amount to be settled. -1 can be passed. Then, when the person who makes the payment makes the payment using the checkout machine, the store can be closed.

However, for example, when the settlement object product is an age-restricted product and there is an error state that the age cannot be confirmed, the information terminal 9a, the information terminal 9b and the error display unit 151 are notified of the error state. . At that time, the settlement gate 5-1 maintains the passage-impossible state by keeping the opening / closing member closed. The settlement gate 5-1 may present the fact that there is an error state by means of sound or light. When the store clerk Mt is in a state in which the restriction can be released, such as when confirming the age, the limit is released by the operation of the error release unit 152 by the operation of the store clerk Mt, and the settlement gate 5-1 can be passed. The restriction can be released by remote control.
Further, even when the moving object Mo is associated with an object (unspecified object) that cannot be identified as which product, the flow is the same as that for the age-restricted product. The unspecified object may be dealt with by the clerk Mt at the manned cash register 6 to specify the product.

Further, the settlement gate 5-2 using electronic money is provided with a card reading unit (not shown), is not provided with a settlement button like the settlement gate 5-1, and does not use the settlement machine 5a. The card reading unit is adapted to handle any of credit cards, debit cards, electronic money, prepaid cards, and the like.

When the person making the payment enters the payment area 45 of the payment gate 5-2 and the card is read by the card reading unit, the product information associated with the moving object Mo is read, and the payment and payment are completed. It becomes possible to pass through the settlement gate 5-2.
When the settlement object product is a trade-restricted product, or when an unspecified object is tied to the moving object Mo, the same operation as that of the settlement gate 5-1 is performed.

In addition, in the settlement gate 5-3, the moving object Mo that has been continuously tracked needs to be personally authenticated and the payment information needs to be specified.
When the person making the settlement enters the settlement area 45 of the settlement gate 5-3, the settlement gate 5-3 is automatically settled and can pass through the settlement gate 5-3 unless there is an error.
When the settlement object product is a trade-restricted product, or when an unspecified object is tied to the moving object Mo, the same operation as that of the settlement gate 5-1 is performed.

The product recognition system according to the fourth embodiment may include a remote control unit (not shown) in addition to the functions described above.

The trade-restricted product determination unit 380 determines whether or not the specified product corresponds to the trade-restricted product. That is, the trade-restricted product determination unit 380 determines from the DB information that the identified product is a trade-restricted product due to the age restriction of alcoholic beverages and tobacco. Further, the trade-restricted product determination unit 380 uses image recognition or the like to determine that the product is a trade-restricted product due to expiration or expiration of the expiration date. In addition, the trade-restricted product determination unit 380 associates the product with the personal information obtained by the personal authentication, and determines that the product is a trade-restricted product other than allergies to shoppers and products other than halal foods.
In addition, when the sale-restricted product determination unit 380 determines that the product is a sale-restricted product, it indicates that it is a sale-restricted product. Further, the trading restricted product determination unit 380 may restrict trading based on the personal information obtained from the personal authentication. Further, the trade-restricted product determination unit 380 may estimate the age and gender from the face and hand recognition, and restrict the trade.

When the remote control unit receives a notification of an error state such as a system processing abnormality, an unspecified product, or a trading restriction, the remote control unit cancels the error state by remote control. The ceiling camera 310, the shelf camera 311, and the checkout gate 5 that have detected the error state send information indicating that there is an error state and a captured image of the state via the network N to the information terminal 9a for remote operation inside or outside the store, 9b and the server 1 are notified. By operating these devices, the error condition can be eliminated.
Of course, the error state may be released at a manned cash register in the store, or the store clerk Mt or the like may go to a place where the error state has occurred and release the error state.

Next, a product settlement method in the product recognition system according to the fourth embodiment will be described with reference to FIG. 39.
39 and 40 are flowcharts illustrating the automatic settlement process executed by the server 1, the sales floor device 3, and the settlement gate 5 in FIG. 36.

In step S401, the shopper (moving object) enters the shop through the entrance 41 of the store 40 (FIG. 33), and the ceiling camera 310 installed near the entrance 41 starts imaging the shopper. When the shopper holds a basket or a card and proceeds through the aisle 44, the ceiling camera 310 at the back starts capturing an image of the shopper. In this way, as in the third embodiment, the plurality of ceiling cameras 310 constantly capture images of the entire store 30 including shoppers, baskets, and carts. Note that the personal authentication unit 320 may perform personal authentication of the shopper and acquire the personal information of the shopper before step S401.

In step S402, the moving object finding unit 3302 using the ceiling camera finds the moving object Mo and assigns an individual ID. The ID is continuously used until a specific timing such as leaving the store or completing settlement. The personal authentication unit 320 may perform personal authentication of the shopper at this timing to acquire the personal information of the shopper. Furthermore, the grouping unit 3308 may group a plurality of moving objects Mo as one group, and the blood relationship determination unit 3309 determines the blood relationship of a plurality of shoppers (moving objects Mo) and complements the grouping unit 3308. You may

In step S403, the moving object area defining unit 3304 with the ceiling camera defines a predetermined area including the moving object Mo found by the moving object finding unit 3302 with the ceiling camera. Further, when the moving object Mo moves within the range imaged by the ceiling camera 310, the position of the area of the moving object Mo after the movement is defined again. The position information is managed in the position information management DB 132, a memory or the like in association with the ID of the moving object Mo, and is updated for each area definition. This defined position is also recognized as a position imaged by another ceiling camera 310.

In step S404, the position at which the moving object Mo moves in the captured image captured by the ceiling camera 310 having the moving object area tracking unit 3305 by the ceiling camera is estimated. Further, the moving object region definition unit 334 defines the region of the moving object Mo with respect to the position estimated to have moved, and updates the position information of the moving object Mo stored in the position information management DB 132 or the memory.

In step S405, the object entry / exit detection unit 3608 by the shelf camera detects entry / exit of an object into / from the shelf, as in the third embodiment.

In step S406, similarly to the third embodiment, the object entry / exit detection unit 3608 triggered by the shelf camera is used as a trigger, and the object recognition unit 3602 by the shelf camera causes images before and after the image in which the object is taken or the image in which the object is placed. And the image area that is the target of product identification is defined.
Further, in step S406, the object acquisition recognition unit using the ceiling camera may recognize that the moving object Mo has acquired an object from the inside of the shelf or the like.

In step S407, as in the third embodiment, the product identification unit 3603 by the shelf camera identifies which product the object is. When the specified product is a trade-restricted product, the trade-restricted product is linked to the moving object Mo. Further, even when it is not possible to specify which product the object is, the product-unspecified object is linked to the moving object Mo. The information terminal 9a, the information terminal 9b, and the server 1 may be informed that the error state is due to unspecified product, and the clerk Mt who recognizes the error state may cancel the error state by remote control (obviously, the clerk Mt. May go directly to release the error condition).
Further, in step S407, the label recognition unit may recognize the label associated with the specified product.
In addition, in step S407, the discount label recognition unit may recognize the attached discount label according to the identified product.
In addition, at the time of step S407, the product specifying unit using the ceiling camera may specify which product is the object region acquired by the object acquisition recognizing unit using the ceiling camera.

In step S408, the shelf-camera product specifying unit 3603 determines whether the specific object visual inspection mode or the all object visual inspection mode is set.
If the specific object visual inspection mode is set, the process proceeds to step S409.
If the all-object visual inspection mode is set, the process proceeds to step S410.
In step S410, the visual inspection result acquisition unit 301a requests a visual inspection (specification of a product) at the visual inspection terminal Q for the target object, and acquires the visual inspection result.
After step S409, the process proceeds to step S411.
In step S410, the product identification unit 3603 by the shelf camera determines whether it is not possible to identify which product the added one object is. If the product specification unit 3603 by the shelf camera cannot specify which product (YES in step S410), the process proceeds to step S409.

In step S410, if the product specification unit 3603 by the shelf camera can specify which product (NO in step S410), the product specification unit 3603 by the shelf camera determines the product name stored in the storage unit of the server 1 or A product is specified including information such as a price and a product that is restricted for sale. As a result, the process proceeds to step S411. The identified product information may be output to the output unit 507.
Further, in step S410, when the product specification unit 3603 by the shelf camera cannot specify which product, the screen display of the display unit (not shown) provided in the sales floor device 3, the voice guidance by the speaker (not shown), etc. Then, the purchaser may be requested to retrieve the product from the shelf. Further, the purchaser and the eye examiner may talk with each other via a microphone (not shown) provided in the sales floor device 3.

In step S411, the trade-restricted product determination unit 380 determines whether or not the product identified by the product identification unit 3603 using the shelf camera is a product for which age confirmation is required.

If it is determined in step S411 that the product identified by the product identification unit 3603 using the shelf camera is a product that requires age confirmation, that is, if YES, the process proceeds to step S412.
If it is determined in step S411 that the product specified by the product specifying unit 3603 using the shelf camera is not a product for which age confirmation is required, that is, if NO is determined, the process proceeds to step S417.
In step S412, the output control unit 437 causes the output unit 507 of the settlement gate 5 to display a screen for age confirmation. However, if the personal information of the shopper is acquired and it is not necessary to confirm the age here, step S412 is skipped and the process proceeds to step S416. Here, when there is a problem in the age verification of the shopper, the purchaser and the eye checker may make a call via a speaker (not shown) and a microphone (not shown) provided in the checkout gate 5.
In step S413, the visual inspection result acquisition unit 301a requests a visual inspection (determination of a trade-restricted commercial product) at the visual inspection terminal Q for the target product, and acquires the visual inspection result.
In step S414, the trading restriction product determination unit 380 determines whether or not an instruction to cancel the trading restriction has been received. The sale restriction of the sale-restricted product is released by either the store clerk on the spot in response to the result of the determination or the result of the visual inspection is adopted as a system. Here, a case where the result of the visual inspection is adopted as the system will be described.

If it is determined in step S414 that the instruction to cancel the trading restriction has not been received, the process proceeds to step S415.
If it is determined in step S414 that the cancel instruction for canceling the trading restriction has been accepted, that is, if YES, the process proceeds to step S416.
In step S415, the visual inspection result transmission unit 412 of the visual inspection terminal Q transmits a warning that the trading restriction has not been released even by the visual inspection result. By receiving this warning, for example, the output control unit 537 of the settlement gate 5 presents, through the output unit 507, a warning that the trading restriction has not been canceled even by the result of the visual inspection. After step S416, the process ends and the settlement is stopped.
In step S416, the trade restriction product determination unit 380 cancels the trade restriction.
In this way, when step S416 is completed, or when it is determined in step S411 that the product is not age-restricted (NO is determined), the process proceeds to step S417.

In step S417, the moving object and the product association unit 3604 associate the moving object Mo with the specified product.

In step S418, the product list management unit 3606 associated with the moving object continues to manage the product list associated with the person until settlement.
In step S419, the settlement gate 5 performs settlement or settlement based on the product information associated with the moving object Mo.
Note that the error display unit 151, the information terminal 9a, the information terminal 9b, or the output unit 507 of the settlement gate 5 may notify the clerk or the like of any error state during the process up to step S419 or at the time of step S419. .
Further, in step S410, when the trade-restricted product is associated with the moving object Mo, the error state may be displayed by keeping the gate closed without opening it.

In the fourth embodiment, as an error assumed in tracking the moving object Mo (shopper), (A) when the shopper enters the store 41 from the entrance 41, the moving object detection unit 3302 by the ceiling camera moves. When the object cannot be detected, (B) when the moving object area tracking unit 3305 with the ceiling camera loses track of the moving object Mo being tracked, (C) it is associated with each of two or more different moving objects Mo. If the IDs are swapped during tracking, etc. are included (not limited to these, of course).

Depending on each error state, the system adopted in this embodiment takes various actions including the following examples.
(A) The image including the shopper captured by the ceiling camera 310 as an object is transmitted to the inspection terminal Q, and the detection of the shopper by the inspection is requested. When the eye checker detects a shopper, a new moving object Mo is defined, an ID is issued, and tracking by the moving object area tracking unit 3305 by the ceiling camera is started. If the eye checker cannot detect the shopper, the fact is notified to the clerk.
(B) When a shopper who is not associated with any ID registered in the positional information management DB 132 is recognized, the imaged image of the shopper and a list of ID candidates to be associated are displayed in the terminal Q for inspection. Sent to. The eye examiner associates the most appropriate ID with the shopper based on the past association information of the shopper and the ID, and starts re-tracking by the moving object area tracking unit 3305 using the ceiling camera. If the re-tracking cannot be started for some reason, the clerk is informed accordingly.
In addition, when an ID that has lost connection with the shopper is detected, the ID and a list of imaged images of shoppers that are not associated with any ID are transmitted to the inspection terminal Q. The eye examiner attempts to associate the shopper with the ID based on the association information of the shopper and the ID in the past, and when the association is successful, the moving object area tracking unit 3305 by the ceiling camera uses Start retracking. If the start of re-tracking fails for some reason, the clerk is informed accordingly.
(C) The image list of each shopper and the list of IDs that should be associated with each shopper are transmitted to the inspection terminal Q. The eye examiner most appropriately attempts to assign the ID to the shopper based on the past information of the shopper and the ID. When the IDs can be appropriately assigned to the shoppers, the re-tracking by the moving object area tracking unit 3305 by the ceiling camera is started. If the IDs cannot be properly allocated to the shoppers, the allocation is abandoned and the clerk is informed that the ties have been exchanged.

[Other embodiments]
The present invention is not limited to the above-described embodiments, but includes modifications and improvements as long as the object of the present invention can be achieved.

For example, the information processing system uses a method in which an eye examiner visually confirms an image including an object as a subject, and acquires a result of attempting to specify the object as a product. And a settlement means for performing settlement processing for the product identified based on the result of the product identification means.
As a result, when the purchaser purchases a product, it is possible to automate the settlement of the price of the product while maintaining the same accuracy of product identification as that of a manned cash register.

Further, for example, the information processing system may include a shelf inventory management function. The shelf inventory management function includes a salesclerk discrimination means and an inventory information update means.
The clerk discrimination means discriminates between the clerk and the shopper. For example, the salesclerk discrimination unit attaches a physical marker that can be used to identify the salesclerk to items such as hats and clothes worn by the salesclerk. The clerk is identified by the ceiling camera and the shelf camera capturing images of this physical marker. This clerk discrimination means can be effectively used especially in the third and fourth embodiments.

Also, the salesclerk determination means may include a button provided in a predetermined area such as a backyard of the store. What is this salesclerk discrimination means? When a button is pushed by a clerk existing in a predetermined area, the clerk is assigned an ID. This clerk discrimination means can be effectively used especially in the fourth embodiment. In addition, in the salesclerk determination unit, the ceiling camera continues to track the area of the person recognized as the salesclerk.

Then, the inventory information updating means makes full use of the object recognition unit and the product identification unit when the store clerk replenishes the shelf with products (addition) or takes out or discards the products (subtraction) to store the inventory information of the shelf. Update in the product DB of the server.
The inventory information updating means updates the inventory information of the shelf where the product is present and the inventory information of the entire store when the product is purchased.
The inventory information updating means manages the inventory information of the shelves and the inventory information of the entire store, and when the inventory number falls below a threshold value, notifies the remote control unit or automatically places an order.

Also, the present invention may include a shopper attribute estimation function. The shopper's attribute estimation function estimates attributes of the shopper, such as approximate age, age, and gender, from face recognition, for example. The attribute estimation function of the shopper is provided in the cashier terminal in the first embodiment, and is provided in the sales floor device in the second to fourth embodiments. In the second embodiment, the cashier terminal may be provided. The present invention also links purchase information and shopper attribute information to the POS system. As the purchase information, information such as a product name and an amount of money is converted into data, and the information is linked to the POS system after the settlement is completed. As the attribute information of the shopper, the information obtained by the attribute estimation function of the shopper described above is linked to the POS system.

The shopper attribute estimation function may be linked with, for example, the trade restricted product determination unit. As a result of the attribute estimation, the shopper attribute estimation function may control the trade restriction product determination unit not to perform age confirmation if the shopper is in the thirties or older who does not obviously need age confirmation. Good.

Here, the problems relating to the above-described first to fourth embodiments and other embodiments will be supplemented.
Image recognition using deep learning is promising as a product recognition method in the product recognition system in each embodiment. However, deep learning requires re-learning when a product cannot be identified or when it is erroneously recognized, and in that case, it is necessary to manually give an image of the correct product, which is a great deal of effort. It takes.

As a solution to the above problems, the product recognition system of the present invention is
When purchasing a product, a function that corrects the product recognition by visually checking by an inspector or scanning a barcode,
A function to link and record the correct product image and necessary information (modified product number, specific code, etc.) when product recognition is corrected,
Based on the correct product image and necessary information, the function to automatically perform learning of deep learning,
A function to automatically deploy learning results to the product recognition system,
By implementing, the series of deep learning re-learning processes can be automated.
Here, the deployment of the learning result refers to a process of updating, for example, a deep learning model or the like of the product recognition system based on the learning result so that it can be used for product recognition.
The re-learning process may be configured to be triggered by a predetermined condition such as product identification failure, or may be configured to be triggered by an explicit operation by a person in charge, for example, pressing a button. Good.

According to the deep learning re-learning process as described above, it is possible to simplify the complicated work involved in the deep learning re-learning and reduce manpower, and to provide a product recognition system in which the product recognition accuracy is continuously improved.

Further, in each of the above-described first to fourth embodiments and other embodiments, as an image recognition method applied when an object is specified as a product, for example, object recognition using AI (Artificial Intelligence), features It is possible to adopt object recognition using points / features, shape recognition of product logo, character recognition (character shape recognition using AI, character recognition using OCR (Optical Character Recognition), etc.). is there. In addition to the image recognition method, weight recognition (weight recognition of an object using a weight sensor), product recognition by scanning identification information such as a bar code or QR code (registered trademark) (a bar by an infrared sensor or image reading, etc.) It is possible to recognize an object as a product by combining one or more recognition methods of product recognition by scanning electronically recorded identification information such as a code or QR code (registered trademark)) or RFID. Is. In this case, it is possible to score the recognition result of the object by each recognition method and to estimate that the object is the product with the highest score.
Furthermore, the above-mentioned recognition methods such as weight recognition, product recognition by scanning identification information such as a bar code and QR code (registered trademark), and product recognition by reading electronically recorded identification information are combined with an image recognition method. Instead, it is possible to recognize the object as a product by combining one or more.

In each of the above-described embodiments, the visual inspection of the object or the product is performed by the visual inspection terminal Q, but the present invention is not limited to this. That is, the eye examination by the eye examiner may be performed by another device such as the cashier terminal 2, the settlement machine 4, and the information terminal 9.
In addition, in each of the above-described embodiments, one store or one cashier terminal 2 is provided with one eye inspection terminal Q, or one store or plurality of cash register terminals is provided with one eye inspection terminal Q. It is possible to prepare.

Also, the hybrid check system (eye check) in the above-described embodiment is not necessarily used only for product identification. That is, for example, the product includes (A) tobacco, liquor, etc. that cannot be purchased until a certain age is reached, (B) expired / expired product, and (C) allergic ingredients, so It can also be used to determine whether or not a product should not be ingested, a product other than (D) halal food, or a product with religious restrictions.

Further, in the present invention, the image processing by the above-described CPU may be processing by a GPU (Graphic Processing Unit).
Further, the illustrated hardware configuration and block diagram are merely examples for achieving the object of the present invention, and the present invention is not limited to the illustrated example.
The location of the functional block is not limited to that shown in the figure, and may be arbitrary. For example, one functional block may be configured by hardware alone, software alone, or a combination thereof. When the processing of each functional block is executed by software, the program forming the software is installed in a computer or the like from a network or a recording medium.

The computer may be a computer incorporated in dedicated hardware, or may be a smartphone or a personal computer.
Further, the processing of the trade-restricted product described in the first embodiment can be applied to the second to fourth embodiments. However, the trade-restricted product in the second embodiment is applied to the age-restricted product.

In the present specification, the steps for writing the program recorded on the recording medium are not limited to the processing performed in time series according to the order, but may be performed in parallel or individually even if the processing is not necessarily performed in time series. It also includes the processing executed in.
Further, in the present specification, the term “system” means an overall device configured by a plurality of devices, a plurality of means and the like.

In summary, the information processing system to which the present invention is applied can have various configurations in addition to the above-described first to fourth exemplary embodiments, as long as the following configuration is adopted.
That is, the information processing system
A first specifying means (for example, a visual inspection) that obtains the result of attempting to specify the object as a product by using the first method in which the eye examiner visually confirms the image including the object as a subject. A result acquisition unit 239 and a visual inspection terminal Q),
A settlement means (for example, a settlement section 237) for performing settlement processing for the product identified based on the result of the first identifying means,
Is provided.
As a result, when the purchaser purchases a product, it is possible to automate the settlement of the price of the product while maintaining the same accuracy of product identification as that of a manned cash register.
Furthermore, such an automatic settlement processing system makes it possible to complete the procedure for purchasing a product between the purchaser and the system, and lowers the hurdle for realizing an unmanned store.

A second specifying means (for example, an object recognizing unit 233) that attempts to specify the object as a product by using a second method other than the first method,
The settlement means may be configured to perform settlement processing on a product identified based on a result of at least one of the first identification means and the second identification means.

The second method may include at least a predetermined image recognition method (for example, deep learning).

For the object not specified as a product by the predetermined image recognition method,
By the first specifying unit and the second specifying unit, the product information of the object specified as a product using one or more methods not including the predetermined image recognition method and the image are linked,
Using the information including the linked product information and the image, re-learning the predetermined image recognition method,
The learning result obtained by the re-learning may be further provided with an image recognition re-learning unit which is provided in the second specifying unit.

The second method may include at least one of weight recognition, scanning of identification information, and reading of electronically recorded identification information.

Regarding the object not specified as the product by the second method, the product specified based on the result by the first specifying means may be the target of the payment processing by the payment means.

The product specified based on the result of each of the first specifying unit and the second specifying unit may be a target of the settlement process by the settlement unit.

1 ... Server, 2,2-1,2-n ... Cashier terminal, 3 ... Sales floor device, 4,4-1,4-n, 5a ... Adjustment machine, 5,5-1 , 5-2, 5-3, 5-n ... Settlement gate, 6 ... Settlement cash register, 9, 9a, 9b ... Information terminal 10, 20, 30, 40 ... Store, 11, 22 ... Doorway, 12, 25 ... Cashier counter, 13, 26 ... Manned register, 14, 23, 33, 43 ... Shelf rack, 15, 24, 34, 44 ... Aisle, 21 ... Gate, 31, 41 ... Entrance, 32, 42 ... Exit, 35, 45 ... Settlement area, 36 ... Cash register, 101, 201, 301, 401, 501 ... CPU, 102, 202, 302, 502 ... ROM, 103, 203, 303, 503 ... RAM, 104, 204 304, 504 ... Bus, 105, 205, 305, 505 ... Input / output interface, 106, 206, 406, 506 ... Output section, 107, 207, 407, 507 ... Input section, 108, 208, 508 ... Storage unit, 109, 213, 315, 409, 509 ... Communication unit, 110, 214 ... Drive, 120, 220 ... Removable media, 131 ... Product DB, 132 ... ..Position information management DB, 141 ... DB management unit, 150 ... Error determination unit, 151 ... Error display unit, 152 ... Error canceling unit, 209 ... Shading unit, 210 ... Presentation unit, 211 ... Cash register camera, 212 ... External camera, 221 ... Illumination unit, 228 ... Emission control unit, 229 ... Shading control unit, 230 ... Presentation control , 231, 320 ... Personal authentication section, 232 ... Image acquisition section, 233 ... Object recognition section, 234 ... Object quantity recognition section, 235 ... Product identification section, 236 ... Trading restriction Product determination unit, 237, 435, 535 ... Settlement unit, 238 ... Display control unit, 239, 301a ... Eye inspection result acquisition unit, 241 ... DB information holding unit, 270 ... Surrounding unit , 271 ... Top plate part, 272 ... Bottom plate part, 273 ... Side plate part, 275 ... Housing part, 276 ... Plate, 310 ... Ceiling camera, 311 ... Shelf camera Reference numeral 312 ... Basket camera, 321 ... Personal information acquisition unit, 330 ... Moving object tracking unit, 340 ... Position information management unit, 341 ... Inter-camera information transfer unit, 342 ... Camera position definition unit, 343 ... Moving object display unit, 350 ... Volume counting unit, 351 ... Volume number recognition unit, 352 ... Moving object and volume number associating unit, 353 ... Volume number unspecified determination unit, 354 ... Volume number managing unit that is associated with moving object Reference numeral 355 ... Passing recognition unit between moving object areas by the ceiling camera, 356 ... Recognizing number of books delivered by the ceiling camera, 360 ... Shelf product recognition unit, 370 ... Basket product recognition unit, 372・ ・ ・ Car entrance / exit detection unit by car camera 373 ・ ・ ・ Object recognition unit by car camera 374 ・ ・ ・ Product identification unit by car camera 375 ・ ・ ・ Product non-identification determination unit for car product 376 ・ ・ ・Basket product label recognition unit, 377 ... Basket product discount sticker recognition unit, 380 ... Trading restricted product determination unit, 390 ... Remote operation unit, 391, D ... Display unit, 411 ... Image display system Section 412 ... Eye inspection result sending section, 436,536 ... Input control section, 437,537 ... Output control section, 3302 ... Moveable object finding section by ceiling camera, 3303 ... Ceiling camera By the ceiling camera, the moving object area definition section by the ceiling camera, 3305 ... the moving object area tracking section by the ceiling camera, 3306 ... the basket area definition section by the ceiling camera, 3307 ... by the ceiling camera Basket region tracking unit, 3308 ... Grouping unit, 3309 ... Blood relationship determination unit, 3310 ... Cargo region passing / recognizing unit by ceiling camera, 3311 ... Moving object region passing / recognizing unit by ceiling camera, 3312 ... Object recognition unit delivered by ceiling camera, 3313 ... Product identification unit delivered by ceiling camera, 36 2 ... object recognition unit by shelf camera, 3603 ... product identification unit by shelf camera, 3604 ... moving object and product association unit, 3605 ... basket and product association unit, 3606 ... move Product list management unit associated with object, 3607 ... Product list management unit associated with basket, 3608 ... Object entry / exit detection unit by shelf camera, 3609 ... Product non-identification determination unit, 3610 ... Label recognition 3611 ... Discount sticker recognition unit, 3612 ... Basket entry / exit detection unit by shelf camera or ceiling camera, A ... Predetermined area, C ... Card reader unit, Mo ... Moving object, Mt. ..Sales staff, M ... Microphone, N ... Network, Q ... Visual inspection terminal, R ... Receipt part, S ... Speaker

Claims (9)

1. First identifying means for obtaining a result of attempting to identify the object as a product by using a first method in which an eye examiner visually confirms an image including the object as a subject;
   Settlement means for performing settlement processing for the product identified based on the result of the first identifying means,
   An information processing system comprising:
2. Further comprising second specifying means for attempting to specify the object as a product by using a second method other than the first method,
   The settlement means performs a settlement process on a product identified based on a result of at least one of the first identification means and the second identification means.
   The information processing system according to claim 1.
3. The second method includes at least a predetermined image recognition method,
   The information processing system according to claim 2.
4. For the object not specified as a product by the predetermined image recognition method,
   By the first specifying unit and the second specifying unit, the product information of the object specified as a product by using one or more methods not including the predetermined image recognition method and the image are linked,
   Using the information including the linked product information and the image, re-learning the predetermined image recognition method,
   Image recognition re-learning means for allocating the learning result obtained by the re-learning to the second specifying means;
   The information processing system according to claim 3, further comprising:
5. The second method includes at least one of weight recognition, scanning of identification information, and reading of electronically recorded identification information.
   The information processing system according to any one of claims 2 to 4.
6. For the object not specified as the product by the second method, the product specified based on the result of the first specifying means is the target of the payment processing by the payment means.
   The information processing system according to any one of claims 2 to 5.
7. The product specified based on the result of each of the first specifying unit and the second specifying unit is an object of the settlement processing by the settlement unit.
   The information processing system according to any one of claims 2 to 5.
8. An information processing method executed by an information processing apparatus,
   An identifying step of acquiring the result of attempting to identify the object as a product by using a method in which an eye checker visually confirms the image including the object as a subject,
   For a product identified based on the result of the process of the specific step, a payment step of performing a payment process,
   Information processing method including.
9. On the computer,
   An identifying step of acquiring the result of attempting to identify the object as a product by using a method in which an eye checker visually confirms the image including the object as a subject,
   For a product identified based on the result of the process of the specific step, a payment step of performing a payment process,
   A program that executes control processing including.

Images