JP6653813B1 - Information processing system - Google Patents

Abstract

An object of the present invention is to make it possible to automate the settlement of the price of a product when the purchaser purchases the product, and to increase the accuracy of specifying the product. An object recognition unit recognizes an object from an image including the object in a subject. The product specifying unit 235 specifies the object as a product using a technique such as image recognition. The inspection result acquisition unit 239 transmits the image to the inspection terminal Q, and acquires a specific result of the product by inspection transmitted from the inspection terminal Q. The settlement unit 237 performs a settlement process on the product specified based on the results of the product specification unit 235 and the inspection result acquisition unit 239. [Selection diagram] FIG.

Description

  The present invention relates to an information processing system.

  Conventionally, at stores such as convenience stores, supermarkets and shopping centers, and various mass retailers, buyers take products from product shelves, put them in shopping carts and shopping carts, and carry them to cash registers located near the exit of the store, The price of the product is settled by a self-cash register (for example, see Patent Document 1).

JP 2001-76261A

However, when checking out a product in a shopping cart or shopping cart at a cash register, each bar code of the product is read even at a self-checkout, so it is not possible to eliminate the queue waiting for the cash register. No, the buyer has to wait for the cash register for a long time.
In addition, when a purchaser purchases a product to be purchased at a store, if there are many other purchasers, the purchaser may give up shopping.
Therefore, in consideration of the above-described circumstances, when a purchaser purchases a product displayed in a store, it is possible to automate the payment of the payment for the product and reduce the time required for payment for the product. The system is required.
Furthermore, in shops of the related art, shoplifting and cashiers have become a problem of shoplifting and other frauds, and a system that can prevent such frauds is also required.

  The present invention has been made in view of such a situation, and aims to improve the accuracy of specifying a product while enabling the purchaser to purchase the product to automate the payment of the price of the product. I do.

In order to achieve the above object, an information processing system according to one embodiment of the present invention includes:
A first specifying unit that obtains a result of attempting to specify the object as a product by using a first method in which an examiner visually checks an image including the object in the subject;
A settlement unit that performs a settlement process for a product specified based on a result of the first specifying unit;
Is provided.

  Advantageous Effects of Invention According to the present invention, it is possible to provide an information processing system that enables a purchaser to purchase a product to automate the payment of the price of the product and increase the accuracy of specifying the product.

It is the figure which made the list the main point of Embodiment 1 thru / or Embodiment 4 of the information processing system concerning the present invention. FIG. 2 is a schematic diagram illustrating an outline of a system flow in the first and second embodiments. FIG. 14 is a schematic diagram illustrating an outline of a system flow in Embodiments 3 and 4. FIG. 2 is a diagram illustrating a layout example of a convenience store that employs the product recognition system according to the first embodiment. FIG. 2 is a schematic perspective view illustrating a configuration example of an appearance of a cashier terminal employed in the first embodiment. 1 is a configuration diagram illustrating a configuration of a product recognition system as a first embodiment of an information processing system according to the present invention. FIG. 7 is a block diagram illustrating a hardware configuration of a server in the product recognition system of FIG. 6. FIG. 7 is a configuration diagram illustrating a hardware configuration of a cashier terminal in the product recognition system of FIG. 6. FIG. 9 is a functional block diagram illustrating an example of a functional configuration of the server in FIG. 7 and the cashier terminal in FIG. 8. 4 shows an example of an imaging screen of an object placed on a cashier terminal. FIG. 9 is a diagram illustrating an example of a truth table for calculating the number of objects placed on a cashier terminal. 10 is a flowchart illustrating an automatic settlement process executed by the server and the cashier terminal in FIG. 9. 10 is a flowchart illustrating a process of a trade-restricted product in the automatic payment process performed by the server and the cashier terminal in FIG. It is a figure showing the example of a layout of the bookstore which adopts the goods recognition system in Embodiment 2. FIG. 11 is a schematic perspective view showing an example in which a book is automatically settled by a cashier terminal employed in a second embodiment. It is a lineblock diagram showing the composition of the goods recognition system as Embodiment 2 of the information processing system of the present invention. FIG. 17 is a block diagram illustrating a hardware configuration of a sales floor device in the product recognition system of FIG. 16. FIG. 18 is a functional block diagram illustrating an example of a functional configuration of the server in FIG. 7, the cashier terminal in FIG. 8, and the sales floor device in FIG. 17. FIG. 19 is a functional block diagram illustrating a detailed functional configuration example of a moving object tracking unit provided in the counter in FIG. 18. FIG. 19 is a functional block diagram illustrating a detailed functional configuration example of a position information management unit provided in the counter in FIG. 18. FIG. 19 is a functional block diagram illustrating a detailed functional configuration example of a book count unit provided in the sales floor apparatus of FIG. 18. It is a flowchart explaining the automatic payment process which the server of FIG. 18, a cash register terminal, and a sales floor apparatus perform. It is a flowchart at the time of verifying the number-of-volume information of goods and the number of books to be settled in step S210 of FIG. It is a figure showing the example of a layout of the supermarket which adopts the goods recognition system in Embodiment 3. It is a block diagram which shows the structure of the goods recognition system as Embodiment 3. FIG. 26 is a block diagram illustrating a hardware configuration of a sales floor device in the product recognition system of FIG. 25. FIG. 27 is a functional block diagram illustrating an example of a functional configuration of a server, a checkout machine, and a counter in FIG. 26. FIG. 28 is a functional block diagram illustrating a detailed functional configuration example of a moving object tracking unit provided in the counter in FIG. 27. FIG. 28 is a functional block diagram illustrating a detailed functional configuration example of a shelf product recognition unit provided in the counter in FIG. 27. FIG. 28 is a functional block diagram illustrating a detailed functional configuration example of a basket product recognizing unit provided in the sales floor apparatus of FIG. 27. It is a flowchart explaining the basic flow of the automatic payment process which the server of FIG. 27, a sales floor apparatus, and a payment machine performs. It is a flowchart explaining the process which recognizes the goods in a basket of the automatic payment process which a server counter apparatus and a payment machine of FIG. 27 perform. It is a figure showing the example of a layout of the supermarket which adopts the goods recognition system in Embodiment 4. FIG. 14 is a configuration diagram illustrating a configuration of a product recognition system as an information processing system according to a fourth embodiment of the present invention. FIG. 35 is a diagram showing a hardware configuration of a payment gate in the product recognition system of FIG. 34. FIG. 37 is a functional block diagram illustrating an example of a functional configuration of the server in FIG. 7, the counter in FIG. 26, and the checkout gate in FIG. FIG. 14 is a functional block diagram illustrating a detailed functional configuration example of a moving object tracking unit provided in a sales floor device according to a fourth embodiment. FIG. 14 is a functional block diagram illustrating a detailed functional configuration example of a shelf product recognition unit provided in a sales floor device according to a fourth embodiment. It is a flowchart explaining the automatic payment process which the server 1, sales floor apparatus, and payment gate of FIG. 36 perform. It is a flowchart explaining the automatic payment process which the server 1, sales floor apparatus, and payment gate of FIG. 36 perform.

〔Overview〕
Hereinafter, an outline of an embodiment of an information processing system of the present invention will be described.
The information processing system of the present invention is applied as a product recognition system for automatically paying out products. The first and second embodiments of the product recognition system including a cashier terminal for specifying a product and the third and fourth embodiments of the product recognition system not including a cashier terminal for specifying a product will be described below.

FIG. 1 is a table listing the main points of the first to fourth embodiments of the information processing system of the present invention.
In the “executing store” column in FIG. 1, each executing store of the first to fourth embodiments is described. For example, the first embodiment is an information processing system that is mainly assumed to be applied in a convenience store.
However, this store is merely an example, and the application destination of each of the first to fourth embodiments is not particularly limited. For example, the store implementing the first embodiment may be a retail store such as a supermarket, a cafeteria, or a store where settlement is performed.
In the “payment place” column in FIG. 1, the place where the shopper makes payment in each of the first to fourth embodiments is described. The settlement place in the first and second embodiments illustrates a cashier terminal. The cashier terminal has a function of specifying an object as a product and adjusting the product. The settlement place in the third embodiment is a cash register stand. The checkout table has a function of placing a product already specified as a product from an object and paying the product. The settlement place in the fourth embodiment exemplifies a settlement gate. The checkout gate has a function to settle the product without placing an item already specified as a product from the object on the cashier.

The “summary” column in FIG. 1 describes an overview of each of the first to fourth embodiments. For example, in summary, the information processing system of the first embodiment is an information processing system that performs automatic payment of a product placed at a cashier terminal.
The details of each embodiment are described in the “details” column in FIG. That is, for example, the information processing system according to the first embodiment recognizes a product on hand at the cashier terminal by a cashier camera installed at the cashier terminal, and performs automatic payment of the merchandise.

Here, a product before being identified as a product sold in the store is called an “object”. Although not a product, a personal property brought by a shopper, for example, is also an “object”. Then, a product in which the “object” is specified as a product is referred to as a “product”. Therefore, "commodity" means a product sold at a store.
Further, the product recognition systems according to the first to fourth embodiments include one or more sensing devices that image an object. As the sensing device, various devices such as a temperature sensor and a distance sensor can be employed in addition to the image sensor (camera or the like).

An image captured by an image sensor such as a camera is hereinafter referred to as a “captured image”. Further, a captured image including an object as a subject is hereinafter referred to as an “object captured image”. On the other hand, a captured image including a product as a subject is hereinafter referred to as a “product captured image”.
In addition, in the present specification, when various image processing is performed on a product captured image and an object captured image, they are actually handled in the form of data, but for convenience of description, the data will be omitted below.
Here, as the object captured image and the product captured image, for example, a captured image of the entire area where the object of the cashier terminal is placed, an image of a product individually cut out, a product logo image, a product barcode image, a product barcode image , An image of the entire shelf of the product, an image of the store imaged from the ceiling or the like, and the like.
Further, in the information processing system of the present invention, when an object is specified as a product, various recognition methods are used for the object captured image to estimate which product the subject of the object captured image is, A list of products to which the subject of the image matches and the degree of matching (hereinafter, also referred to as a “product candidate list”) is generated.

  Further, in the information processing systems according to the first to fourth embodiments, when checking out a product, an object captured image or a product captured image of an object that cannot be specified as a product or a product whose trading is restricted is output to the inspection terminal. At the inspection terminal, the inspector specifies the product and determines the trading restrictions (such as whether to cancel the trading restrictions). Cancellation of the trade restriction of the trade-restricted product is performed by a method in which the clerk responds on the spot in response to the result of the determination, or employs the result of the inspection as a system. Then, the inspection result of the inspection person is appropriately referred to, and the settlement of the product is performed.

FIG. 2 is a schematic diagram illustrating an outline of a system flow in the first and second embodiments.
As shown in FIG. 2, in the first and second embodiments, (1) an object image is captured by a cashier terminal and a product is specified. Is determined, (3) related information on these objects or commodities is transmitted to the inspection terminal. As the related information, for example, an object captured image, a product captured image, a log of a recognition result, a product candidate list, and the like can be adopted. At this time, a configuration may be adopted in which a plurality of frames of the object image are transmitted. Then, (4) the inspection terminal is notified of the inspection request (reception of the relevant information on the inspection target) at the inspection terminal, and (5) the inspector executes the inspection. As a result, it is possible to uniquely specify an object for which a product could not be specified at the cashier terminal as a product, or to cancel the trade restriction of a trade-restricted product. Further, (6) a result of the inspection by the inspector (a result of specifying the product or a result of the determination of the trade restriction) is transmitted to the cashier terminal, and (7) the inspection result is referred to at the cashier terminal. At this time, it is also possible to notify the cashier terminal that the product is a trading restricted product.
When the result of unspecified product is returned even after the inspection, or when the image of the object (such as the purchaser's face or ID card) used to determine the release of the trading restriction of the trading restricted product is unclear, etc. Alternatively, the configuration may be such that the purchaser is informed to the purchaser by a screen display or the like at the cashier terminal, or a configuration in which guidance is provided by voice. Specific cases that can be considered are (A) when the product is unspecified, requesting the purchaser to replace the product (separate the products, do not stack the products), and (B) determine the release of the trading restrictions. When the captured image of the purchaser's face or ID card is not clear, there is a request to retake the image, and (C) guide the purchaser on how to use the cashier terminal.
When providing voice guidance, a pre-recorded voice may be played. Further, a configuration may be adopted in which the purchaser and the inspector talk with each other via a microphone and a speaker provided in the cashier terminal.

FIG. 3 is a schematic diagram illustrating an outline of a system flow in the third and fourth embodiments.
As shown in FIG. 3, in the third and fourth embodiments, (1) the imaging of the object image and the identification of the product are performed in the store (in the sales floor), and (2) the object or the purchase or sale of the product cannot be identified. If it is determined that the product is a restricted product, (3) relevant information on these objects or the product is transmitted to the inspection terminal. As the related information, for example, an object captured image, a product captured image, a log of a recognition result, a product candidate list, position information at which the object captured image is captured, information for tracking a product or a shopper in a store, and the like are employed. be able to. At this time, a plurality of frames of the object image may be transmitted. Then, (4) the inspection terminal is notified of the inspection request (reception of the relevant information on the inspection target) at the inspection terminal, and (5) the inspector executes the inspection. As a result, it is possible to uniquely specify an object in the store for which the product could not be specified as the product, or to cancel the trade restriction of the trade-restricted product. Further, (6) a result of the inspection performed by the inspector (a result of specifying the product or a result of the determination of the trade restriction) is transmitted to the sales floor device installed in the sales floor, and (7) the inspection result is referred to in the sales floor device. Is done. At this time, it is also possible to notify the sales floor device that the product is a trading restricted product.
When the result of unspecified product is returned even after the inspection, or when the image of the object (such as the purchaser's face or ID card) used to determine the release of the trading restriction of the trading restricted product is unclear, etc. Alternatively, a configuration may be adopted in which a purchaser is notified on a screen display or the like in a sales floor device or the like, or a configuration in which guidance is provided by voice. Specific cases that can be considered are: (A) when the product is unspecified, requesting the purchaser to reload the product (reloading from the shelf or reloading the basket); (B) determination of cancellation of trading restrictions When the captured image such as the purchaser's face or identification card is unclear, there is a request to retake the image, or (C) a guide to the purchaser on how to use the sales floor device or the like.
When providing voice guidance, a pre-recorded voice may be played. Further, a configuration may be adopted in which the purchaser and the inspector communicate with each other via a microphone and a speaker provided in the sales floor device or the like.
The first to fourth embodiments of the information processing system in which such a system flow is executed can be specifically realized as a product recognition system as described below.

The product recognition system according to the first embodiment recognizes the existence of an object based on an object captured image including an object placed at a cashier terminal as a subject.
Specifically, for example, in the first embodiment, the cashier terminal includes one or more cashier cameras as an example of a sensing device. The cashier camera images a predetermined area of the cashier terminal. Specifically, the cash register camera captures an image of a predetermined area before an object is placed. The cash register camera captures an image of a predetermined area after the object is placed in the predetermined area. Therefore, the cashier terminal in the product recognition system of the first embodiment compares the captured image before the object is placed in the predetermined area of the cashier terminal with the captured image of the object after the object is placed in the predetermined area of the cashier terminal. By doing so, the presence of the object is recognized.
In addition, by using a segmentation technique in image recognition other than comparing the captured image before the object is placed in the predetermined area of the cashier terminal and the captured image of the object after the object is placed in the predetermined area of the cashier terminal, The presence of the object may be recognized only from the image of the object after the is placed in the predetermined area of the cashier terminal.
In the first embodiment, the cashier terminal specifies which of the recognized objects is a product by an object recognition method using image recognition. Here, for example, a method of specifying a product with high accuracy by creating a product candidate by deep learning and then performing a verification function is adopted.
In the first embodiment, the examiner confirms an object placed on the cashier terminal at an inspection terminal connected to the cashier terminal via a network according to the set conditions. This confirmation result (inspection result) is transmitted to the cashier terminal. Note that “visual inspection” means that a human visually confirms an object and, for example, specifies what kind of product the object is, and makes some conclusion. Such a conclusion is hereinafter referred to as a “confirmation result” or a “check result”.
When such a confirmation result (inspection result) is transmitted to the cashier terminal, the cashier terminal determines the final product based on the identification result by the object recognition method based on the image recognition described above and the confirmation result. I do. The final product may be specified not by the cashier terminal but by another device (not shown) or a natural person.
Then, in the first embodiment, the cashier terminal recognizes the quantity of the product specified next. In the first embodiment, the specified product is settled next.

The product recognition system according to the second embodiment is applied to a store such as a bookstore. More specifically, the product recognition system according to the second embodiment will be described as "in a shelf" including a space between shelves installed in a counter in a bookstore or on a table such as a wagon. ) Was recognized as the number of books, the shopper was tracked until the book was picked up and placed on the cashier, and if the book was placed on the cashier, the number of books placed was recognized. The book is identified as a product by specifying the book, and the book is automatically settled. At this time, in the second embodiment, according to the set conditions, the examiner checks the book placed on the cashier terminal at an inspection terminal connected to the cashier terminal via a network (the book is sold as a product Specified). Therefore, the product recognition system according to the second embodiment may further specify the product in consideration of the confirmation result (the inspection result).
Note that the inspector may visually check the shopper before the above-described “book” is visually checked. Thereby, even if the tracking by the product recognition system fails or the like, the product can be specified.

  The product recognition system according to the third embodiment is applied to a retail store such as a supermarket. Specifically, the product recognition system according to the third embodiment recognizes baskets (shopping carts and carts) placed at the sales floor of a retail store such as a supermarket, and tracks the baskets moving through the sales floor. In the product recognition system according to the third embodiment, when the object is taken out of the shelf, the object is recognized and specified as a product, and the baskets are placed on the cashier so that the list of the products put in the baskets is displayed. It is read and the product is automatically settled. At this time, in the third embodiment, according to the set conditions, the object taken from the shelf is confirmed by the inspector at an inspection terminal connected to a camera or the like installed at the sales floor via a network. You. Therefore, the product recognition system according to the third embodiment may further specify the product in consideration of the confirmation result (the inspection result).

  The product recognition system according to the fourth embodiment is applied to a retail store such as a supermarket. Specifically, the shopper recognizes and tracks not only shoppers and shopping carts and carts placed in supermarkets, but also shopping carts including shoppers' my bags and shopping bags as moving objects. In the commodity recognition system according to the fourth embodiment, the commodity is recognized and specified at the stage when the object is taken from the shelf, so that the cash can be automatically settled at the cash register even if the commodity is not placed at the cash register terminal. At this time, in the fourth embodiment, according to the set conditions, the object taken from the shelf is also checked by the inspector at an inspection terminal connected to a camera or the like installed at the sales floor via a network. Is done. Therefore, the product recognition system according to the fourth embodiment may further specify the product in consideration of the confirmation result (the inspection result).

  Hereinafter, Embodiments 1 to 4 will be described with reference to the drawings.

[Embodiment 1]
The information processing system according to the first embodiment is a product recognition system having a cashier terminal 2 as shown in FIG. 5 employed in a store such as a convenience store as shown in FIG. The information processing system according to the first embodiment can automatically perform the payment by placing a product on a cashier terminal.

FIG. 4 is a diagram illustrating a layout example in a case where a store that employs the information processing system of the first embodiment is a convenience store.
A cashier counter 12 is installed near the entrance 11 in the store 10. On the cashier counter 12, an unmanned cashier terminal 2 for automatically checking out goods is installed. A manned cash register 13 is installed next to the cashier terminal 2.
In the store 10, a plurality of shelf racks 14 for displaying products are installed, and a space 15 between the facing shelf racks 14 is a passage 15 through which shoppers move.
The merchandise in the shelf is picked up by the shopper who has moved along the aisle 15, and is placed in a predetermined area of the cashier terminal 2 (a predetermined area A in FIG. 5 described later). With respect to the products placed in the predetermined area, a plurality of products are specified collectively by the cashier terminal 2 and automatically settled by a predetermined operation of the shopper on the cashier terminal 2 as a trigger.
In the manned register 13, the store clerk recognizes the goods one by one with a barcode and makes a payment as usual.

A configuration example of the appearance of the unattended cashier terminal 2 will be described with reference to FIG.
FIG. 5 is a schematic perspective view showing a configuration example of the external appearance of the cashier terminal 2.
The cashier terminal 2 includes a surrounding part 270 surrounding a predetermined area A on which an object is placed. The surrounding part 270 includes a top plate part 271, a bottom plate part 272, and a pair of side plate parts 273.

A registration camera 211 that captures an image of the predetermined area A is fixed to each of the top plate 271 and the pair of side plates 273. The cash register camera 211 captures an image of an object placed in the predetermined area A.
In FIG. 5, only three registration cameras 211 are drawn, but five registration cameras 211 may be used as described later, and it is sufficient if at least one camera exists, and the number is not limited. The cashier terminal 2 also includes an external camera 212 that images the face, hands, and the like of the shopper.

The housing 275 is provided on the bottom plate 272. A receipt output unit and a display unit (not shown in FIG. 5) (a receipt output unit R and a display unit D of the output unit 206 in FIG. 8 described later) are provided on the front surface of the housing unit 275.
A translucent plate 276 on which an object is placed is provided on the housing 275. The upper surface of the plate 276 is a predetermined area A. The board surface of the plate 276 is formed in a wave shape. The wave shape may be not only a sine wave shape but also a rectangular wave, and the pitch and the amplitude may be not only uniform but also unequal.
The plate 276 is such that the predetermined area A is formed by repeatedly forming the concave portion and the convex portion, so that at least a part of the columnar or spherical object is sandwiched between the convex portion and the convex portion, It can be prevented from rolling.

An illumination unit 221 for illuminating the predetermined area A is provided in the plate 276 and the top plate 271 of the surrounding unit 270. The lighting part 221 may be provided on the side plate part 273 of the surrounding part 270.
The illumination unit 221 emits light of not only white color but also blue, red, and other various colors. When the illumination unit 221 emits light, the shadow of the object placed in the predetermined area A does not occur or decreases in the predetermined area A.

The surrounding unit 270 determines whether the cashier terminal 2 is in a state, for example, in a state of normal standby, in a state of settlement, in a state of a clerk being operated, or in an abnormal state. The presentation unit 210 can be changed in color so that it can be visually recognized in color.
At least the top plate portion 271 and the side plate portion 273 of the surrounding portion 270 may be constituted by an instantaneous light control sheet so that the transparent plate can be switched between a transparent state in which it becomes transparent and an opaque state in which it becomes opaque.
In this case, visibility of the predetermined area A can be ensured by making the surrounding portion 270 transparent. By setting the surrounding portion 270 in the opaque state, it is possible to acquire an object captured image while suppressing the influence of external light at the time of shooting.

Such a cashier terminal 2 is incorporated in a product recognition system as the first embodiment of the information processing system.
FIG. 6 is a configuration diagram showing a configuration of the product recognition system as the first embodiment of the information processing system of the present invention.
The product recognition system according to the first embodiment includes a server 1, n (n is an integer of 1 or more) cashier terminals 2-1 to 2-n, and a visual inspection terminal Q. .
The server 1, the n cashier terminals 2-1 to 2-n, and the inspection terminal Q are mutually connected via a network N such as the Internet. The inspection terminal Q can be carried by a clerk who is remote from the cashier terminal 2, prepared for a back yard of a store, or provided for a call center remote from the store.

For convenience of explanation, only one server 1 in FIG. 6 is drawn, but there may be a plurality of servers 1 in practice.
Hereinafter, when it is not necessary to individually distinguish the cashier terminals 2-1 to 2-n, these will be collectively referred to as "cashier terminals 2".

The server 1 performs necessary processing so that the operations of the plurality of cashier terminals 2 are performed in harmony. The cashier terminal 2 is placed on the cashier counter 12 shown in FIG. The cashier terminal 2 specifies the number of objects placed in the predetermined area A of the cashier terminal 2 by the shopper, specifies a product, and performs automatic payment.
However, the cashier terminal 2 does not necessarily require the server 1 and can function alone. In this case, some or all of the functions of the server 1 are included in another information processing apparatus (for example, the cashier terminal 2).

FIG. 7 is a block diagram illustrating a hardware configuration of the server 1 in the information processing system according to the first embodiment in FIG.
The server 1 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a bus 104, an input / output interface 105, an output unit 106, an input unit 107, , A storage unit 108, a communication unit 109, and a drive 110.

The CPU 101 executes various processes according to a program stored in the ROM 102 or a program loaded into the RAM 103 from the storage unit 108.
The RAM 103 also appropriately stores data and the like necessary for the CPU 101 to execute various processes.

  The CPU 101, the ROM 102, and the RAM 103 are mutually connected via a bus 104. An input / output interface 105 is also connected to the bus 104. The output unit 106, the input unit 107, the storage unit 108, the communication unit 109, and the drive 110 are connected to the input / output interface 105.

The output unit 106 includes a display, a speaker, and the like, and outputs various information as images and sounds.
The input unit 107 includes a touch panel, a keyboard, a mouse, a microphone, and the like, and inputs various information.

The storage unit 108 is configured by a hard disk, a DRAM (Dynamic Random Access Memory), or the like, and stores various data.
The communication unit 109 communicates with the cashier terminal 2 via a network N including the Internet, as shown in FIG.

A removable medium 120 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted on the drive 110. The program read from the removable medium 120 by the drive 110 is installed in the storage unit 108 as needed.
Further, the removable medium 120 can also store various data stored in the storage unit 108 in the same manner as the storage unit 108.

FIG. 8 is a block diagram illustrating a hardware configuration of the cashier terminal 2 in the information processing system according to the first embodiment in FIG.
The cashier terminal 2 includes a CPU 201, a ROM 202, a RAM 203, a bus 204, an input / output interface 205, an output unit 206, an input unit 207, a lighting unit 221, a light blocking unit 209, a presentation unit 210, a cash register The camera includes a camera 211, a storage unit 208, a communication unit 213, and a drive 214.
A removable medium 220 is appropriately mounted on the drive 214.

  The CPU 201, the ROM 202, the RAM 203, the bus 204, the input / output interface 205, the storage unit 208, the communication unit 213, the drive 214, and the removable medium 220 of the cashier terminal 2 are configured similarly to those of the server 1.

The output unit 206 is provided in the housing unit 275 shown in FIG. The output unit 206 includes a display unit D that displays information about merchandise, information about settlement, and the like, a receipt output unit R that outputs a receipt, and a speaker S that outputs audio.
The input unit 207 is provided in the housing unit 275 shown in FIG. The input unit 207 includes a touch panel (not shown), a card reader unit C, and a microphone M. Further, a barcode reader (not shown) may be provided.

When the surrounding part 270 is constituted by an instantaneous light control sheet, the light shielding part 209 switches the surrounding part 270 shown in FIG. 5 between a transparent state and an opaque state.
The presentation unit 210 can determine whether the state of the cashier terminal 2 is in a normal standby state, a checkout state, a state in which a clerk is operating, or whether an abnormal situation has occurred. Then, switching is performed so that the presentation unit 210 illustrated in FIG. 5 emits light of a different color. The presentation unit 210 is provided not only on the front but also on the back.
The cash register camera 211 captures an image of an object placed in the predetermined area A, and outputs one or more captured images obtained as a result as an object captured image.

  Since the hardware configuration of the inspection terminal Q is the same as the hardware configuration of the server 1, the hardware configuration of the inspection terminal Q should be referred to FIG. Description is omitted.

FIG. 9 is a functional block diagram illustrating an example of a functional configuration of the server 1, the cashier terminal 2, and the inspection terminal Q.
In the CPU 101 of the server 1, a DB management unit 141 that manages personal information, product information, and the like functions. The DB management unit 141 may be provided in each of the cashier terminals 2.
In one area of the storage unit 108 of the server 1, a product DB 131 is provided. The product DB 131 is a DB (Data Base) that stores product information.

In the CPU 201 of the cashier terminal 2, as shown in FIG. The object quantity recognizing unit 234, the product specifying unit 235, the trading restricted product determining unit 236, the settlement unit 237, the display control unit 238, and the inspection result obtaining unit 239 function.
Note that the cashier terminal 2 includes a DB information holding unit 241 that holds personal information, product information, and the like.
In the CPU 101 of the inspection terminal Q, the image display control unit 411 and the inspection result transmission unit 412 function.

The light emission control unit 228 of the CPU 201 of the cashier terminal 2 performs control to switch between a state in which the illumination unit 221 emits light at the timing of imaging the object and a state in which the illumination unit 221 does not emit light at the timing of not imaging the object. The control for switching the emission color of the illumination unit 221 is performed according to the situation of recognizing the placed object.
When the surrounding portion 270 is constituted by an instantaneous light control sheet, the light shielding control portion 229 switches the surrounding portion 270 between an opaque state and a transparent state. In other words, the light-shielding control unit 229 determines whether the light-shielding unit 209 provided in the surrounding unit 270 has an opaque state at the time of shooting an object placed in the predetermined area A and a transparent state at a time when the object is not imaged. Execute the control to switch to.
The presentation control unit 230 performs control such that the presentation unit 210 changes the emission color for presenting the state of the cashier terminal 2.

The personal authentication unit 231 performs the personal authentication of the shopper by referring to the personal information managed by the DB management unit 141 during the settlement process. Specifically, the personal authentication unit 231 performs an authentication process using various authentication methods such as face authentication, card authentication, fingerprint authentication, vein authentication, and iris authentication before recognizing the image of the product. . The personal information managed by the DB management unit 141 includes information such as age, allergy, and halal. Therefore, the personal information acquired by the personal authentication unit 231 is used by the trading restricted product determination unit 236.
The image acquisition unit 232 acquires an object image captured by the cashier camera 211 with the object placed in the predetermined area A as shown in FIG.

FIG. 10 illustrates an example of an imaging screen of an object placed on the cashier terminal 2. Here, FIG. 10 is a diagram illustrating an example of an object image obtained as a result of imaging the objects X, Y, Z, and Z ′ placed in the predetermined area A by each of the three cash registers 211.
FIG. 10A illustrates an example of an object captured image obtained as a result of imaging the objects X, Y, Z, and Z ′ by the cash register camera 211 fixed to the top panel 271 of the cashier terminal 2. The object image includes six objects. Since the object image does not include the objects X, Y, Z, and Z ′ that are shadows of the objects, all the objects X, Y, Z, and Z ′ placed in the predetermined area A are imaged.
FIG. 10B illustrates an example of an object captured image obtained as a result of imaging the objects X, Y, Z, and Z ′ by the cash register camera 211 fixed to one side plate unit 273 of the cashier terminal 2. The object image includes two objects. In this object image, the objects X, Y, Z, and Z 'on the side of the cash register camera 211 hide the objects on the back side.
FIG. 10C illustrates a captured image obtained as a result of imaging the objects X, Y, Z, and Z ′ by the cash register camera 211 fixed to the other side plate 273 of the cashier terminal 2. The object image includes two objects. In this object image, the objects X, Y, Z, and Z 'on the side of the cash register camera 211 hide the objects on the back side.
One of the two objects Z included in the object captured images of FIGS. 10B and 10C is the same object Z ′, but the other is a different object.

The object recognizing unit 233 recognizes the presence of an object placed in the predetermined area A from the object captured image acquired by the image acquiring unit 232, using the above-described predetermined image recognition technique. That is, the object recognizing unit 233 compares the background image before the object is placed in the predetermined area A of the cashier terminal 2 with the image of the object after the object is placed, and executes the background subtraction process to determine the object area for each object. By defining (specifying) the presence of an object is recognized.
Note that the object recognizing unit 233 uses a method other than the background subtraction process, and does not compare the background image before the object is placed in the predetermined area A with the image of the object after the object is placed. The presence of an object may be recognized by defining the object region from.

The object quantity recognition unit 234 recognizes the number of objects placed in the predetermined area A by comparing the object recognition quantity and the settlement quantity. The object recognition quantity is the quantity of the objects recognized by the object recognition unit 233 from the object captured images captured by the plurality of cashier cameras 211 of the cashier terminal 2.
The settlement quantity is the quantity of the commodities to be settled.
The quantity of object recognition may vary depending on the cash register camera 211 as shown in FIG. That is, in FIG. 10A, six objects are imaged, but in FIGS. 10B and 10C, two objects are imaged.
In such a case, the object quantity recognition unit 234 recognizes the number of objects by calculating the logical sum as shown in FIG.

FIG. 11 is a diagram illustrating an example of a truth table for calculating the number of objects placed on the cashier terminal 2.
FIG. 11 shows an image captured by the first to fifth cashier cameras 211 provided in the cashier terminal 2 (FIG. 5) for the objects X, Y, Z, and Z ′ shown in FIG. ", And" 0 "indicates that the image could not be captured. Note that the object Z and the object Z ′ are the same object, but represent that the image is taken in different states because the image of the registration camera 211 is different.
In FIG. 11, the first cash register camera 211 is represented as “register camera 1”, and similarly, the second to fifth cash register cameras 211 are represented as “register camera 2” to “register camera 5”. ing.
In the example of FIG. 11, the object X is imaged by the first, fourth, and fifth cash register cameras 211. The object Y is imaged by the second, fourth, and fifth cash register cameras 211. The object Z is imaged by the second, third, and fifth cash register cameras 211. The object Z ′ is imaged by the first and third cash register cameras 211.

In the example of FIG. 11, there is no product imaged by all of the first to fifth cashier cameras 211. Further, the object Z and the object Z 'are imaged like different objects because the image of the registration camera 211 is different even though they are the same product. It is considered that this is because the angle of the cash register camera 211 for imaging is different.
Based on this, the object quantity recognition unit 234 recognizes the quantity of the product by a method using the logical sum. That is, even if an object is placed in the predetermined area A so as to overlap or an object is placed in the predetermined area A so as to be a shadow of another object, the logical image of the object captured by any one of the cash register cameras 211 is obtained. Is recognized as being placed in the predetermined area A.

  If the object recognition quantity recognized from the plurality of object captured images using the truth table and the settlement quantity are not the same, the object quantity recognition unit 234 informs the display control unit 238 that the quantities are different. Output.

  Returning to FIG. 9, the product specifying unit 235 matches the object whose existence has been recognized by the object recognition unit 233 with the product information stored in the DB information storage unit 241. That is, the product specifying unit 235 first lists product candidates by an image recognition method such as specific object recognition, general object recognition, and deep learning. The listed product candidates are referred to as “product candidate list S”. Thereafter, the product specifying unit 235 performs a verification function and specifies the product with high accuracy.

The verification function is a function of listing "commodity candidate list P" by an algorithm different from the above-described technique of listing commodity candidates. If the results of the product candidate lists S and P are matched, and the value exceeds a predetermined threshold, the product is specified.
The listing method of the “commodity candidate list” is realized by, for example, a method of matching image information of an object obtained from an object whose existence has been recognized with image information held in the DB information holding unit 241 or a memory. You may. That is, when the feature information of both images match (exceeds the threshold), the object whose existence has been recognized by the object recognition unit 233 is a product registered in the DB information holding unit 241. , It is specified as a product registered in the DB information holding unit 241.

  Here, a specific example will be described in which when the product specifying unit 235 cannot specify a product by the image processing method, the result is output as product unspecified. First, the product specifying unit 235 compares the product image stored in the DB information holding unit 241 with the image of the object captured in the predetermined area A, and finds a feature point (similar feature point) and a feature that both images are similar. Calculate the amount. Next, the product specifying unit 235 reads the feature points and the feature amounts of the product images included in the product candidate list from the product DB 131 of the server 1. Next, the product specifying unit 235 compares the feature amount of each of the feature points of the product included in the read product candidate list with the feature amount of each of the feature points of the recognized object, and stores the feature amount in the DB information holding unit 241. The similar feature point between the product image and the object image captured in the predetermined area A is matched.

  Next, the product specifying unit 235 compares the positional relationship using the corresponding point coordinates of each set of similar feature points, and detects a similar feature point that does not correctly correspond to the change due to rotation or translation (the positional relationship does not match). Are removed, and the number of remaining similar feature points is calculated. The product specifying unit 235 determines that the product is not specified when the number of similar feature points is less than the threshold.

  In such a method for unspecified merchandise, a non-packaged merchandise such as a lunch box may be an unspecified merchandise. The lunch box may have a slightly different side dish position, and in such a case, it may be regarded as an unspecified product. In such a case, the product identification unit 235 detects various codes such as a multi-dimensional code including a barcode written on a label attached to the product and characters such as a product name, and detects the DB information holding unit 241 and the like. The product name or the product code may be read based on the product information stored in the product DB 131 of the server 1 to specify the product.

  Further, the product specifying unit 235 verifies similar products and related products (hereinafter, referred to as “group products”) using the product information stored in the DB information holding unit 241 and the product DB 131 of the server 1. The product specifying unit 235 compares, for example, threshold values of group products of a series having different sizes, colors, and the like, using characteristics such as size and color of the products. The product specifying unit 235 determines that the product is unspecified when the threshold is less than a predetermined threshold.

In the present embodiment, a mode in which inspection is performed by the inspection terminal Q for a specific object (hereinafter, referred to as a “specific object inspection mode”), and an inspection performed by the inspection terminal Q for all objects. (Hereinafter, referred to as “all object inspection mode”) can be set (the same applies to each of the following embodiments).
When the specific object inspection mode is set, the product specifying unit 235 outputs the object captured image of the object determined as the product unspecified and related information to the inspection terminal Q via the inspection result acquisition unit 239. Send to On the other hand, when the all-object inspection mode is set, the product specifying unit 235 determines the result of specifying the product, the product captured image of the specified product, the object captured image of the object for which the product is not specified, and related information. Are transmitted to the inspection terminal Q via the inspection result acquisition unit 239.
Then, the product specifying unit 235 specifies the product based on the inspection result acquired from the inspection terminal Q by the inspection result acquisition unit 239. Specifically, for the product specified by the product specifying unit 235, when the specified result is approved or corrected by the inspection result, the product specifying unit 235 specifies the product according to the content indicated by the inspection result. . In addition, for an object that has been determined to be unspecified by the product specifying unit 235, the product specifying unit 235 specifies the object as a product indicated by the inspection result.

The sales-restricted product determination unit 236 determines whether or not the product specified by the product specification unit 235 is a sales-restricted product based on the determination information.
Restricted trading products include (A) products that cannot be purchased unless they reach a certain age, such as tobacco and alcoholic beverages, (B) products that have expired or expired, and (C) that contain allergic components. And (D) products other than halal foods and the like, which are restricted by religion.
In the present embodiment, the sales-restricted product determination unit 236 transmits a product captured image of a product corresponding to the sales-restricted product to the inspection terminal Q via the inspection result acquisition unit 239. At this time, the sales-restricted product determination unit 236 checks the image (such as a face image or an ID card image) related to the shopper according to the type of the trade-restricted product (the above (A) to (D), etc.). It is transmitted to the inspection terminal Q via the inspection result acquisition unit 239 as appropriate. At this time, if the image related to the shopper is unclear, a configuration may be adopted in which the purchaser is notified at the cashier terminal 2 by screen display, voice guidance, or the like, and a request to retake the image is made. Further, a configuration may be adopted in which the purchaser and the examiner talk via the microphone M and the speaker S provided in the cashier terminal 2.
Then, based on the inspection result acquired by inspection result acquisition section 239 from inspection terminal Q, trading restricted product determination section 236 determines whether or not sale of the trading restricted product is permitted (trading restriction is released. Or not).

  The settlement unit 237 calculates the total amount of money for all the products specified by the product specifying unit 235. At this time, if the sale-restricted product determination unit 236 determines that there is a sale-restricted product, the restriction needs to be released for all of the sale-restricted products. Therefore, the settlement unit 237 reads the price of the product placed in the predetermined area A from the DB information holding unit 241 and displays the price on the display unit D (FIG. 8).

  The display control unit 238 controls the output unit 206 to warn a shopper or a clerk to check the number of products when the object recognition amount and the settlement amount do not match by the object number recognition unit 234. I do. The display control unit 238 alerts the shopper or the clerk (object) when the product is not specified by the product specifying unit 235, or when the product is not specified by the inspection at the inspection terminal Q. The output unit 206 is controlled so as to output an instruction for re-positioning. Specifically, a configuration may be adopted in which the purchaser is notified by a screen display, voice guidance, or the like, and the purchaser is requested to replace the product. Further, a configuration may be adopted in which the purchaser and the examiner talk via the microphone M and the speaker S provided in the cashier terminal 2. The display control unit 238 controls the output unit 206 to warn a shopper or a sales clerk of the fact that the object is a trade-restricted product when the object is recognized as a trade-restricted product by the trade-restricted product determination unit 236. I do. Further, when the total amount is calculated by the settlement unit 237, the output unit 206 is controlled so that the product name, price, and the like of the product are displayed to shoppers and clerks.

The inspection result acquisition unit 239 requests the identification of the product by inspection by transmitting the object captured image or the product identification result and the product captured image of the identified product to the inspection terminal Q. The inspection result transmitted from the inspection terminal Q in response to the request is obtained. The inspection result acquisition unit 239 outputs the acquired inspection result (product identification result) to the product identification unit 235. In addition, the inspection result acquisition unit 239 requests the determination of the sales-restricted product by inspection by transmitting the product image of the product corresponding to the sales-restricted product to the inspection terminal Q. Get the inspection result sent from. At this time, the inspection result acquisition unit 239 appropriately transmits an image (a face image, an image of an identification card, and the like) relating to the shopper to the inspection terminal Q according to the type of the sale-restricted product. The inspection result acquisition unit 239 outputs the acquired inspection result (trading-restricted product determination result) to the trading-restricted product determination unit 236.
The sale-restricted-commodity determination unit 236 determines that the object is a trade-restricted product, and issues a warning from the presentation unit 210 and the output unit 206 for a product for which the trade restriction has not been released by the inspection result. The clerk who notices this presentation takes measures such as collecting the trade-restricted goods so as not to sell the trade-restricted goods.
In the product recognition system, a remote control unit for instructing release of the trading restriction on the trading restricted product may be provided separately from the inspection terminal Q. The remote control unit can be carried by a clerk who is remote from the cashier terminal 2 or can be provided in the backyard of the store. The remote control unit can remotely control a plurality of cashier terminals 2 by one.

In the inspection terminal Q, when an inspection request is made from the cashier terminal 2, the image display controller 411 transmits an image such as an object captured image or a product captured image transmitted from the cashier terminal 2 and these images. Then, various information (commodity candidate list and the like) transmitted along with is output to the output unit 106.
The inspection result transmitting unit 412 is configured to execute an inspection result (specification result of a product or determination of a sales-restricted product) input by the examiner via the input unit 107 with respect to the object captured image or the product captured image output to the output unit 106. Is transmitted to the cashier terminal 2 which has requested the inspection.

The product recognition system applied as the information processing system of the present specification includes a product registration system for capturing the appearance of a product sold at the store 10 and registering the image together with product information such as the price of the product. . The product registration may be performed in the store 10 or may be performed outside the store such as a product maker or a wholesaler, and the place is not limited.
The product registration system includes a registration image generation unit (not shown), a product master registration unit, a captured image acquisition unit, and a recognition unit.

The registration image generation unit generates an image of a product that can be placed in the predetermined area A of the cashier terminal 2 as a product registration image.
The product master registration unit registers the product registration image generated by the registration image generation unit and a product identifier uniquely assigned to a product included as a subject in the product registration image in association with each other.
The captured image acquisition unit acquires a captured image of an object placed in a predetermined area A of the cashier terminal 2 as an object captured image. Based on the captured object image and the product registration image, the product specifying unit 235 specifies which product is the object whose existence has been recognized.
The object recognizing unit 233 recognizes the presence of an object placed in the predetermined area A based on the acquired object image.
This product recognition system matches the object captured image of the object whose existence has been recognized with the image of the product stored in the DB information storage unit 241 or the storage unit 108 of the server 1 as described above. Which product is specified by 235.

  In the store 10 in which such a product recognition system is installed, a product registration image is generated, and a product identifier included in the product registration image is assigned to the product, whereby the master registration based on the product identifier uniquely assigned to the product is performed. Will be possible. Further, the product recognition system provided with this product registration system can manage products for which a barcode seal cannot be attached.

Here, a method for adjusting a product in the product recognition system according to the first embodiment will be described with reference to FIG.
FIG. 12 is a flowchart for explaining a product settlement process executed by the server 1, the cashier terminal 2, and the inspection terminal Q in FIG.
Prior to the automatic payment process for the product, an image of the predetermined area A captured in advance with no object placed in the predetermined area A of the cashier terminal 2 is stored in the image acquisition unit 232.
The image of the predetermined area A captured in advance is updated at a predetermined timing, is not updated every time the shopper uses the cashier terminal 2, and may be shared or changed.
Then, when the shopper presses the operation button of the cashier terminal, the automatic payment process is started.

In step S101, the cash register camera 211 of the cashier terminal 2 captures an image after an object is placed on the predetermined area A of the cashier terminal 2. The image is input to the image acquisition unit 232, for example, as an object captured image as shown in FIG.
In this object image, the inside of the cashier terminal 2 is illuminated by the illumination unit 221 so that the shadow of the object does not occur or is reduced.
The captured object image may be input to the display control unit 238 and output from the output unit 206.
In addition, when the external camera 212 of the cashier terminal 2 captures an image of the shopper, the personal authentication unit 231 performs personal authentication of the shopper.

  In step S102, the object recognition unit 233 of the cashier terminal 2 recognizes the presence of the object placed in the predetermined area A from the object captured image acquired by the image acquisition unit 232 using the above-described predetermined image recognition method. I do. That is, the object recognizing unit 233 compares the background image before the object is placed in the predetermined area A of the cashier terminal 2 with the captured image of the object after the object is placed. By defining (specifying) the presence of an object is recognized.

  In step S103, the product specifying unit 235 of the cashier terminal 2 determines whether any of the objects placed in the predetermined area A of the cashier terminal 2 includes an object that cannot be identified as any product.

In step S104, the product specifying unit 235 of the cashier terminal 2 determines whether the mode is set to the specific object inspection mode or the all object inspection mode.
If the specific object inspection mode has been set, the process proceeds to step S106.
If the all-object inspection mode has been set, the process proceeds to step S105.

In step S105, the inspection result acquisition unit 239 of the cashier terminal 2 requests an inspection (identification of a product) at the inspection terminal Q for the target object or product, and acquires the inspection result.
After step S105, the process proceeds to step S107.

In step S105, when there is an object whose product is not specified even by the inspection at the inspection terminal Q, the presenting unit 210 of the cashier terminal 2 notifies the error state by color, sound, or the like.
When an error occurs due to unspecified merchandise, the object may be repositioned, and a predetermined operation may be performed on the cashier terminal 2 to return to step S101 and re-image the object. The replacement may be performed by a clerk who has found the error state, or may be performed by the shopper himself. A configuration may be adopted in which the purchaser and the examiner talk via the microphone M and the speaker S provided in the cashier terminal 2.
The error state occurs when the system processing of the cashier terminal 2 becomes abnormal, when the characteristic portion of the object cannot be imaged, when the object overlaps, or when the object is shaded, the trading restricted product is registered. For example, a case where a product or personal property is left behind on the cashier terminal 2 when the terminal 2 is placed on the terminal 2, or the like. When the objects overlap or become shadows, by re-positioning the object, it is possible to take an image of which product the object is and to specify the product.

  If there is an object for which the product specifying unit 235 cannot specify which product (YES in step S105), the process proceeds to step S105.

  If there is no object that the product specifying unit 235 cannot identify which product in step S106 (NO in step S103), the product specifying unit 235 stores the product stored in the DB information storage unit 241 or the storage unit of the server 1. Identify the product, including information such as name, price, and sale-restricted product. Accordingly, the process proceeds to step S107. The specified product information may be output to the display control unit 238.

In step S107, the sales-restricted product determination unit 236 executes processing of the sales-restricted product. The processing of the sales-restricted product is a process of determining whether the specified product is a sales-restricted product. Specifically, the trading-restricted product determination unit 236 determines, for example, whether the specified product is an age-restricted product, whether it is a halal non-applicable product or a product containing allergic components, whether the product is expired or expired, or the like. I do. The details of the processing of the sale-restricted product will be described later with reference to FIG.
Note that the personal authentication is acquired in advance at any timing up to step S105 (including before step S101), so that the sales-restricted product determination unit 236 can execute the processing of the sales-restricted product for each shopper. it can.

In step S108, the settlement unit 237 adjusts the goods placed in the predetermined area A.
Specifically, the settlement unit 237 acquires all the commodities placed in the predetermined area A by individually acquiring and summing up the prices of the commodities specified in step S103.
Product information such as the product name and price of the adjusted product is output from the display control unit 238 to the output unit 206, displayed on the display unit D of the output unit 206, and printed and output on the receipt from the receipt output unit R. You.
Further, since the cashier terminal 2 is connected to the server 1 via the communication unit 213, the cashier terminal 2 can be used as a POS (Point Of Sale). That is, the cashier terminal 2 links the purchase information and the estimated gender estimation information of the adjusted merchandise to the POS system.
In step S108, the presenting unit 210 of the cashier terminal 2 determines that the number of objects identified by the object quantity recognizing unit 234 is different from the number of items identified by the item identifying unit 235 and the settlement amount. May be presented. The cashier terminal 2 can cancel the payment process by providing the cancel function.

Next, the processing of the sale-restricted product in step S107 will be described in detail with reference to FIG.
FIG. 13 is a flowchart illustrating the processing of the trade-limited product executed by the server 1, the cashier terminal 2, and the inspection terminal Q of FIG. The sale restriction of the sale-restricted product is released by a method in which the clerk responds on the spot in response to the result of the determination, or employs the result of the inspection as a system. FIG. 13 illustrates a case where the result of inspection is adopted as a system.

  In step S111, the sales-restricted product determination unit 236 determines whether the product specified by the product specification unit 235 is a product such as an alcoholic beverage that requires age confirmation.

If it is determined in step S111 that the product specified by the product specifying unit 235 is a product that requires age confirmation, that is, if it is determined as YES, the process proceeds to step S112.
In step S112, the cashier terminal 2 uses the external camera 212 to obtain an image of the purchaser's face and age confirmation document. However, if the shopper's personal information is acquired and there is no need to confirm the age here, step S112 is skipped and the process proceeds to step S116.
In step S113, the sales-restricted product determination unit 236 of the cashier terminal 2 transmits the obtained image to the inspection terminal Q via the inspection result acquisition unit 239, and requests an inspection. The inspector who receives this image confirms the age of the purchaser based on the image.
At this time, when there is a problem with the age verification of the purchaser, for example, when the image of the purchaser's face or the image of the age confirmation document is unclear, the purchaser is notified by a screen display or voice guidance at the cashier terminal 2. Alternatively, a configuration may be employed in which the purchaser is requested to retake the image. Further, a configuration may be adopted in which the purchaser and the examiner talk via the microphone M and the speaker S provided in the cashier terminal 2.
In step S114, the trade-restricted product determination unit 236 determines whether an instruction to release the trade restriction has been received from the inspection terminal Q via the inspection result acquisition unit 239.

If it is determined in step S114 that an instruction to cancel the trading restriction has not been received, the process proceeds to step S115.
If it is determined in step S114 that a release instruction for releasing the trading restriction has been received, that is, if it is determined as YES, the process proceeds to step S116.
In step S115, the inspection result transmitting unit 412 of the inspection terminal Q transmits a warning that the trading restriction has not been released by the inspection result. Upon receiving this warning, for example, the display control unit 238 of the cashier terminal 2 presents, via the output unit 206, a warning that the trade restriction has not been released by the inspection result. After step S115, the trading restriction process ends.
In step S116, the trading restriction product determination unit 236 releases the trading restriction.
When step S116 is completed in this way, or when it is determined in step S111 that the product is not an age-restricted product (when it is determined that the product is not NO), the process proceeds to step S117.

In step S117, the trading-restricted product determining unit 236 determines whether the product specified by the product specifying unit 235 is a product other than halal (permitted) food, or whether it is an allergic product. I do.
If the sales-restricted product determination unit 236 determines that the product is a product other than a halal product (a product that is not permitted) or an allergic product, the process proceeds to step S118.

  In step S118, the display control unit 238 causes the display unit D of the cashier to display that the product is a product other than a halal product or that the product is an allergic product. However, if the shopper's personal information is acquired and it is not necessary to determine whether the product is a halal non-applicable product or an allergic product, step S118 is skipped and the process proceeds to step S121.

In step S119, the trading restriction product determination unit 236 determines whether or not an instruction to cancel the trading restriction has been received.
If it is determined that the instruction to cancel the trading restriction has not been received (if NO has been determined), the process proceeds to step S115.
If it is determined that an instruction to cancel the trading restriction has been received (if YES is determined), the process proceeds to step S120.

In step S120, the trading restriction product determination unit 236 releases the trading restriction.
In this way, if step S120 is completed, or if the product is not a halal product or is not an allergic product in step S117 (if NO is determined), the process proceeds to step S121.

In step S121, the sales-restricted product determination unit 236 determines whether the product specified by the product specification unit 235 is a product whose consumption period has expired.
If the sale-restricted product determination unit 236 determines that the product includes a product whose consumption period has expired, “YES” is determined in step S121, and the process proceeds to step S122.

In step S122, the display control unit 238 causes the display unit D of the cashier terminal 2 to display a message indicating that there is a possibility that the expired product is included.
If it is determined in step S123 that the instruction to cancel the trading restriction has not been received, the determination is NO, and the process proceeds to step S115.

If it is determined in step S123 that an instruction to cancel the trading restriction has been received, the determination is YES, and the process proceeds to step S124.
In step 124, the trading restriction product determination unit 236 releases the trading restriction.
If the sales-restricted product determination unit 236 determines that the product does not include a product whose consumption period has expired, NO is determined in step S121, and the process ends.

Thus, the trading restriction processing ends. Then, the process proceeds to the settlement at the cashier terminal in step S106.
In this way, the present information processing system can recognize the product placed at the cashier terminal and automatically make the payment. At this time, for the object or the product to be inspected (for example, the object that cannot be specified at the cashier terminal 2 or the product corresponding to the trading restricted product), Inspection is performed, and automatic settlement is performed according to the inspection result (specification result of the product, determination result of the sale-restricted product, etc.).
Therefore, according to the present information processing system, when a shopper purchases a product, it is possible to automate the settlement of the price of the product and improve the accuracy of specifying the product.

The first embodiment is not limited to the above-described embodiment, and includes modifications and improvements as long as the object of the present invention can be achieved.
For example, the appearance configuration of the cashier terminal 2 shown in FIG. 5 is an exemplification, and is not limited to this appearance. The cashier terminal 2 only needs to include at least the predetermined area A, an imaging unit such as the cashier camera 211, and the output unit 206, and may add other components.
Further, the product recognition system according to the first embodiment may include a transport mechanism (for example, a belt conveyor) that transports one or more objects from the upstream side to the downstream side. A predetermined area having an imaging unit is arranged on the upstream side. A settlement area is arranged on the downstream side.
In the predetermined area, the number of objects imaged by the imaging unit is counted. The merchandise recognition system detects an error when the counted number of objects is different from the number of items settled in the checkout area.

[Embodiment 2]
The information processing system according to the second embodiment is a product recognition system including a cashier terminal 2 as shown in FIGS. 5 and 15 in a store 20 such as a bookstore as shown in FIG. In the information processing system according to the second embodiment, a book, which is a product, is placed on a cashier, so that an automatic payment can be made.
FIG. 14 is a diagram illustrating a layout example of a bookstore employing the information processing system according to the second embodiment.

A store 20 such as a bookstore has an entrance 22 having a gate 21. Depending on the scale of the store 20, the number of the entrances 22 is not limited to two as shown, but may be one or three or more. The gate 21 is not only a type having an opening and closing member as shown in the figure, but also a type not shown but having a function of notifying the occurrence of an abnormal situation by sound or light such as a speaker or a lamp. It has been made possible.
The store 20 has a plurality of shelf racks 23 for displaying books. The shelf rack 23 arranges a plurality of shelves in the vertical direction at intervals and displays books. A passage 24 is formed between the shelf racks 23 facing each other in the horizontal direction.

A plurality of (only one is illustrated in the drawing) ceiling cameras 310 are installed on the ceiling of the passage 24. The ceiling camera 310 always images the shopper who has entered the store without blind spots.
Further, a plurality of (only one is illustrated in the drawing) shelf cameras 311 that constantly capture the inside of each shelf rack 23 may be installed in each shelf rack 23. A plurality of shelf cameras 311 are arranged so as to capture an image of the inside of the shelf without blind spots, and also to capture a shopper standing in front of the shelf rack 43.

A cashier counter 25 is provided near the entrance 22 in the store 20. On the cashier counter 25, a plurality of unmanned cashier terminals 2 for automatic payment are provided. A manned cash register 26 is installed next to the cashier terminal 2. A shopping basket (not shown) for storing books may be placed near the entrance 22 or the passage 24.
A store clerk Mt is working in the passage 24, the cashier counter 25, or the like. The clerk Mt has an information terminal 9. The information terminal 9 is also installed in a backyard of the store 20, a headquarters outside the store 20, and the like.

In this information processing system, the ceiling camera 310 captures an image of an act of a shopper taking out or returning one or more books from the shelf of the shelf rack 23, and grasps the number of books taken by the shopper. As shown in FIG. 15, information such as the price of a book placed on the unattended cashier terminal 2 is acquired and the balance is automatically settled.
In the second embodiment, the shopper is described as “moving object Mo”.
FIG. 15 is a schematic perspective view showing an outline configuration of the cashier terminal 2 employed in the second embodiment, and shows a state where a book (without numbering) is being settled.

The unattended cashier terminal 2 of the second embodiment employs the same appearance as the cashier terminal 2 of the first embodiment shown in FIG.
Therefore, the cashier terminal 2 includes the surrounding portion 270 surrounding the predetermined area A on which the book is placed. The surrounding part 270 includes a top plate part 271, a bottom plate part 272, and a pair of side plate parts 273. The surrounding portion 270 has the same configuration as the surrounding portion 270 of the first embodiment shown in FIG.

  Therefore, the registration camera 211 that captures an image of the predetermined area A is fixed to the top plate 271 and the pair of side plates 273. At least one cash register camera 211 captures at least the spine of the book placed in the predetermined area A as shown in FIG. In FIG. 15, the book is placed so that the spine is directed to one side plate 273 and the spine is directed to the other side plate 273, but the spine is attached to the cash register 211 installed on the top plate 271. May be directed, and how to place the book is not limited.

Such a cashier terminal 2 is incorporated in a product recognition system as one embodiment of an information processing system.
FIG. 16 is a configuration diagram showing a configuration of a product recognition system as Embodiment 2 of the information processing system of the present invention.
The product recognition system includes a server 1, cashier terminals 2-1 to 2-n, a counter device 3, and an inspection terminal Q. The counter device 3 has a function of recognizing the number of books from the image of the book captured by the ceiling camera 310.
The server 1 is installed in the backyard of the store 20 or outside the store to manage the cashier terminals 2-1 to 2-n, the sales floor device 3, and the inspection terminal Q. In addition, the sales floor device 3 controls the ceiling camera 310 installed in the store 20 to discover and track the moving object Mo in the store 20 shown in FIG.
The server 1, the cashier terminals 2-1 to 2-n, the sales floor device 3, and the inspection terminal Q are interconnected via a network N such as the Internet (Internet) line. The inspection terminal Q can be carried by a clerk who is remote from the cashier terminal 2, prepared for a back yard of a store, or provided for a call center remote from the store.

Note that, for convenience of explanation, only one server 1 in FIG. 16 is drawn, but there may be a plurality of servers 1 in practice.
Hereinafter, when it is not necessary to individually distinguish the cashier terminals 2-1 to 2-n, these will be collectively referred to as "cashier terminals 2".

The server 1 executes each process in order to manage each operation of the cashier terminal 2 and the counter device 3.
The server 1 includes a CPU 101, a ROM 102, a RAM 103, a bus 104, an input / output interface 105, an output unit 106, an input unit 107, a storage unit 108, a communication unit 109, and a drive 110. I have. These are configured similarly to the server 1 described in the first embodiment illustrated in FIG.

The sales floor device 3 is connected to such a server 1 via a network N.
FIG. 17 is a block diagram illustrating a hardware configuration of the sales floor device 3 in the product recognition system of FIG.

  The sales floor device 3 includes a CPU 301, a ROM 302, a RAM 303, a bus 304, an input / output interface 305, a ceiling camera 310, a shelf camera 311, a communication unit 315, and an information terminal 9.

  The CPU 301, the ROM 302, the RAM 303, the bus 304, the input / output interface 305, and the communication unit 315 of the sales floor device are configured similarly to those of the server 1 shown in FIG.

The ceiling camera 310 is connected to a network by a USB (Universal Serial Bus) cable.
As the shelf camera 311, a camera that can capture a wide angle image such as a fisheye camera may be employed.
The shelf camera 311 is connected to a network via a USB cable.

The information terminal 9 is an information device such as a smartphone or a tablet including the remote control unit 390 and the display unit 391. The remote control unit 390 has a function of resolving an error state such as a system processing error by remote control. The display unit has a screen for displaying an error state, a moving object Mo, and the like. In addition, the information terminal 9 includes a sound generation unit (not shown) for notifying an error state.
The error state in the store includes, for example, a case where the ceiling camera 310 cannot recognize the number of books taken from the shelf or a case where an unpaid book is taken out of the store. In addition, as an error state at the cashier terminal, when the cashier terminal 2 cannot identify a book or recognize the number of books, when an age-restricted product is settled, or when a book is left in the cashier terminal 2 And so on.
Further, the server 1 is provided with an error display unit 151 for displaying such an error and an error canceling unit 152 for canceling the error state.
In the present embodiment, when the cashier terminal 2 cannot identify a book or recognize the number of books, or when an age-restricted product is to be settled, a request is made for inspection by the inspection terminal Q. By doing so, it is also possible to eliminate the error state.

The cashier terminal 2 is connected to such a sales floor device 3 via the network N.
The cashier terminal 2 is configured similarly to the cashier terminal of the first embodiment shown in FIG.
Therefore, the cashier terminal 2 of the second embodiment includes a CPU 201, a ROM 202, a RAM 203, a bus 204, an input / output interface 205, an output unit 206, an input unit 207, an illumination unit 221, a light shielding unit 209, The display unit 210 includes a presentation unit 210, a cash register camera 211, a storage unit 208, a communication unit 213, and a drive 214.

  The cashier camera 211 captures an image of a book placed in the predetermined area A, and outputs a captured image obtained as a result to the image acquisition unit 232 in the CPU 201 as an object captured image. If the cash register camera 211 can capture images at a wide angle like a fish-eye camera, or if it specializes in capturing only the spine of a book, only one cash register camera 211 may be provided.

  Since the hardware configuration of the inspection terminal Q is the same as the hardware configuration of the server 1, the hardware configuration of the inspection terminal Q should be referred to FIG. Description is omitted.

FIG. 18 is a functional block diagram illustrating an example of a functional configuration of the server 1, the cashier terminal 2, the sales floor device 3, and the inspection terminal Q.
The CPU 101 of the server 1 includes an error determination unit 150.
In one area of the storage unit 108 of the server 1, a product DB 131 and a position information management DB 132 are provided.
The product DB 131 is a DB (Data Base) that stores information on book titles, prices, author names, publishers, and the like. The position information management DB 132 manages the position of the moving object Mo.

In the CPU 201 of the cashier terminal 2, as shown in FIG. 18, the light emission control unit 228, the light blocking control unit 229, the presentation control unit 230, the image acquisition unit 232, the object recognition unit 233, and the product identification unit 235 The sale / restricted product determination unit 236, the settlement unit 237, the display control unit 238, and the inspection result acquisition unit 239 function.
In the CPU 101 of the inspection terminal Q, the image display control unit 411 and the inspection result transmission unit 412 function.

The light emission control unit 228 controls switching between a state in which the illumination unit 221 emits light at the timing of imaging the book and a state in which the light emission unit 221 does not emit light at the timing of not imaging the book, and a state in which a book placed in the predetermined area A is recognized. Control for switching the light emission color.
The light-shielding control unit 229 executes control for switching between the opaque state when the light-shielding unit 209 provided in the surrounding unit 270 captures a book placed in the predetermined area A and the transparent state when not capturing an image.
The presentation control unit 230 performs control such that the presentation unit 210 changes the emission color for presenting the state of the cashier terminal 2.

The image acquisition unit 232 acquires data of an object image captured by the cash register camera 211 with the object placed in the predetermined area A.
The object recognition unit 233 recognizes the presence of an object placed in the predetermined area A by using the above-described predetermined image recognition method.

The product specifying unit 235 lists the product candidates for the objects whose presence has been recognized by the object recognizing unit 233 by using an image processing method such as specific object recognition, general object recognition, character recognition, and deep learning. The listed product candidates are referred to as “product candidate list S”. Thereafter, the product specifying unit 235 performs a verification function and specifies the product with high accuracy. Note that the “image recognition method” described below means a method of recognizing something using some image. For example, specific object recognition, general object recognition, character recognition, and deep learning are examples of image recognition methods.
The verification function lists the “product candidate list P” using an algorithm different from the above-described method of listing the product candidates. If the results of the product candidate lists S and P are matched, and the value exceeds a predetermined threshold, the product is specified.
The listing method of the “commodity candidate list” is realized by, for example, a method of matching image information of an object obtained from an object whose existence has been recognized with image information held in the DB information holding unit 241 or a memory. You may. That is, when the feature information of both images match (exceeds the threshold), the object whose existence has been recognized by the object recognition unit 233 is a product registered in the DB information holding unit 241. , It is specified as a product registered in the DB information holding unit 241.
Also in the present embodiment, as in the first embodiment, when a product cannot be specified at the cashier terminal 2, the result is output as a product unspecified.
Then, similarly to the first embodiment, when the specific object inspection mode is set, the product specifying unit 235 outputs the object captured image of the object determined to be a product unspecified and related information to the inspection result obtaining unit. 239 to the inspection terminal Q. On the other hand, when the all-object inspection mode is set, the product specifying unit 235 determines the result of specifying the product, the product captured image of the specified product, the object captured image of the object for which the product is not specified, and related information. Are transmitted to the inspection terminal Q via the inspection result acquisition unit 239.
Further, in the present embodiment, even when the number of products (books) placed on the cashier terminal 2 does not match the number of books counted by the sales floor device 3, the result is the number of products (books). Output as mismatch. In this case, the product specifying unit 235 transmits the object captured image and the number of books counted by the counter device 3 to the inspection terminal Q via the inspection result acquisition unit 239.
Further, the product specifying unit 235 specifies the product based on the inspection result acquired from the inspection terminal Q by the inspection result acquisition unit 239. Specifically, for the product specified by the product specifying unit 235, when the specified result is approved or corrected by the inspection result, the product specifying unit 235 specifies the product according to the content indicated by the inspection result. . In addition, for an object that has been determined to be unspecified by the product specifying unit 235, the product specifying unit 235 specifies the object as a product indicated by the inspection result. If the number of books is not confirmed by the inspection (the mismatch of the number of books is not resolved), an error is displayed and the settlement is stopped.

The sales-restricted product determination unit 236 determines whether or not the product specified by the product specification unit 235 is a sales-restricted product based on the determination information, and presents the product. The sale-restricted merchandise is, for example, a book whose sale is restricted to shoppers under a certain age. When a clerk sells a book, the clerk who looks at the shopper can check the age of the shopper and determine whether or not to sell the book.
However, this system that employs automatic payment, not face-to-face sales, requires a mechanism that allows the clerk to confirm the age of the shopper.
In the present embodiment, the trading-restricted-product determining unit 236 of the cashier terminal 2 that has specified the trading-restricted product outputs the product captured image of the product corresponding to the trading-restricted product via the inspection result obtaining unit 239 to the inspection terminal Q. Send to At this time, the sales-restricted product determination unit 236 appropriately transmits an image (a face image, an image of an identification card, and the like) relating to the shopper to the inspection terminal Q via the inspection result acquisition unit 239. At this time, if the image related to the shopper is unclear, a configuration may be adopted in which the purchaser is notified at the cashier terminal 2 by screen display, voice guidance, or the like, and a request to retake the image is made. Further, a configuration may be adopted in which the purchaser and the examiner talk via the microphone M and the speaker S provided in the cashier terminal 2.
Then, based on the inspection result acquired by inspection result acquisition section 239 from inspection terminal Q, trading restricted product determination section 236 determines whether or not sale of the trading restricted product is permitted (trading restriction is released. Or not).
If the sale of the trade-restricted product is permitted according to the inspection result, the payment of the product is continued. On the other hand, if the sale of the trade-restricted product is not permitted according to the inspection result, the fact that the product is a trade-restricted product is presented, and the settlement process is interrupted. In this case, the clerk who has received the error state may check the age of the shopper and operate the cashier terminal 2 to release the restricted state from the error display section 151. Thereby, the settlement process is restarted. In addition, the cashier terminal 2 includes a cashier camera 211 that captures the shopper's face, hands, and the like. By estimating the shopper's age, the shopper who is determined not to have reached the predetermined age can purchase and sell a restricted product. May not be sold.

Therefore, the sales-restricted product determination unit 236 specifies a sales-restricted product such as an age-restricted book from the information in the DB management unit 141 of the server 1.
The sales-restricted product determination unit 236 may limit the sales by linking it with the shopper information obtained in the personal authentication. When it is determined that the book is a sales-restricted product, the display unit D indicates that the book is a sales-restricted product.

  The settlement unit 237 calculates the total amount of the books specified by the product specifying unit 235 and which can be sold by the trading-restricted product determining unit 236. For example, the settlement unit 237 reads out the price of the book placed in the predetermined area A from the DB information holding unit 241 and adds it, displays the price on the display unit D (FIG. 8), and performs the settlement.

  The display control unit 238 executes control to display the title, price, and the like of the book, which is captured by the cash register camera 211 of the cash register terminal 2 and adjusted by the adjustment unit 237, to the purchaser and the clerk. The display control unit 238 alerts the shopper or the clerk (object) when the product is not specified by the product specifying unit 235, or when the product is not specified by the inspection at the inspection terminal Q. The output unit 206 is controlled so as to output an instruction for re-positioning. Specifically, a configuration may be adopted in which the purchaser is notified by a screen display, voice guidance, or the like, and the purchaser is requested to replace the product. Further, a configuration may be adopted in which the purchaser and the examiner talk via the microphone M and the speaker S provided in the cashier terminal 2.

  The inspection result acquisition unit 239 requests the identification of the product by inspection by transmitting the object captured image or the product identification result and the product captured image of the identified product to the inspection terminal Q. The inspection result transmitted from the inspection terminal Q in response to the request is acquired. The inspection result acquisition unit 239 outputs the acquired inspection result (product identification result) to the product identification unit 235. In addition, the inspection result acquisition unit 239 requests the determination of the sales-restricted product by inspection by transmitting the product image of the product corresponding to the sales-restricted product to the inspection terminal Q, and Get the inspection result sent from. At this time, the inspection result acquisition unit 239 appropriately transmits an image (a face image, an image of an identification card, etc.) relating to the shopper to the inspection terminal Q. The inspection result acquisition unit 239 outputs the acquired inspection result (trading-restricted product determination result) to the trading-restricted product determination unit 236.

In the inspection terminal Q, when an inspection request is made from the cashier terminal 2, the image display controller 411 transmits an image such as an object captured image or a product captured image transmitted from the cashier terminal 2 and these images. Then, various information (commodity candidate list and the like) transmitted along with is output to the output unit 106.
The inspection result transmitting unit 412 is configured to execute an inspection result (specification result of a product or determination of a sales-restricted product) input by the examiner via the input unit 107 with respect to the object captured image or the product captured image output to the output unit 106. Is transmitted to the cashier terminal 2 which has requested the inspection.

  In the CPU 301 of the counter unit 3, as shown in FIG. 18, a personal authentication unit 320, a moving object tracking unit 330, a position information management unit 340, and a book count counting unit 350 function.

The personal authentication unit 320 includes a personal information acquisition unit 321.
The personal authentication unit 320 personally authenticates a shopper registered in the DB management unit 141 of the server 1 from the shopper's personal information acquired by the personal information acquisition unit 321. Note that the personal authentication unit 320 and the personal information acquisition unit 321 may be provided in the cashier terminal 2 as in the first embodiment.

Here, personal information includes not only information that can identify individuals, such as name, gender, date of birth, address, and telephone number, but also biometric information such as fingerprints, veins, and irises, and financial information, such as credit card numbers and bank account numbers. It also includes privacy-related information such as information about privacy.
The personal information acquisition unit 321 is provided, for example, at the gate 21 installed at the entrance 22. As the personal information acquisition unit 321, a reader that touches a portable information terminal such as an IC card, a smartphone, or a tablet of a shopper, or a reader that reads biological information such as a fingerprint, a vein, and an iris is used.

If personal authentication cannot be performed when entering the store, personal authentication may be performed based on the image of the shopper captured by the ceiling camera 310 during shopping.
The acquired personal information is used for trade restrictions (including cancellation) and purchase analysis.

FIG. 19 is a functional block diagram showing a detailed functional configuration example of the moving object tracking unit 330 provided in the sales floor device 3 of FIG.
As shown in FIG. 19, the moving object tracking unit 330 finds a moving object Mo from an image captured by the ceiling camera 310, and tracks the moving object Mo. A moving object area definition unit 3304 using a ceiling camera and a moving object area tracking unit 3305 using a ceiling camera are provided.

  The moving object tracking unit 330 using the ceiling camera is connected to the ceiling camera 310 via a USB cable, a network N, or the like. Therefore, the ceiling camera 310 is linked with another ceiling camera 310, a personal computer, or the like.

  The moving object detection unit 3302 based on the ceiling camera estimates the state of the moving object Mo using a state space model (such as a Bayesian filter) based on the image captured by the ceiling camera 310, and discovers the moving object Mo. A uniquely identifiable ID is assigned.

By the way, in the video from the ceiling camera 310, the peripheral area of the moving object Mo not directly facing the ceiling camera 310 is imaged at an angle (from an oblique direction). is there. Therefore, it is conceivable that the captured image is corrected so as to be directly facing the image. However, even with such correction, it may not be possible to acquire position information with high accuracy.
Therefore, the ceiling camera 310 may acquire high-precision position information by acquiring height information of the moving object Mo using a distance sensor or the like.

  The moving object area definition unit 3304 based on the ceiling camera updates the position information of the area of the moving object Mo after moving. Since the moving object Mo keeps moving, the moving area changes within the range imaged by one ceiling camera 310, and also moves within the range imaged by another ceiling camera 310. Each time the moving object Mo moves, a moving object region is defined, and the position information of each moving object region, such as a position information management DB 132 for managing position information and a memory, is updated.

  The moving object area tracking unit 3305 estimates the position of the moving object area and keeps track of the moving object area of the moving object Mo.

FIG. 20 is a functional block diagram illustrating a detailed functional configuration example of the position information management unit 340 provided in the sales floor device 3 in FIG.
The position information management unit 340 includes an inter-camera information transfer unit 341, a position definition unit 342 for each camera, and a moving object display unit 343.
The camera-to-camera information transfer unit 341 shares the image information captured by each ceiling camera 310 with the image information captured by another ceiling camera 310, thereby capturing the captured image of the ceiling camera 310 including the moving object Mo. , The moving object area can be continuously tracked even if the image is captured by another ceiling camera 310.

The inter-camera information transfer unit 341 exchanges information between the ceiling cameras 310 on the storage unit 108, including the product DB 131, for example, through the server 1 that controls information obtained by being imaged by the ceiling camera 310.
As another example, in consideration of the large number of ceiling cameras 310, the camera information transfer unit 341 receives an image captured by each ceiling camera 310 without passing through the server 1 between the ceiling cameras 310 by, for example, P2P. You may pass it.

The position definition unit 342 of each camera defines position information indicating where in the store each ceiling camera 310 is projected. That is, the position definition unit 342 of each camera grasps where in the store the moving objects imaged by the different ceiling cameras 310 by the inter-camera information transfer unit 341 are located.
The position definition unit 342 of each camera combines the captured images of each ceiling camera 310 and creates one store map. The position definition unit 342 of each camera replaces the coordinates of each ceiling camera 310 and shelf camera 311 with the coordinates on the store map. In addition, the position definition unit 342 of each camera corrects, by calculation, the captured image captured by each ceiling camera 310 so that the captured image directly faces the floor in the store by perspective transformation.

In addition, by installing the distance sensor on the ceiling camera 310 and acquiring height information, the position information management unit 340 accurately corrects the distorted captured image and accurately recognizes the moving object Mo. can do.
The moving object display unit 343 displays the position information captured by the position definition unit 342 of each camera as the moving object Mo in the store 30. The moving object display unit 343 may be employed as the information terminal 9 held by the clerk Mt, a screen of a backyard of the store, or the like.

FIG. 21 is a functional block diagram illustrating a detailed functional configuration example of the number-of-books counting unit 350 provided in the counter device 3 of FIG.
The book number counting section 350 includes a book number recognition section 351, a moving object and book number linking section 352, a book number unspecified determination section 353, a book number management section 354 linked to a person, and a moving object area transfer recognition section by a ceiling camera. 355, and the number-of-books recognition unit 356 transferred by the ceiling camera.

  The number-of-books recognition unit 351 recognizes the number of books taken by the moving object Mo from the shelf and the number of books returned to the shelf from the images captured by the ceiling camera 310. The number-of-books recognition unit 351 detects entry and exit of the moving object area into the shelf by providing an “object entry detection line” or the like in the captured image, and detects the moving object Mo from the detected image at the time of detection. Define the area of the object taken from inside and the object returned to the shelf. The number of the object areas is recognized as the number of books. When the ceiling camera 310 has a zoom function, the book number recognition unit 351 performs the book number recognition after zooming the ceiling camera in consideration of the distance between the ceiling camera 310 and the moving object Mo. Good.

  Further, the book number recognition unit 351 may recognize the number of books taken by the moving object Mo from the shelf and the number of books returned to the shelf by the shelf camera 311. The number of books taken from the shelf and the number of books returned to the shelf may be recognized. At this time, the shelf camera 311 may be a camera that can capture images over a wide range.

The moving object and book number association unit 352 associates the book number recognized by the book number recognition unit 351 with the person who took the book.
When the number of books cannot be recognized, the unspecified number of books determining unit 353 associates that the number of books could not be recognized with the moving object Mo.

  The number-of-books management unit 354 linked to the person always uses the position information management DB 132 and the like to continuously manage the book count list linked to the ID of the moving object Mo. When a book is taken from the shelf, the number of the acquired books is added. Conversely, when a book is returned to the shelf, the number of returned books is subtracted.

When a book is transferred between the moving object regions by the ceiling camera, the book transfer unit 355 transfers the book information between the moving object regions to the moving object Mo to which the book number information linked to each moving object Mo has been transferred.
Note that the transfer recognition unit 355 between the moving object regions by the ceiling camera may analyze the movement of a person and perform the transfer recognition using an object recognition technique such as deep learning. The hand may be recognized, or the overlap between the moving object regions (which may include the hand) may be recognized. The transfer recognition unit 355 between the moving object areas by the ceiling camera may use the shelf camera 311 instead of the ceiling camera 310. At this time, the shelf camera 311 may be a camera that can capture images over a wide range.

The number-of-books received by ceiling camera recognition unit 356 recognizes the number of books when the books are transferred between the moving object area bodies. For example, the number-of-books recognition unit 356 transferred by the ceiling camera recognizes the number of books from the captured image at the time when the transfer is recognized.
The number-of-books recognition unit 356 delivered by the ceiling camera may include a ceiling camera having a zoom function, thereby zooming up on a portion where the delivery is estimated to be performed and recognizing the number of books.
The number-of-books recognition unit 356 transferred by the ceiling camera may recognize the number of books by using the shelf camera 311 that can capture a wide area instead of the ceiling camera 310.
The number-of-books recognition unit 356 transferred by the ceiling camera associates each moving object Mo specified by the transfer recognition unit 355 between moving object regions with the ceiling camera with the number of books recognized here, Update the book list.

Next, a book settlement method in the commodity recognition system according to the second embodiment will be described with reference mainly to FIG. 22 and also to FIG.
FIG. 22 is a flowchart illustrating a book settlement process in the second embodiment.

In step S201, when a shopper (moving object Mo) enters the store from the entrance 22 of the store (FIG. 14), the ceiling camera 310 installed near the entrance 22 starts imaging the shopper.
As the shopper proceeds along the aisle 24, the ceiling camera 310 at the back captures the shopper. In this way, the plurality of ceiling cameras 310 constantly image the shopper. The gate 21 may be provided near the entrance 22.
When a gate 21 is provided in the entrance 22, the gate 21 is always closed, but is opened when a shopper enters the store, and is closed after entering the store. In addition, before step S201, the personal authentication unit 320 may perform personal authentication of the shopper and acquire personal information of the shopper.

  In step S202, the moving object discovery unit 3302 using the ceiling camera extracts only the shopper's area (moving object area) imaged in step S201, defines the shopper as a moving object Mo, and assigns an ID to the moving object Mo. The ID number and the location information in the store associated with the ID are registered in the location information management DB 132 and the RAM 303 of the sales floor device 3.

  In step S203, when the moving object region definition unit 3304 using the ceiling camera moves within the range imaged by the ceiling camera 310, the position of the region of the moving object Mo after the movement is defined again. The position information is managed in a position information management DB 132 for managing the position of a person, a memory, and the like, and is updated for each area definition. This defined position is also recognized at a position where another ceiling camera 310 is capturing an image.

  In step S204, the moving object area tracking unit 3305 using the ceiling camera estimates the position of the moving object area and keeps track of the moving object Mo, based on the fact that the shopper moves along the passage 24 aiming at the book to be purchased. In step S201, the moving object Mo is constantly imaged. In step S202, an ID is issued to the moving object Mo. In step S203, the position information of the area of the moving object Mo after the movement is updated. Since the position information is exchanged by the mobile terminal 341, even if the moving object Mo moves along the passage 24 and is imaged by the different ceiling camera 310, the moving object region tracking unit 3305 using the ceiling camera keeps tracking the moving object Mo. Can be.

In step S205, the number-of-books recognition unit 351 recognizes the number of books taken from the shelf and returned to the shelf by the moving object Mo whose ID has been issued.
If the number of books taken cannot be counted in step S205, an unspecified number-of-books determination unit (not shown) outputs an error.

  In step S206, the moving object and book number associating unit 352 associates the book number recognized (counted) in step S205 with the moving object Mo. Therefore, the number-of-books recognition unit 351 recognizes how many books the uniquely specified shopper has taken.

  In step S207, the number-of-books management unit 363 linked to a person continues to manage the number of books linked to a person using the DB management unit 141 of the server 1. Therefore, even when the book taken by the shopper from the shelf is returned to the shelf, the book number management unit 363 associated with the person recognizes the book number of the book held by the shopper.

  When the shopper finishes taking the book, he goes to the cashier terminal 2 and places the book in a predetermined area A of the cashier terminal 2. As shown in FIG. 15, the book is placed so that the spine is photographed by at least one or more cameras 211 of the cashier terminal 2. Then, the shopper presses a button as the input unit 207 provided on the cashier terminal 2 or the like. The cashier terminal 2 specifies the product by using the pressing of the button as a trigger. Product specification is performed by making full use of an image recognition method and specifying a book name from the cover or spine of the book.

In step S210, the merchandise identification unit 235 of the cashier terminal 2 determines whether the booklet number information associated with the moving object Mo recognized in step S207 matches the booklet number information of the book placed in the predetermined area A of the cashier terminal 2. Verify whether. The specific flow of this verification will be described later with reference to FIG.
If the number-of-books information associated with the moving object Mo does not match the number-of-books information of the books placed in the predetermined area A of the cashier terminal 2, a request for inspection (confirmation of the number of books) at the inspection terminal Q is made. If the number of books is not confirmed by the inspection (the mismatch of the number of books is not resolved), an error is displayed on the error display unit 151, and the settlement cannot be performed. Further, if an attempt is made to leave the store in that state, the error display unit 151 installed near the entrance 22 issues a warning by sound, light, or the like. When the gate 21 is provided, the closed state is maintained.
In step S210, the trading restricted product determination unit 236 determines whether the book placed on the cashier terminal 2 corresponds to the restriction. By acquiring personal information in advance at any timing up to step S207 (including before step S201), the sales-restricted merchandise determination unit 236 determines whether a shopper who cannot purchase a restricted book is trying to purchase. , An error is displayed by the error display unit 151.

In step S211, the settlement unit 237 of the cashier terminal 2 adjusts the total amount of the books placed in the predetermined area A. That is, as described in the first embodiment, information such as the price of a book is obtained by the product specifying unit 235, and the total amount is calculated. The shopper pays by credit card, electronic money, or the like, and the settlement is completed.
Note that, when the settlement is completed, the DB management unit 141 updates the book count count list linked to the person to the state of “settlement completed”.

In addition, when the shopper tries to take out an unpaid book near the entrance 22, that is, when the book count count list associated with the moving object Mo is not “paid”, the take-out detection unit Detects this, and the error display section 151 near the entrance 22 or the presentation section 210 of the cashier terminal 2 presents the sound or light. Further, when the gate 21 is installed, the gate 21 is kept closed, and the shopper cannot leave the store. In addition to keeping the gate 21 closed, the error display unit 151 or the like may issue a warning by sound or light. Thereby, shoplifting can be prevented.
The error display unit 151 and the presentation unit 210 of the cashier terminal 2 may be configured to emit light in different colors according to the error state.

FIG. 23 is a flowchart illustrating verification of book number information and the number of books to be adjusted in step S210 in FIG.
In step S221, when the reading start button of the input unit 207 of the cashier terminal 2 is pressed, the image obtaining unit 232 of the cashier terminal 2 obtains an image of the object placed in the predetermined area A of the cashier terminal 2 and then displays the image. The product specifying unit 235 specifies which product is the recognized object, and also recognizes the number of products.
In step S222, the person of the book number counting unit 350 of the counter device 3 and the book number associating unit 362 refer to the information of the book number placed on the cashier terminal 2 from the ID of the moving object Mo.

In step S223, the product identification unit 235 determines whether the number of books placed on the cashier terminal 2 matches the number of books counted by the book number management unit associated with a person.
If the number of books placed on the cashier terminal 2 matches the number of books counted by the number-of-books management unit linked to a person (YES), the process proceeds to step S226.
If the number of books placed on the cashier terminal 2 does not match the number of books counted by the number-of-books management unit linked to a person (NO), the process proceeds to step S224.
In step S224, the product specifying unit 235 transmits the object captured image and the number of books counted by the counter unit 3 to the inspection terminal Q via the inspection result acquisition unit 239, and thereby the inspection terminal Q Request the inspection (confirmation of the number of books) in the above, and obtain the inspection result.
In step S225, the product specifying unit 235 determines whether or not the inspection result indicating that the mismatch in the number of books has been resolved has been obtained.
If it is determined in step S225 that the inspection result indicating that the discrepancy in the number of books has been resolved has not been obtained, the error display unit 151 of the cashier terminal 2 displays an error and issues a warning.
At this time, a configuration may be adopted in which the purchaser and the examiner talk via the microphone M and the speaker S provided in the cashier terminal 2.
If an attempt is made to exit the store with the warning being issued, the error display section 151 near the entrance 22 issues a warning by sound or light. When the gate 21 is installed, the gate 21 is kept closed, and the shopper cannot leave the store. Thereby, shoplifting can be prevented.
If it is determined in step S225 that the inspection result indicating that the discrepancy in the number of books has been resolved is obtained, the process proceeds to step S226.
In step S226, the product specifying unit 235 determines whether the mode is set to the specific object inspection mode or the all object inspection mode.
If the specific object inspection mode has been set, the process proceeds to step S227.
If the all-object inspection mode has been set, the process proceeds to step S228.
In step S227, the inspection result acquisition unit 239 requests an inspection (specification of a product) at the inspection terminal Q for the target book, and acquires the inspection result.
After step S227, the process proceeds to step S229.
In step S228, the product specifying unit 235 determines whether any of the books placed in the predetermined area A of the cashier terminal 2 cannot be identified as any of the books. When there is a book which the product specifying unit 235 cannot specify which book is present (YES in step S228), the process proceeds to step S227.

  If there is no book whose product identification unit 235 cannot identify which book in step S228 (NO in step S228), the product identification unit 235 reads the book held in the DB information holding unit 241 or the storage unit of the server 1. Identify the book, including information such as name, price, and sale-restricted product. Thereby, the process proceeds to step S229. Information on the specified book may be output to the display control unit 238.

  In step S229, the sales-restricted product determination unit 236 determines whether the book specified by the product specification unit 235 is a book for which age confirmation is required.

If it is determined in step S229 that the book specified by the product specifying unit 235 is a book for which age confirmation is required, that is, if it is determined as YES, the process proceeds to step S230.
In step S229, if it is determined that the book specified by the product specifying unit 235 is not a book requiring age confirmation, that is, if it is determined as NO, the process returns to the book automatic payment process.
In step S230, the display control unit 238 causes the display unit D of the cashier terminal 2 to display a screen for confirming the age. However, if the shopper's personal information is acquired and there is no need to confirm the age here, step S230 is skipped and the process proceeds to step S234.
At this time, when there is a problem with the age verification of the purchaser, for example, when the image of the purchaser's face or the image of the age confirmation document is unclear, the purchaser is notified by a screen display or voice guidance at the cashier terminal 2. Alternatively, a configuration may be employed in which the purchaser is requested to retake the image. Further, a configuration may be adopted in which the purchaser and the examiner talk via the microphone M and the speaker S provided in the cashier terminal 2.
In step S231, the inspection result acquisition unit 239 requests an inspection (judgment of a trading restricted product) at the inspection terminal Q for the target book, and acquires the inspection result.
In step S232, the trading restriction product determination unit 236 determines whether an instruction to cancel the trading restriction has been received.

If it is determined in step S232 that an instruction to cancel the trading restriction has not been received, the process proceeds to step S233.
If it is determined in step S232 that an instruction to release the trade restriction has been received, that is, if it is determined as YES, the process proceeds to step S234.
In step S233, the inspection result transmitting unit 412 of the inspection terminal Q transmits a warning that the trading restriction has not been released by the inspection result. Upon receiving this warning, for example, the display control unit 238 of the cashier terminal 2 presents, via the output unit 206, a warning that the trade restriction has not been released by the inspection result. After step S234, the process ends, and the settlement stops.
In step S234, the trading restriction product determination unit 236 releases the trading restriction.
When step S234 is completed in this way, or when it is determined in step S229 that the product is not an age-restricted product (determined as NO), the process returns to the book automatic payment process.
In this way, the information processing system recognizes the number of books taken by the shopper in the store and determines whether the number matches the number of books placed on the cashier terminal. In addition, at this time, if the item is to be inspected (for example, if the number of books does not match, if the product cannot be specified at the cashier terminal 2, or if the book corresponds to a sale-restricted product, etc.), the inspection is performed. Inspection is performed in the terminal Q, and automatic settlement is performed in accordance with the inspection result (the result of checking the number of books, the result of specifying a book, the result of determining a sale-restricted product, etc.).
Therefore, according to the present information processing system, when a shopper purchases a product such as a book, it is possible to automate the settlement of the price of the product, and to improve the accuracy of specifying the product.

  In the second embodiment, the errors assumed in tracking the moving object Mo (the shopper) include (A) when the shopper enters the store 20 from the entrance 22, the moving object discovery unit 3302 using the ceiling camera moves. When no object is detected, (B) when the moving object area tracking unit 3305 using the ceiling camera loses track of the moving object Mo being tracked, (C) the moving object Mo is tied to each of two or more different moving objects Mo. This includes, but is not limited to, a case where the ID is changed during the tracking.

In accordance with each error state, the system adopted in this embodiment performs various measures including the following examples.
(A) An image including a shopper as a subject imaged by the ceiling camera 310 is transmitted to the inspection terminal Q, and a request for detection of the shopper by the inspection is made. When the examiner detects a shopper, a new moving object Mo is defined, an ID is issued, and tracking by the moving object region tracking unit 3305 by the ceiling camera is started. If the patron does not detect the shopper, it informs the clerk to that effect.
(B) When a shopper who is not associated with any ID registered in the location information management DB 132 is recognized, a captured image of the shopper and a list of ID candidates to be associated are displayed on the inspection terminal Q. Sent to. The examiner links the most appropriate ID to the shopper based on the past linking information between the shopper and the ID, and starts re-tracking by the moving object area tracking unit 3305 using the ceiling camera. If re-tracking could not be started for any reason, a clerk is notified to that effect.
In addition, when an ID that has lost the association with the shopper is detected, the ID and a list of captured images of the shopper who are not associated with any ID are transmitted to the inspection terminal Q. The examiner attempts to link the shopper to the ID based on the link information between the shopper and the ID in the past, and when the link is successful, the moving object area tracking unit 3305 by the ceiling camera Start retracking. If the start of re-tracking fails for any reason, the clerk is informed accordingly.
(C) The image list of each shopper and the list of IDs to be associated with each shopper are transmitted to the inspection terminal Q. The examiner attempts to allocate the ID to the shopper most appropriately based on the past association information between the shopper and the ID. When the ID can be appropriately assigned to the shopper, re-tracking by the moving object area tracking unit 3305 by the ceiling camera is started. If the ID cannot be appropriately assigned to the shopper, the ID is abandoned and the clerk is notified that the link has been changed.

[Embodiment 3]
FIG. 24 is a diagram illustrating a layout example of a supermarket that employs the product recognition system according to the third embodiment.
The product recognition system according to the third embodiment is a system applied to a store 30 such as a supermarket as shown in FIG.

  In the store 30, a sales floor is provided from an entrance 31 to an exit 32. In the counter, a plurality of shelf racks 33 for displaying products are installed. A passage 34 is formed between the two facing racks 33.

  At the entrance 31, shopping baskets and baskets (not numbered) such as shopping carts are placed. Before the exit 32, a settlement area 35 is provided. In the settlement area 35, a plurality of cash register stands 36 are installed. The cash register stand 36 includes n settlement machines 4. The third embodiment does not include the cashier terminal 2 employed in the first and second embodiments. In the third embodiment, unlike the second embodiment in which a shopper is treated as a moving object Mo, baskets grasped (discovered, tracked, and the like) on the system are treated as a moving object Mo.

The shopper enters the store from the entrance 31 and proceeds along the passage 34 with baskets. The shopper takes the goods in the shelves, places them in baskets, and proceeds along the aisle 34. When the shopper takes all the products he / she wants to purchase, the shopper proceeds to the settlement area 35 and makes payment with the payment machine 4.
In the passage 34, the settlement area 35, and the like, the store clerk Mt is on a tour. The clerk Mt has an information terminal 9a. The information terminal 9a is a portable information processing terminal such as a smartphone, and has a screen or the like for displaying a state in a store.
In FIG. 24, the inside of the shop 30 but the outside of the shop 30 and the backyard of the shop 30 are drawn inside the diagram drawn in a cloud shape. Outside the store 30, the server 1 (FIG. 7) is installed. In a management center, a store 30 backyard, or the like outside the store 30, the store clerk Mt monitors the inside of the store 30 through a large monitor screen (not shown) or a screen of the information terminal 9b. Hereinafter, when it is not necessary to distinguish between the information terminal 9a and the information terminal 9b, they are collectively referred to as "information terminal 9". Although not shown in FIG. 24, in the present embodiment, similarly to the first and second embodiments, the inspection terminal Q is installed.

  A plurality of ceiling cameras 310 are separately installed on a ceiling above a passage 34, a shelf rack 33, and other arbitrary positions in the store 30. The ceiling camera 310 captures an image of the passage 34, the shelf rack 33, and other predetermined areas below. That is, when the moving object Mo enters, the ceiling camera 310 captures an image of the predetermined area including the moving object Mo as a subject.

Further, a shelf camera 311 as an example of a sensing device is installed at each of a plurality of locations on each shelf in each shelf rack 33. The shelf camera 311 captures images of the inside of the shelf, products on the shelf, and other predetermined areas. Also, when a shopper's hand or the like enters a predetermined area in the shelf or an object is taken from the shelf, the shelf camera 311 captures the hand or the object in the shelf as an object image. I do.
A basket camera 312 (not shown in FIG. 24) may be attached to each of the baskets as an example of a sensing device.
In that case, one or more basket cameras 312 of the sales floor device 3 are installed in the basket, and always take an image of the object put in the basket. The basket camera 312 is installed so that the inside of the basket can be imaged without blind spots. The basket camera 312 captures an image of at least a part of the object placed in the basket, such as the front, of the object.

  The basket camera 312 is linked with the ceiling camera 310 and the shelf camera 311 through the network N. By this cooperation, the sales floor apparatus 3 can share both the captured image captured by the basket camera 312 and the captured image of the object captured by the ceiling camera 310 or the shelf camera 311. By this sharing, it is possible to increase the accuracy of specifying a product from the object captured in the captured image.

The product recognition system according to the third embodiment has a function of keeping track of the moving object Mo even if the moving object Mo captured by the ceiling camera 310 moves. The information processing system according to the third embodiment has a function of specifying which product the object taken from the shelf rack 33 is from an image captured by the shelf camera 311.
The product recognition system according to the third embodiment may have a function of specifying which product is an object put in a basket from an image captured by the basket camera 312.
The merchandise recognition system of the third embodiment has a function of specifying which merchandise the object in the shelf is, and further automatically adjusting the merchandise in the checkout machine 4. At the time of the automatic settlement, the information of the product linked with the moving object Mo is read out by the settlement machine 4, and the automatic settlement becomes possible.
The checkout machine 4 includes functions necessary for completing the shopping, such as display of the total amount, points, and details of the purchased products, display of the determination of the sales-restricted products, and a settlement function.

  The ceiling camera 310, the shelf camera 311 and the basket camera 312 are incorporated in the sales floor apparatus 3 as shown in FIGS. The sales floor device 3 has a function for specifying a product from images captured by the ceiling camera 310, the shelf camera 311 and the like, and a function for finding and tracking the moving object Mo. This sales floor device 3 is incorporated in a product recognition system as shown in FIG.

FIG. 25 is a configuration diagram illustrating a configuration of the product recognition system according to the third embodiment.
The product recognition system according to the third embodiment includes a server 1, a sales floor device 3, a checkout machine 4, and an inspection terminal Q. In the third embodiment, a counter 4 is provided in place of the cashier terminal 2 described in the first and second embodiments, in order for the sales floor device 3 to specify which product the object is.
In FIG. 25, only one server 1 and sales floor device 3 are drawn, but there may be a plurality of devices in practice. Further, hereinafter, when it is not necessary to distinguish the settlement machines 4 individually, these are collectively referred to as “payment machines 4”.
The server 1, the sales floor device 3, the checkout machine 4, and the inspection terminal Q are mutually connected via a network N such as the Internet.
The server 1 is configured similarly to the server 1 of the first embodiment (FIG. 7). The inspection terminal Q has the same configuration as the inspection terminal Q of the first embodiment.

FIG. 26 is a block diagram showing a hardware configuration of the sales floor device 3 in the product recognition system of FIG.
The sales floor device 3 includes a CPU 301, a ROM 302, a RAM 303, a bus 304, an input / output interface 305, a ceiling camera 310, a shelf camera 311, a basket camera 312, a communication unit 315, and the information terminal 9. Prepare.

The CPU 301, the ROM 302, the RAM 303, the bus 304, the input / output interface 305, and the communication unit 315 of the counter device 3 are configured in the same manner as those of the server 1 shown in FIG.
The ceiling camera 310, shelf camera 311, communication unit 315, and information terminal 9 of the sales floor device 3 are configured in the same manner as those of the sales floor device 3 (FIG. 17) described in the second embodiment.

  FIG. 27 is a functional block diagram illustrating an example of a functional configuration of the server 1, the sales floor device 3, the checkout machine 4, and the inspection terminal Q.

The server 1 includes a CPU 101, a storage unit 108, a communication unit 109, an error display unit 151, and an error release unit 152.
These are configured similarly to the second embodiment shown in FIG.

In the inspection terminal Q, when an inspection request is made from the sales floor device 3 or the checkout machine 4, the image display control unit 411 controls the object imaging image or the product imaging transmitted from the sales floor device 3 or the checkout machine 4. Images such as images and various information (such as a product candidate list) transmitted along with these images are output to the output unit 106.
The inspection result transmitting unit 412 is configured to execute an inspection result (specification result of a product or determination of a sales-restricted product) input by the examiner via the input unit 107 with respect to the object captured image or the product captured image output to the output unit 106. The result is transmitted to the sales floor apparatus 3 or the checkout machine 4 which has requested the inspection.

  In the CPU 301 of the sales floor device 3, as shown in FIG. 27, a personal authentication unit 320, a moving object tracking unit 330, a position information management unit 340, a shelf product recognition unit 360, a basket product recognition unit 370, The restricted product determination unit 380 and the inspection result acquisition unit 301a function.

The personal authentication unit 320 includes a personal information acquisition unit 321.
The personal authentication unit 320 and the personal information acquisition unit 321 are configured in the same manner as in the second embodiment shown in FIG.

FIG. 28 is a functional block diagram illustrating a detailed functional configuration example of the moving object tracking unit 330 provided in the sales floor apparatus 3 of FIG.
As shown in FIG. 28, the moving object tracking unit 330 includes a moving object finding unit 3302 using a ceiling camera, a basket finding unit 3303 using a ceiling camera, a basket area defining unit 3306 using a ceiling camera, and a car area tracking unit using a ceiling camera. 3307, a grouping unit 3308, an inter-car area transfer recognition unit 3310 by the ceiling camera, an object recognition unit 3312 transferred by the ceiling camera, and a product specification unit 3313 transferred by the ceiling camera.
However, unlike the second embodiment in which the shopper is treated as a moving object, the moving object tracking unit 330 according to the third embodiment treats a basket grasped (found, tracked, etc.) on the system as a moving object Mo.

  The moving object tracking unit 330 is connected to the ceiling camera 310 via a USB cable, a network N, or the like. Therefore, the ceiling camera 310 is linked with another ceiling camera 310, a personal computer, or the like.

  The moving object detection unit 3302 using the ceiling camera finds an object (shopper, basket, cart, etc.) moving in the store using a state space model (eg, Bayesian filter) based on the image captured by the ceiling camera 310. .

The ceiling camera-based basket finding unit 3303 finds baskets (moving objects Mo) from objects moving in the store 30 found by the moving object finding unit 3302, and assigns individual IDs to the moving objects Mo. . The ID of the moving object Mo is continuously used until a predetermined timing such as closing the store or completing settlement.
As a method of finding a moving object Mo from an object moving in the store 30, for example, one or more markers holding information that can be individually identified are attached to each basket, and a basket finding unit 3303 by a ceiling camera is used. Discovers the moving object Mo using this marker as a mark. The marker is not limited as long as it can identify the moving object Mo, such as a two-dimensional code or a characteristic shape.
As another method of finding the moving object Mo from the object moving in the store 30, the basket finding unit 3303 using the ceiling camera may use information unique to the cage such as color information and shape information of the baskets. At this time, the basket finding unit 3303 using the ceiling camera can find the moving object Mo because the color and shape of the baskets can be distinguished from the floor or the like.

As a method of finding the moving object Mo from the object moving in the store 30, thermography or the like may be used by lowering the temperature of the baskets. At this time, the car discovery unit 3303 using the ceiling camera discovers the moving object Mo that has been cooled down based on the temperature difference from the area other than the baskets.
As a method of finding the moving object Mo from the object moving in the store 30, a harmless gas may be generated from the baskets, and thermography or the like may be used. At this time, the car discovery unit 3303 using the ceiling camera detects a temperature change associated with the generation of harmless gas generated from the basket by thermography or the like, and discovers the moving object Mo.

As a method of finding the moving object Mo from the object moving in the store 30, sound may be generated from the basket regardless of whether or not the sound is audible. At this time, the car discovery unit 3303 using the ceiling camera detects the moving object Mo by detecting the sound generated from the baskets.
As a method of finding the moving object Mo from the object moving in the store 30, a sensor that can apply an invisible paint to the basket and recognize the invisible paint may be used. At this time, the car discovery unit 3303 by the ceiling camera discovers the moving object Mo by the sensor recognizing the invisible paint applied to the baskets.
As a method of finding the moving object Mo from the object moving in the store 30, the ceiling camera 310 may irradiate baskets with visible light, infrared light, or the like. At this time, the car discovery unit 3303 by the ceiling camera discovers the moving object Mo by receiving visible light or infrared reflected light applied to the car.

The ceiling camera-based car area definition unit 3306 defines, as a region of the moving object Mo, a certain range in the captured image in the vicinity of the moving object Mo found by the ceiling camera-based basket finding unit 3303. In order to define the region of the moving object Mo, for example, when a marker is attached to a basket as described above, a certain range from the attached position where the marker is attached is defined as the region of the moving object Mo.
When a plurality of markers are attached to the carts, the basket area defining unit 3306 based on the ceiling camera determines that each marker has its own position in the area of the moving object Mo even if the ceiling camera 310 cannot image all the markers. Since the information is held, the area of the moving object Mo may be defined from one or more markers.

The car area definition unit 3306 using the ceiling camera may use a technique that complements the technique for defining the car area described above. As a complementary method, in the case where the temperature of the basket is lowered as described above, the low-temperature area is defined as the area of the moving object Mo from thermography and image recognition.
As another complementary method, when harmless gas is generated from baskets as described above, the temperature change accompanying the gas generation is detected by thermography, etc., and by making full use of image recognition technology, the temperature change Is defined as a region of the moving object Mo.

As yet another complementary method, the basket area defining unit 3306 by the ceiling camera estimates the edge of the basket from the application position of the paint when the invisible paint is applied to the edge of the basket as described above, A region closed by the edge is defined as a region of the moving object Mo.
As yet another complementary method, the basket area defining unit 3306 using the ceiling camera, when irradiating visible light or infrared light from the ceiling camera 310 as described above, determines the area of the moving object Mo from the measurement result of the reflection. Define.

  The basket area tracking unit 3307 based on the ceiling camera estimates (defines) the position of the area of the moving object Mo. The basket area tracking unit 3307 based on the ceiling camera tracks the moving object Mo with the same ID from the time when the moving object Mo is discovered to a predetermined time such as when the store is closed or the settlement is completed, and keeps grasping the position information.

  For this reason, the car area tracking unit 3307 based on the ceiling camera always keeps track of the car area in the captured image, for example, in cooperation with a large number of ceiling cameras 310. In order for a large number of ceiling cameras 310 to cooperate, the cage area tracking unit 3307 based on the ceiling camera performs processing such that an image captured by a certain ceiling camera 310 is taken over by an image captured by an adjacent ceiling camera. I do. The cage area tracking unit 3307 based on the ceiling camera stores the position information of the tracked moving object Mo in the position information management DB 132 or the storage unit 108 of the server 1.

As another method for the basket area tracking unit 3307 using the ceiling camera to track the moving object Mo, a marker having information for identifying the moving object Mo is attached to each basket, and the ceiling camera 310 is configured to move the moving object including the marker. Image Mo. The basket area tracking unit 3307 based on the ceiling camera extracts the marker from the captured image, finds the moving object Mo, and acquires the position information.
The basket area tracking unit 3307 using the ceiling camera finds the marker from the captured image and acquires the position information of the moving object Mo, so that even if the moving object Mo moves, it can keep tracking the moving object Mo.

The basket area tracking unit 3307 using the ceiling camera tracks the moving object Mo by using an object tracking technique in an image such as a Bayesian filter, a fast Fourier transform, or TLD as another method for tracking the moving object Mo. Is also good.
Alternatively, the basket area tracking unit 3307 based on the ceiling camera estimates that the moving objects Mo having acquired the same feature data are the same moving objects Mo based on the feature data such as colors and shapes of the baskets. The moving object Mo may be tracked. At this time, the basket area tracking unit 3307 using the ceiling camera continues to collect the feature data of the tracking target.

In any case, in the video from the ceiling camera 310, since the moving object Mo not directly facing the ceiling camera 310 is imaged at an angle (from an oblique direction), there is a possibility that position information cannot be accurately acquired. Therefore, it is conceivable that the captured image is corrected so as to be directly facing the image. However, even with such correction, it may not be possible to acquire the position information of the moving object Mo with high accuracy.
Therefore, the basket area tracking unit 3307 based on the ceiling camera acquires high-precision position information by acquiring height information of the moving object Mo using a distance sensor or the like. In this way, the basket area tracking unit 3307 using the ceiling camera may continue to track the moving object Mo.

  The grouping unit 3308 may link a plurality of baskets (moving objects Mo). By being linked in this way, the payment machine 4 installed in the payment area 35 can collectively settle the product list of each moving object Mo.

  The ceiling camera handover recognition unit 3310 recognizes, by the ceiling camera 310 or the like, that an object has been delivered (moved in or out) between the moving objects Mo. The inter-car area transfer recognition unit 3310 by the ceiling camera may recognize the transfer by recognizing the overlap between the areas of the moving object Mo. The cargo area transfer recognition unit 3310 by the ceiling camera specifies the moving objects Mo to which the objects have been transferred, and reads a product list associated with the ID of each moving object Mo.

The object recognition unit 3312 transferred by the ceiling camera defines an area of the object from the captured image at the time when the transfer is recognized.
The object recognizing unit 3312 transferred by the ceiling camera may use the ceiling camera 310 as a zoomable camera, zoom up on a portion estimated to have been transferred, and define a region of the object.

The product specifying unit 3313 transferred by the ceiling camera specifies, from the image after defining the object region, any of the products in the product list associated with the read moving object Mo, from the image after defining the object area. The moving object Mo identified by the ceiling camera handover recognition unit 3310 is linked to the product identified by the delivery, and the list of products linked to each moving object Mo is updated.
The object recognition unit 3312 transferred by the ceiling camera and the product specifying unit 3313 transferred by the ceiling camera may be realized by a shelf camera 311 capable of capturing a wide area instead of the ceiling camera 310.

The position information management unit 340 has the same configuration as the position information management unit 340 described in the second embodiment.
That is, as shown in FIG. 20, the position information management unit 340 includes an inter-camera information transfer unit 341, a position definition unit 342 of each camera, and a moving object display unit 343.

FIG. 29 is a functional block diagram showing a detailed functional configuration of the shelf product recognition unit 360 provided in the counter device 3 of FIG.
The shelf product recognition unit 360 includes a shelf camera-based object recognition unit 3602, a shelf camera-based product identification unit 3603, a basket-product linking unit 3605, a basket-linked product list management unit 3607, and a shelf camera-based object entry / exit. A detection unit 3608, a product unspecified determination unit 3609, a label recognition unit 3610, a discount seal recognition unit 3611, and a basket entry / exit detection unit 3612 using a shelf camera or a ceiling camera are provided.
The shelf product recognition unit 360 is linked with another camera, a personal computer, or the like via the shelf camera 311 and a USB cable or a network.

The object recognition unit 3602 by the shelf camera compares the images before and after the image in which the object is taken from the shelf or the object is placed (returned) in the shelf, and determines the image area to be specified for the product. Define (area definition).
That is, the object recognizing unit 3602 by the shelf camera can obtain an image of the object before the object is taken from the shelf or placed on the shelf, and an image of the object after the object is taken from the shelf or placed on the shelf. The object images before and after the change of the image are compared, and the changed image area is specified. The object recognition unit 3602 using the shelf camera confirms a change in each of the RGB data when specifying the image area.

In addition, the object recognition unit 3602 by the shelf camera uses the object entry / exit detection unit as a trigger to detect an image of the object before the object is taken from the shelf or placed on the shelf and the object is taken or taken from the shelf. Alternatively, the area of the object may be defined from only one image of the object using a method different from that before and after the change of the image of the object after being placed.
Since the color data of the object is the same, the object recognition unit 3602 using the shelf camera also defines the area by using the shadow of the object so that it is not determined that there is no change even if there is a change such as entry / exit. May be.
Information on the recognized object area is transferred to the product specifying unit 3603 by the shelf camera.

  The shelf camera-based product specifying unit 3603 specifies which product in the shelf is recognized by the shelf camera-based object recognition unit 3602. The product specifying unit 3603 by the shelf camera lists product candidates by an image processing method such as specific object recognition, general object recognition, and deep learning. The listed product candidates are referred to as “product candidate list S”. After that, the product specifying unit 3603 by the shelf camera specifies the product with high accuracy by performing the verification function.

The verification function lists the “product candidate list P” using an algorithm different from the above-described method of listing the product candidates. If the results of the product candidate lists S and P are matched, and the value exceeds a predetermined threshold, the product is specified.
For example, the listing method of the “commodity candidate list” may be realized by a method of matching image information of an object obtained from an object whose existence has been recognized with image information stored in the product DB 131 or a memory. . That is, when the feature information of the two images match (exceeds the threshold), the object whose existence has been recognized by the object recognition unit 3602 by the shelf camera is a product registered in the product DB 131. The unit 3603 specifies that the product is a product registered in the product DB 131. By creating product candidates by deep learning, and then performing a verification function, products are specified with high accuracy.

  The product identification unit 3603 by the shelf camera does not identify the product in one frame of the image captured by the shelf camera 311 but also makes full use of the image captured by the ceiling camera 310 to identify the product over a plurality of captured images. You may. At that time, the product identification unit 3603 by the shelf camera gives a percentage to the product candidate, adds the percentage based on information such as purchase history, time, place, and personal preference, and exceeds a certain threshold. Identify the product when

  The basket and product linking unit 3605 link the product information of the product specified by the product specifying unit 3603 with the shelf camera and the moving object Mo. As a premise, if only one moving object Mo is present in a predetermined area imaged by one shelf camera 311, the ID of the moving object Mo is specified, and the moving object Mo and the product are identified. It can be linked to information. At this time, the basket and product linking unit 3605 specifies the ID of the moving object Mo by using position information attached to a marker or the like attached to the moving object Mo.

As another premise, when a plurality of moving objects Mo are present in a predetermined area imaged by one shelf camera 311, a basket in / out detection unit using a shelf camera or a ceiling camera described later enters the moving object Mo. As a result of detecting the entry / exit of the object, the moving object Mo containing the object taken from the shelf is specified, and the moving object Mo is associated with the product information.
As another premise, if it is assumed that no moving object Mo exists within a predetermined area imaged by one shelf camera 311, at least one or more shelf cameras 311 or at least one or more The car camera 312 detects entry / exit to the moving object Mo, specifies the moving object Mo, and associates the specified moving object Mo with the product information.

The product list management unit 3607 linked to the basket continues to manage the product list that links the moving object Mo and the specified product until the settlement. That is, the product list management unit 3607 linked to the basket always manages the list of products linked to the ID of the moving object Mo by utilizing the position information management DB 132 and the like.
When the object is taken out of the shelf, the product list management unit 3607 associated with the basket is added by the number of obtained products. Conversely, when the merchandise is returned to the shelf, the merchandise list management unit 3607 associated with the basket subtracts the number of returned merchandise.

The object entry / exit detection unit 3608 using the shelf camera can trigger the activation of the object recognition unit 3602 by the shelf camera by detecting that the object has entered the shelf.
For example, the object entry / exit detection unit 3608 using the shelf camera detects entry of an object into the shelf from a change in image data in an “entry detection area” set in an image captured by each shelf camera 311. The object entry / exit detection unit using the shelf camera also detects that the object has left the area by tracking the entered object in the image.

In order to track an object, the object entry / exit detection unit 3608 based on the shelf camera applies a particle filter to an entry detection area in the shelf. In order to be able to track a plurality of moving objects Mo at the same time as in the case where a plurality of shoppers reach at the same time, after the first object enters, the particles are again scattered in the “entry detection area”. , Preparing for the entry of the next object.
However, assuming that a plurality of objects do not enter the same area, the object entry / exit detection unit 3608 based on the shelf camera does not spread particles to the entry detection area over an area where an object already exists.

The object entry / exit detection unit 3608 using the shelf camera determines that the ratio of particles having a certain likelihood in a predetermined area is equal to or greater than a threshold, and determines that the object has entered the object. Is determined to be “object out”.
The object entry / exit detection unit of the shelf camera detects entry / exit each time a product enters, so that an area of an object with a small amount is estimated at a time.
The images before and after the change are stored in the storage unit 108 of the server 1 so that they can be used in the estimation area.

  The object entry / exit detection unit 3608 using the shelf camera may detect an object entry / exit from spectrum (wavelength) data obtained from a captured image of the object, in addition to the above-described example. Further, the object entry / exit detection unit using the shelf camera may detect the object entry / exit using a method such as a weight / pressure sensor, an infrared sensor, an ethylene gas sensor, or the like.

  The product unspecified determination unit 3609 associates the fact that the product could not be specified by the product specifying unit 3603 with the shelf camera with the moving object Mo that took the object from the shelf.

  Here, a case where a product cannot be specified by the verification function of the product specifying unit 3603 by the shelf camera will be described. First, the product specifying unit 3603 using the shelf camera specifies the number of feature points in which the image of the object taken from the shelf and the image of the object placed on the shelf are similar. For example, the sizes of the objects captured in both images are specified and compared, the color difference is specified, and the colors are compared to determine whether the two objects are similar. If the number of feature points is small, it is not possible to specify a product, and it is possible to prevent an incorrect payment.

  The label recognition unit 3610 recognizes the attached label according to the product specified by the product specifying unit 3603 by the shelf camera. The label recognizing unit 3610 reads out a multidimensional code including a character, a barcode, and the like described on the label by making full use of an image recognition technique, and complements the product specification.

  The discount seal recognition unit 3611 recognizes the discount seal attached according to the product specified by the product identification unit 3603 by the shelf camera. The discount seal recognition unit 3611 specifies the discount amount and the discount rate of the discount sticker attached to the product by making full use of the image recognition technique. The discount seal recognition unit 3611 is executed during the processing of the product specifying unit.

  The basket entry / exit detection unit 3612 using the shelf camera or the ceiling camera detects that a moving object Mo exists in a predetermined area of the shelf camera and the ceiling camera from a marker or the like in which at least one of the shelf camera 311 and the ceiling camera 310 is attached to a basket. It detects whether or not it is moving in and out of the moving object Mo when it does not exist. The basket entry / exit detection unit 3612 using the shelf camera or the ceiling camera may further detect entry of the moving object Mo into an area defined by a marker or the like. At this time, entry of the moving object Mo into the area is performed by comparing the position information of the tracked product with the position information of the entry detection line (eg, edge of a basket) of the area of the moving object Mo.

The information processing system according to the third embodiment may further include a basket product recognition unit 370.
FIG. 30 is a functional block diagram illustrating a detailed functional configuration example of the basket product recognition unit 370 provided in the sales floor apparatus 3 of FIG.
The basket product recognition unit 370 includes a basket entry / exit detection unit 372 with a basket camera, an object recognition unit 373 with a basket camera, a product identification unit 374 with a basket camera, a basket product unspecified determination unit 375, and a label recognition of the basket product. A part 376 and a basket product discount seal recognition part 377 are provided.

  The basket entry / exit detection unit 372 using the basket camera activates the object recognition unit 373 using the basket camera by detecting the entry of an object such as a shopper's hand into a basket (moving object Mo) when the object enters. Can be used as a trigger. The basket entry / exit detection unit 372 of the basket camera detects entry of an object into the basket from a change in an image in an “entry detection area” set in an image captured by each of the basket cameras 312.

The car entering / leaving detection unit 372 of the car camera also detects that the moving object Mo has exited the frame of the captured image by tracking the object entering the frame in the captured image. The car entrance / exit detection unit 372 of the car camera tracks the object using the particle filter.
In order to simultaneously track a plurality of moving objects Mo, the basket in / out detection unit 372 of the basket camera re-particles the entry detection area after the first object enters, and prepares for the entry of the next object.
The basket entry / exit detection unit 372 of the cart camera determines “object-in” when the proportion of particles having a certain likelihood occupies a threshold or more in a predetermined area, and determines “object-in” when it is less than the threshold. Is determined. The basket entry / exit detection unit of the cart camera detects entry / exit each time an object enters / exits so as to estimate an area of the object as small as possible once.

The object recognition unit 373 using the basket camera compares the image when the object such as the shopper's hand enters the basket (moving object Mo) with the image after the object exits from the basket (moving object Mo). , An image area is defined in order to identify an article taken from the shelf and placed in the basket (moving object Mo) (or returned to the shelf from the basket (moving object Mo)). The object recognition unit 373 using the basket camera confirms a change in the area using the RGB data.
The object recognition unit 373 using the basket camera may recognize the object from spectrum (wavelength) data obtained from a captured image of the object, a weight sensor, a pressure sensor, an infrared sensor, a methylene gas sensor, and the like.

  The product specifying unit 374 based on the basket camera specifies which product is placed in the basket. The product specifying unit 374 based on the basket camera lists the product candidates for the object recognized by the object recognizing unit 373 based on the basket camera by using an image processing method such as specific object recognition, general object recognition, and deep learning. The listed product candidates are referred to as “product candidate list S”. Thereafter, the product specifying unit 374 using the basket camera performs a verification function and specifies the product with high accuracy.

The verification function lists the “product candidate list P” using an algorithm different from the above-described method of listing the product candidates. The product specifying unit 374 of the basket camera matches the product candidate lists S and P, and specifies a product when the product list exceeds a predetermined threshold.
As a method of listing product candidates, for example, a method of matching information of an object captured image obtained from an object whose existence has been recognized with image information held by the product DB 131 or the storage unit 108 is implemented. Is also good. That is, when the feature information of the two images matches, that is, when the feature information exceeds the threshold, the object whose existence is recognized by the object recognition unit 373 of the basket camera is specified as a product registered in the product DB 131.
The product specifying unit 374 by the basket camera may specify the product from the captured image over a plurality of frames without specifying the product in one frame of the captured image. At that time, the product specifying unit 374 based on the basket camera assigns a percentage to the product candidate, adds the percentage based on information such as purchase history, time, place, and personal preference, and exceeds a certain threshold. The product may sometimes be specified.
In the present embodiment, the specific object inspection mode in which the inspection is performed by the inspection terminal Q for a specific object, and the all object inspection mode in which the inspection is performed by the inspection terminal Q for all objects are provided. Can be set.
When the specific object inspection mode is set, the product identification unit 374 of the basket camera inspects the object captured image of the object identified as the product unspecified and related information via the inspection result acquisition unit 301a. To the terminal Q for use. On the other hand, when the all-object inspection mode is set, the product specifying unit 374 based on the basket camera determines the result of specifying the product, the product captured image of the specified product, the object captured image of the object determined to be product unspecified, and The related information and the like are transmitted to the inspection terminal Q via the inspection result acquisition unit 301a.
The product specifying unit 374 based on the basket camera specifies the product based on the inspection result obtained from the inspection terminal Q by the inspection result obtaining unit 301a. Specifically, for the product specified by the product specifying unit 374 by the basket camera, when the specification result is approved or corrected by the inspection result, the product specifying unit 374 by the basket camera according to the content indicated by the inspection result. , To specify the product. In addition, for an object that has been determined to be unspecified by the product identification unit 374 by the basket camera, the product identification unit 374 by the basket camera identifies the object as a product indicated by the inspection result. When the product specifying unit 374 of the basket camera requests the specification of the product by inspection, the request may be performed after complementing the product specification by the label recognition unit 376 of the basket product. In this case, as a result of complementation of the product specification by the basket product label recognition unit 376, it is possible to specify products by visual inspection, excluding products for which products have been specified.

The basket product unspecified determination unit 375 associates the fact that the object placed in the basket could not be recognized as a product with the product camera identification unit 374 with the ID of the moving object Mo.
The cart product label recognition unit 376 recognizes the attached label in accordance with the product specified in the basket by the product specifying unit 374 of the basket camera. The label recognition unit of the basket product reads the multidimensional code including characters, barcodes, and the like described on the label by making full use of the image recognition technique, and complements the product specification.
The basket merchandise discount sticker recognition unit 377 specifies the discount amount and discount rate of the discount sticker attached to the merchandise specified by the merchandise specifying unit 374 by using the image recognition technique. The basket product discount sticker recognition unit executes specification of a discount amount or the like while the product specifying unit of the basket camera is processing.

Further, the present information processing system includes a sales-restricted product determination unit 380 and a remote operation unit 390.
As described in the first embodiment, the trading-restricted product determination unit 380 determines whether the specified product is a trading-restricted product. When the trading-restricted product determination unit detects the trading-restricted product, the information is displayed on the error display unit 151.
In the present embodiment, the trading-restricted product determination unit 380 transmits a product image of a product corresponding to the trading-restricted product to the inspection terminal Q via the inspection result obtaining unit 301a. At this time, the trading-restricted product determining unit 380 appropriately transmits information (face image, personal information, etc.) regarding the shopper to the inspection terminal Q via the inspection result acquiring unit 301a according to the type of the trading-restricted product. Send.
Then, based on the inspection result obtained from inspection terminal Q by inspection result obtaining section 301a, trading restricted product determination section 380 determines whether or not sale of the trading restricted product is permitted (trading restriction is released. Is determined.
The remote operation unit 390 is provided in the information terminal 38 or the server 1 and has a function of, for example, resolving an error state when notified of the error state.

  The inspection result acquisition unit 301a requests identification of a product by inspection by transmitting the object captured image or the product identification result and the product captured image of the identified product to the inspection terminal Q. The inspection result transmitted from the inspection terminal Q in response to the request is acquired. The inspection result acquisition unit 301a outputs the acquired inspection result (product identification result) to the shelf product recognition unit 360. In addition, the inspection result acquisition unit 301a requests the inspection terminal Q to transmit a product captured image of an item corresponding to the sales-restricted item, thereby requesting a determination of the sales-restricted item by inspection. Get the inspection result sent from. At this time, the inspection result acquisition unit 301a transmits information about the shopper (face image, personal information, etc.) to the inspection terminal Q as appropriate. The inspection result acquisition unit 301a outputs the acquired inspection result (trading-restricted product determination result) to the trading-restricted product determination unit 380.

The payment machine 4 installed in the payment area 35 calculates the total price of all commodities in one or more baskets, and performs payment or settlement. The payment machine performs the payment based on the moving object Mo that has been continuously tracked and the product information associated with the moving object Mo.
Therefore, as shown in FIG. 27, the payment machine 4 includes a CPU 401, an input unit 406, an output unit 407, and a communication unit 409.
In the CPU 401, a settlement unit 435, an input control unit 436, and an output control unit 437 function.

The input unit 406 includes a credit card, electronic money reading unit, and the like.
The output unit 407 has a screen for displaying a settlement product, a function for outputting a receipt, and the like.
The settlement unit 435 determines the settlement amount and the product to be settled.
The input control unit 436 inputs a signal from the input unit 406 and operates the CPU 401.
The output control unit 437 outputs the calculation result of the settlement unit 435 to the output unit 407.
The settlement machine 4 compares the positional information of the moving object Mo with a predetermined positional relationship of the settlement area 35, and determines that the position information of the moving object Mo has entered the settlement area 35, or that the moving object Mo Settlement at 4 is triggered.

Alternatively, a button for instructing the settlement machine 4 to start settlement may be provided, and the settlement may be performed by pressing a button or the like as a trigger. The settlement machine 4 is provided with a weight sensor (not shown), and when the moving object Mo is placed. The change of the weight may be recognized, and the payment machine 4 may perform the payment.
The payment machine 4 can pay not only cash, but also gift certificates, cash vouchers, virtual currency, and the like.
In such a payment machine 4, an error condition may occur. The error state includes (A) a system processing abnormality, (B) a case where an unspecified product is linked to the moving object Mo, (C) a case where a trading restricted product is linked to the moving object Mo, and the like. Examples include (but are not limited to).

The system adopted in the present embodiment performs various measures according to each error state.
In the case of (A), the system processing abnormality is presented to the checkout machine 4, the information terminal 38a, the information terminal 38b, and the server 1. Thus, the clerk can eliminate the error state. The elimination of the error state may be performed by the remote operation unit 390.
In the case of (B), the fact that the unspecified product is linked to the moving object Mo is presented to the checkout machine 4, the information terminal 38a, the information terminal 38b, and the server 1. In the present embodiment, when the error state (B) occurs, the inspection is performed at the inspection terminal Q, and the settlement is continued or stopped according to the inspection result. Thus, the number of cases in which the settlement can be automatically performed can be increased. When the error state (B) occurs, the clerk may cancel the error state. The elimination of the error state can be performed by the remote operation unit 390.
In the case of (C), the fact that the sale-restricted product is linked to the moving object Mo is presented to the checkout machine 4, the information terminal 38a, the information terminal 38b, and the server 1. In the present embodiment, when the error state (C) occurs, the inspection is performed in the inspection terminal Q, and the settlement is continued or stopped according to the inspection result. Thus, the number of cases in which the settlement can be automatically performed can be increased. When the error state (C) occurs, for example, in the case of a sale-restricted product due to age restriction, a store clerk checks the age of the shopper, or in the case of a sale-restricted product due to a product whose consumption / expiration date has expired. If so, the store clerk can exchange the product, or if the product is a trade-restricted product other than allergic or halal food, the shopper may check the error by himself to eliminate the error state. The elimination of the error state can be performed by the remote operation unit 390.

Here, a method of specifying a product from an object by the product recognition system according to the third embodiment and adjusting the price of the product will be described with reference to FIG.
FIG. 31 is a flowchart illustrating a basic flow of the automatic settlement process executed by the server 1, the sales floor device 3, and the settlement machine 4 in FIG.

  In step S301, the shopper (moving object) enters the shop from the entrance 31 of the store 30 (FIG. 24), and the ceiling camera 310 installed near the entrance 31 starts imaging the shopper. When the shopper takes the basket or card and proceeds along the passage 34, the ceiling camera 310 at the back starts imaging the shopper. In this manner, the plurality of ceiling cameras 310 constantly image the entire store 30 including the shopper, the basket, and the cart. Note that, before step S301, the personal authentication unit 320 may perform personal authentication of the shopper and acquire personal information of the shopper.

In step S302, the moving object finding unit 3302 using the ceiling camera finds a moving object (not numbered) in the captured image using a state space model (Bayesian filter) or the like.
In the captured image, not only the object moving in the store 30 but also the still object is imaged. Therefore, the moving object detection unit 3302 using the ceiling camera finds only the object moving in the store 30 from which the still object has been removed. .

  In step S303, the car discovery unit 3303 using the ceiling camera finds baskets (moving object Mo in the third embodiment) from the objects discovered by the moving object discovery unit 3302 using the ceiling camera, and collects individual IDs. Turn. The ID is continuously used until a specific timing such as closing the store or completing payment.

  In step S304, the ceiling camera-based car area definition unit 3306 defines the position of the area of the moving object Mo discovered by the ceiling camera-based car discovery unit 3303. Further, when the moving object Mo moves within the range imaged by the ceiling camera 310, the position of the area of the moving object Mo after the movement is defined again. The position information is managed in the position information management DB 132, a memory, and the like in association with the ID of the moving object Mo, and is updated for each area definition. This defined position is also recognized at a position where another ceiling camera 310 is capturing an image.

  In step S305, the basket area tracking unit 3307 based on the ceiling camera estimates the position where the moving object Mo moves in the image captured by the certain ceiling camera 310. Furthermore, the area of the moving object Mo is defined with respect to the position where the basket area defining unit 3306 by the ceiling camera is estimated to have moved, and the position information of the moving object Mo stored in the position information management DB 132 or the memory is updated. I do.

In step S306, the object entry / exit detection unit 3608 using the shelf camera detects that an object such as a shopper's hand has entered or exited the shelf. This detection becomes a trigger, and the object recognition unit 3602 by the shelf camera is activated. The entry of the object into the shelf is detected based on whether the image data in the entry detection area set in each shelf camera 311 has changed.
The object entry / exit detection unit 3608 using the shelf camera detects that the object has left the shelf by continuously tracking the object that has entered the shelf.

  In step S307, the object recognition unit 3602 by the shelf camera compares the captured image before the object enters the shelf and the captured image after the moving object Mo has exited from the shelf, and takes the image from the shelf. Recognize objects that have fallen.

  In step S308, the product identification unit 3603 based on the shelf camera identifies which product the detected moving object Mo is. The product specifying unit 3603 using the shelf camera lists product candidates for the object detected by the object recognizing unit 3602 by using an image processing method such as specific object recognition, general object recognition, and deep learning. The listed product candidates are referred to as “product candidate list S”. Thereafter, the product specifying unit 3603 using the shelf camera performs a verification function, and specifies the product with high accuracy.

The verification function lists the “product candidate list P” using an algorithm different from the above-described method of listing the product candidates. The product specifying unit 3603 by the shelf camera matches the product candidate lists S and P, and specifies a product when the product candidate list S exceeds a predetermined threshold.
As a method of listing product candidates, for example, a method of matching information of an object captured image obtained from an object whose existence has been recognized with image information held by the product DB 131 or the storage unit 108 is implemented. Is also good. That is, when the feature information of the two images coincides with each other, that is, when the feature information exceeds the threshold, the object whose existence has been recognized by the object recognition unit 3602 by the shelf camera is identified as a product stored in the product DB 131.
Note that the product unspecified determination unit 3609 transmits error information to the server 1 when the product cannot be specified. This error information is displayed on the error display unit 151 and the information terminal 38.

In step S309, the basket and product linking unit 3605 links the product specified by the product specifying unit 3603 by the shelf camera with the ID of the moving object Mo. That is, one of the baskets (moving objects Mo) of the shopper who has taken the specified product is specified.
In addition, when the product is not specified by the product unspecified determination unit 3609 when the product or the object is taken or returned from the shelf, the information on the product unspecified (error information) is associated with the ID of the moving object Mo.

  In step S310, the product list management unit 357 linked to the moving object Mo keeps managing the list of products linked to the ID of the moving object Mo. This management is continued until a predetermined timing, such as when the moving object Mo moves to the settlement area 35.

  In step S311, in the settlement area 35, when all the commodities of the moving object Mo are summed up and settled or settled by the settlement machine 37, the commodity list associated with the moving object Mo is updated to a settled status, and the automatic settlement processing is performed. To end. When the moving object Mo is linked to information (error information) that is not specified for the product, the automatic settlement process is not terminated, and the error information is transmitted to the server 1. This error information is displayed on the error display unit 151 and the information terminal 38. That is, the clerk can confirm and cancel the error by calling the shopper.

When a moving object Mo that has not been settled passes through the exit 32, error information is transmitted to the server 1. This error information is displayed on the error display unit 151, the information terminal 38a, and the information terminal 38b. If an alarm (not shown) is provided near the exit 32, the alarm issues a warning by sound or light.
In the automatic settlement process according to the third embodiment, since the product is specified based on the image captured by the shelf camera, it is not necessary to specify the product for the object placed in the basket.
However, in order to verify that the product has not been incorrectly specified, it is preferable to specify the product even for an object placed in the basket.

  FIG. 32 is a flowchart illustrating a process of recognizing a product in a basket in an automatic payment process executed by the server counter device and the payment machine in FIG. 27.

In step S321, the car entrance / exit detection unit 372 of the car camera places the particles in the entry detection area set in advance at a predetermined position in the captured image so that the likelihood other than the area of the car becomes low.
In step S322, the car entrance / exit detection unit 372 of the car camera detects that an object (a shopper's hand or the like) has entered the entrance detection area in the car based on the likelihood of the particles. In addition, in order to prepare for the entry of a plurality of objects, after the object enters the entry detection area in the car, the car entry / exit detection unit 372 of the car camera sets new particles.
In step S323, the basket camera 312 captures an image of the state of the basket at the time when the object (the shopper's hand or the like) enters the entry detection area. The basket camera 312 also captures an image outside the entry detection area.
For example, it is assumed that three products are captured in the captured image (previous image). The product captured in the previous image is a product because it has already been specified from the object.

  In step S324, the car camera 312 captures an image of the state inside the car at the time when the object exits the entry detection area. For example, it is assumed that an image (back image) captured at this time includes one object in addition to the three products described above.

In step S325, the object recognition unit 373 based on the basket camera compares the previous image and the rear image, and defines an image area of the added one object.
In step S326, the product specifying unit 374 using the basket camera specifies which product the increased one object is. For the specification of this product, the same method as the specific method performed by the shelf product recognition unit 360 can be adopted.

In step S327, the product specifying unit 374 using the basket camera determines whether the mode is set to the specific object inspection mode or the all object inspection mode.
If the specific object inspection mode has been set, the process proceeds to step S328.
If the all-object inspection mode has been set, the process proceeds to step S329.
In step S328, the inspection result acquisition unit 301a requests an inspection (identification of a product) at the inspection terminal Q for the target object, and acquires the inspection result.
After step S328, the process proceeds to step S329.
In step S329, the product specifying unit 374 based on the basket camera determines whether the added one object cannot be specified as any product. If the product identification unit 374 of the basket camera cannot identify any of the products (YES in step S329), the process proceeds to step S327.

If the product identification unit 374 by the basket camera has been able to identify which product it is in step S329 (NO in step S329), the product identification unit 374 by the basket camera determines the product name or the product name stored in the storage unit of the server 1. The product is specified including information such as the price and the sale-restricted product. Accordingly, the process proceeds to step S330. The specified product information may be output to the output unit 407.
If it is not possible to specify which product the product specifying unit 374 using the basket camera is in step S329, the screen display on a display unit (not shown) provided in the basket camera, voice guidance using a speaker (not shown), or the like is used. A configuration in which the purchaser is requested to put the product in the basket again may be adopted. Further, a configuration may be adopted in which the purchaser and the inspector talk with each other via a microphone (not shown) provided in the basket camera.

  In step S330, the sales-restricted product determination unit 380 determines whether the product specified by the product specification unit 374 of the basket camera is a product for which age confirmation is required.

If it is determined in step S330 that the product specified by the product specifying unit 374 of the basket camera is a product for which age confirmation is required, that is, if it is determined as YES, the process proceeds to step S331.
If it is determined in step S330 that the product specified by the product specifying unit 374 of the basket camera is not a product for which age confirmation is required, that is, if it is determined to be NO, the process of recognizing the product in the basket ends. Becomes
In step S331, the output control unit 437 causes the output unit 407 of the checkout machine 4 to display a screen for confirming the age. However, if the shopper's personal information is acquired and there is no need to confirm the age here, step S330 is skipped and the process proceeds to step S335. Here, if there is a problem with the age authentication of the shopper, the purchaser and the inspector may communicate via a speaker (not shown) and a microphone (not shown) provided in the checkout machine 4.
In step S332, the inspection result acquisition unit 301a requests an inspection (determination of a trade-restricted item) at the inspection terminal Q for the target product, and acquires the inspection result.
In step S333, the trading restriction product determination unit 380 determines whether an instruction to cancel the trading restriction has been received.

If it is determined in step S333 that an instruction to cancel the trading restriction has not been received, the process proceeds to step S334.
If it is determined in step S333 that a release instruction for releasing the trading restriction has been received, that is, if it is determined as YES, the process proceeds to step S335.
In step S334, the inspection result transmitting unit 412 of the inspection terminal Q transmits a warning that the trading restriction has not been released by the inspection result. Upon receiving this warning, for example, the output control unit 437 of the checkout machine 4 presents, via the output unit 407, a warning that the trading restriction has not been released by the inspection result. After step S334, the process ends, and the settlement stops.
In step S335, the trading restriction product determination unit 380 releases the trading restriction.
When step S335 is completed in this way, or when it is determined in step S330 that the product is not an age-restricted product (when it is determined that the product is not NO), the process of recognizing the product in the basket is completed.

In this way, the present information processing system specifies which merchandise the object placed in the basket is.
Then, the information processing system can verify whether the product specified in step S308 and the product specified in step S326 match. When the product is unspecified, an error state is displayed on the error display unit 151, the information terminal 38a, and the information terminal 38b. Then, for the object or the product to be inspected (for example, the object whose product cannot be specified at the cashier terminal 2 or the product corresponding to the sale-restricted product), the inspection is performed at the inspection terminal Q. Automatic payment is performed in accordance with the inspection result (such as the result of specifying a product or the result of determining a trade-restricted product).
Therefore, according to the present information processing system, when a shopper purchases a product, it is possible to automate the settlement of the price of the product and improve the accuracy of specifying the product.
In addition, when an object or a product that is not appropriate for settlement is settled, it is possible to prevent the settlement by mistake. When the information indicating that the product is unspecified is displayed on the information terminal 38a or the information terminal 38b, the store clerk may go to the sales floor where the basket is located to verify the product, and the store clerk inside or outside the store may remotely check the product. The error state may be canceled by an operation.

  In the third embodiment, errors assumed in tracking a moving object Mo (a basket or a cart) include (A) a shopper entering the store from the entrance 31 of the store 30 and picking up the basket or the cart. (B) When the moving object detecting unit 3302 by the camera cannot detect the moving object Mo, (B) when the cage area tracking unit 3307 by the ceiling camera loses track of the moving object Mo being tracked, (C) two or more different movements This includes, but is not limited to, a case where the ID linked to each of the objects Mo is changed during the tracking.

In accordance with each error state, the system adopted in this embodiment performs various measures including the following examples.
(A) An image including a shopper holding a basket or a cart as an object imaged by the ceiling camera 310 is transmitted to the inspection terminal Q, and a request is made to detect the moving object Mo by the inspection. When the examiner detects the moving object Mo, a new moving object Mo corresponding to the basket or the cart held by the shopper is defined, an ID is issued, and the basket area tracking unit 3307 by the ceiling camera. Start tracking by. When the examiner cannot detect the moving object Mo, the clerk is notified of the fact.
(B) If a basket or cart that is not linked to any ID registered in the location information management DB 132 is found, an imaged image including a shopper holding the basket or cart and an ID candidate to be linked Is transmitted to the inspection terminal Q. The examiner links the most appropriate ID to the basket or cart based on the past link information of the basket or cart and the ID, and starts re-tracking by the basket area tracking unit 3307 using the ceiling camera. If re-tracking could not be started for any reason, a clerk is notified to that effect.
In addition, when an ID that has been lost in association with a basket or a cart is detected, the ID and a list of captured images of shoppers holding a basket or a cart that is not associated with any ID are used for inspection. Sent to terminal Q. The examiner attempts to associate the basket or cart with the ID based on the past association information between the basket or cart and the ID, and if the association is successful, the basket area tracking unit 3307 using the ceiling camera. Start retracking by. If the start of re-tracking fails for any reason, the clerk is informed accordingly.
(C) An image list of shoppers holding each basket or cart and a list of IDs to be associated with each basket or cart are transmitted to the inspection terminal Q. The examiner attempts to allocate the ID to the basket or cart most appropriately based on the past association information between the basket or cart and the ID. When the ID can be appropriately assigned to the car or the cart, re-tracking by the moving object area tracking unit 3305 by the ceiling camera is started. If the ID cannot be appropriately assigned to the basket or the cart, the ID is abandoned and the clerk is notified that the link has been replaced.

[Embodiment 4]
FIG. 33 is a diagram illustrating a layout example of a supermarket that employs the product recognition system according to the fourth embodiment.
The information processing system according to the fourth embodiment is a product recognition system applied to a store 40 such as a supermarket as shown in FIG. The information processing system according to the fourth embodiment is configured such that an automatic payment can be performed simply by passing a commodity through the payment gates 5-1 to 5-3 without being placed on a cashier.

  In the store 40, a shopping basket, a shopping cart, and the like (without numbering) are placed at the entrance 41. In front of the exit 42, settlement gates 5-1 to 5-3 and a settlement register 6 operated by a clerk Mt are installed. A plurality of shelf racks 43 for displaying commodities are installed between the entrance 41 and the exit 42. In the shelf rack 43, a plurality of shelves are arranged at predetermined intervals in the vertical direction, and various products are displayed on each of the plurality of shelves. Hereinafter, the space between shelves is also referred to as “in the shelf”. A passage 44 is provided between the shelf racks 43 facing each other in the horizontal direction.

The shopper enters the store from the entrance 41, takes a shopping basket, pushes the shopping cart, or proceeds along the aisle 44 with the my bag brought. The shopper takes the goods in the shelves, puts them in baskets, and proceeds along the aisle 44. When the shopper takes all the products he / she wants to purchase, the shopper proceeds to the settlement area 45 and makes the payment.
In the passage 44, the settlement area 45, and the like, the store clerk Mt goes around. The clerk Mt has an information terminal 9a. The information terminal 9a is a portable information processing terminal such as a smartphone, and has a function of displaying a state in a store, a function of displaying an error state occurring in the store, a remote operation function, and the like.

In FIG. 33, the inside of the cloud 40 is drawn not inside the store 40 but outside the store 40 or the backyard of the store 40. The server 1 (FIG. 7) is installed outside the store 40 or in the backyard of the store. Outside the store or in the backyard, the clerk Mt can monitor the inside of the store 40 through the screen of a large monitor (not shown) or the screen of the information terminal 9b.
In addition, a shopping basket, a shopping cart, and a my bag are called baskets. The passage 44 including the settlement gates 5-1 to 5-3 is set as a settlement area 45. In addition, baskets and the like and shoppers are referred to as moving objects Mo.

A plurality of ceiling cameras 310 are installed on the ceiling of the passage 44 between the entrance 41 and the exit 42. A plurality of shelf cameras 311 are installed at a plurality of locations on each shelf in each shelf rack 43.
In FIG. 33, the inside of the shop 40 but the outside of the shop 40 and the backyard of the shop 40 are drawn inside the diagram drawn in a cloud shape. Outside the store 40, the server 1 (FIG. 7) is installed. In the backyard, a store clerk Mt monitors the inside of the store 30 through a screen of a large monitor (not shown) or a screen of the information terminal 9b.

FIG. 34 is a configuration diagram illustrating a configuration of a product recognition system as a fourth embodiment of the information processing system.
The product recognition system according to the fourth embodiment has a configuration as shown in FIG.
The product recognition system includes a server 1, a sales floor device 3, and n (n is an arbitrary integer) settlement gates 5-1 to 5-n. Note that the cashier terminal 2 employed in the first and second embodiments is not employed in the fourth embodiment.
Each of the server 1, the sales floor device 3, and the checkout gates 5-1 to 5-n is mutually connected via a network N such as the Internet (Internet) line.
For convenience of explanation, only one server 1 in FIG. 34 is drawn, but there may be a plurality of servers 1 in practice.
Hereinafter, when it is not necessary to individually distinguish the settlement gates 5-1 to 5-n, they are collectively referred to as "payment gate 5".

  The server 1 executes each process in order to manage each operation of the counter 3 and the settlement gate 5. The server 1 has the same configuration as the hardware configuration of the server 1 of the first embodiment shown in FIG.

  Therefore, the server 1 includes the CPU 101, the ROM 102, the RAM 103, the bus 104, the input / output interface 105, the output unit 106, the input unit 107, the storage unit 108, the communication unit 109, and the drive 110. Have.

  The sales floor device 3 is configured in the same way as the hardware configuration of the third embodiment shown in FIG. 26, but does not include the illustrated basket camera 312.

  Therefore, the sales floor device 3 includes the CPU 301, the ROM 302, the RAM 303, the bus 304, the input / output interface 305, the ceiling camera 310, the shelf camera 311, the information terminal 9, and the communication unit 315. .

  However, in the error state in the fourth embodiment, when an abnormality occurs in the system processing or when a product cannot be specified in shelf product recognition, an unspecified object or a moving object Mo linked to a trade-restricted product is detected. This occurs in various situations, such as when trying to pass through the checkout gate 5 or when a shopper (moving object Mo) who has not settled is about to leave the store.

The settlement gate 5 is connected to such a sales floor device 3 via the network N.
The settlement gate 5 is divided into a settlement gate 5-1 using a settlement machine 5a, a settlement gate 5-2 using electronic money or the like, and a settlement gate 5-3 that can be settled just by passing.
The payment gates 5-1 to 5-3 other than the manned payment register 6 may be provided with a normally closed opening / closing member (not numbered).

The settlement gate 5-1 using the settlement machine 5a is provided with a settlement button (not shown), calculates the total amount of the purchased merchandise, and performs the settlement by the settlement machine 5a provided at the exit 42 side. The payment machine 5a is installed on the exit 42 side of the payment gate 5-1. The payment machine 5a includes a settlement unit that can pay by cash, credit card, electronic money, point payment, gift certificate / virtual currency, prepaid card, and the like.
In the settlement gate 5-1 using the settlement machine 5a, when the settlement person presses the settlement button, this press triggers the reading of the product information associated with the moving object area described later, The settlement amount is settled, and it becomes possible to pass through the settlement gate 5-1. If an opening / closing member is provided, the opening / closing member opens. Then, when the person who settles the account is settled by the settlement machine 5a, it is possible to leave the store.

The settlement gate 5-2 using electronic money or the like performs settlement by holding electronic money or the like over the gate body. That is, the settlement gate using electronic money or the like includes a card reading unit (not shown), does not include a settlement button like the settlement gate 5-1 using the settlement machine 5a, and does not use the settlement machine 5a.
Electronic money and the like include not only IC cards that can be settled but also cards in a narrow sense, such as credit cards, so-called point cards, and prepaid cards, as well as information portable terminals. Hereinafter, various types of cards will be described.
When a person who setstles enters the settlement area 45 of the settlement gate 5-2, and various cards are read by the card reading unit, merchandise information associated with a moving object area described later is read, and the settlement and settlement are completed. , Through the settlement gate 5-2.

  The settlement gate 5-3 completes the settlement and settlement only when the shopper whose personal information has been acquired at the entrance 41, the passage 44, or the like passes through the settlement gate 5-3. That is, settlement and settlement are completed without paying cash, reading a card, or the like.

In the manned checkout register 6, the clerk Mt inputs the prices of the commodities individually and performs the checkout.
FIG. 35 is a block diagram showing a hardware configuration of the settlement gate 5 in the product recognition system of FIG.
The settlement gate 5 includes a CPU 501, a ROM 502, a RAM 503, a bus 504, an input / output interface 505, an input unit 506, an output unit 507, a storage unit 508, and a communication unit 409.

  The CPU 501, the ROM 502, the RAM 503, the bus 504, the input / output interface 505, the storage unit 508, and the communication unit 509 of the settlement gate 5 are configured in the same manner as those of the server 1.

  The input unit 506 is a payment button provided on the gate body in the payment gate 5-1 using the payment machine 5a, and a payment gate that can be paid only by passing through a payment gate 5-2 using electronic money. 5-3 is an information reading unit for detecting information on various cards and the like.

  The output unit 507 outputs a signal for opening / closing an opening / closing member (not numbered) provided in the settlement gates 5-1 to 5-3. The settlement gate 5-1 using the settlement machine 5a outputs an amount to be settled to the settlement machine 5a, a product name, and the like.

  FIG. 36 is a functional block diagram showing an example of a functional configuration of the server 1 in FIG. 7, the counter 3 in FIG. 26, and the settlement gate 5 in FIG.

The server 1 includes a CPU 101 similar to the third embodiment, a storage unit 108, an error display unit 151, and an error release unit 152.
The server 1 is configured similarly to the server 1 of the third embodiment.

FIG. 37 is a functional block diagram illustrating a detailed functional configuration example of the moving object tracking unit 330 provided in the counter 3 according to the third embodiment.
In the CPU 301 of the sales floor device 3, as shown in FIG. 37, a personal authentication unit 320, a moving object tracking unit 330, a position information management unit 340, a shelf product recognition unit 360, a basket product recognition unit 370, And a restricted product determination unit 380.
The personal authentication unit 320, the moving object tracking unit 330, the position information management unit 340, the basket product recognition unit 370, and the trading restricted product determination unit 380 are configured in the same manner as the third embodiment.

  The personal authentication unit 320 includes a personal information acquisition unit 321 similar to the third embodiment. The personal authentication unit 320 acquires personal information from the personal information acquisition unit 321 or the DB management unit 141 of the server 1.

As shown in FIG. 37, the moving object tracking unit 330 includes a moving object detection unit 3302 using a ceiling camera, a moving object region definition unit 3304 using a ceiling camera, a moving object region tracking unit 3305 using a ceiling camera, and a grouping unit 3308. , A blood relation determination unit 3309, a moving object area transfer recognition unit 3311 using a ceiling camera, an object recognition unit 3312 transferred using a ceiling camera, and a product specification unit 3313 transferred using a ceiling camera. I have.
The moving object finding unit 3302 using the ceiling camera finds a moving object Mo (a shopper, a basket, a cart, or the like) using a state space model (such as a Bayesian filter) based on the image captured by the ceiling camera 310.
The moving object region definition unit 3304 using the ceiling camera defines the region of the moving object Mo found by the moving object finding unit 3302 using the ceiling camera as a moving object region. The moving object area definition unit 3304 based on the ceiling camera defines a moving object area by continuously finding a changed area around the moving object Mo. That is, the moving object region definition unit 3304 based on the ceiling camera defines the moving object Mo, the moving object Mo, and a certain range of the periphery of the moving object Mo as a moving object region.

Then, by applying a skeletal model when the shopper is viewed from above, an approximate pose is estimated, and the moving object Mo is clearly identified by comparing the estimated pose with the moving object region in the actually obtained video. May be defined.
Here, when an area centered on a person is called a person area, the person area is a subordinate concept of the moving object area.
In addition, when an area around a basket is referred to as a basket area, the basket area is a lower concept of the moving object area.
When the area around the cart is called a cart area, the cart area is a lower concept of the moving object area.

The moving object area tracking unit 3305 using the ceiling camera tracks the movement of the moving object Mo. For example, the moving object area tracking unit 3305 using the ceiling camera tracks the movement of the moving object Mo by collecting feature data (color, shape, and the like) of the moving object Mo.
Alternatively, the moving object area tracking unit 3305 using the ceiling camera tracks the movement of the moving object Mo by using an object tracking technique in an image such as a Bayesian filter, a fast Fourier transform, or a TLD (Tracking-Learning-Detection).

The grouping unit 3308 groups a plurality of people, such as family members and friends, when they visit the store.
In addition, the grouping unit 3308 may group a plurality of persons using information such as a sense of distance (overlapping or sticking) between the moving objects Mo and a moving direction (vector).
The grouping unit 3308 may associate the person area with the basket area / cart area.
By the functioning of the grouping unit 3308 in this way, the payment can be made by one person at the payment gate 5.

  The blood relation determination unit 3309 uses a face authentication method to distinguish a parent-child relationship, a sibling relationship, and the like. The blood relationship determination unit assists the function of the grouping unit. The blood relation determination unit may determine the similarity of the face using a deep learning face recognition method, and may estimate the blood relation.

  When goods are transferred from the shopper (moving object Mo) to the shopper (moving object Mo), the moving object area transfer recognition unit 3311 using the ceiling camera transfers the object from the moving object Mo to the moving object Mo. That is, the transferred / moved moving object Mo is identified, and a product list associated with each moving object Mo is read.

The moving object area transfer recognition unit 3311 using the ceiling camera recognizes that a product has been delivered from the shopper (moving object Mo) to the shopper (moving object Mo), and is passed with the transferred moving object Mo. The moving object Mo is specified, and a product list associated with each moving object Mo is read. The object recognition unit 3312 transferred by the ceiling camera thereafter defines an area of the object from the captured image at the time when the transfer is recognized.
Furthermore, from the image after the object area is defined, the object passed by the ceiling camera and passed by the product identification unit is any of the products in the product list associated with the read and passed moving object Mo. The moving object Mo specified by the specified moving object area transfer recognition unit 3311 by the ceiling camera is linked to the product specified in the transfer, and the list of each product is updated.
The moving object area transfer recognition unit 3111 using the ceiling camera may analyze the movement of the shopper (moving object Mo) by using an object recognition technique such as deep learning, and may perform transfer recognition. May be recognized, and handover recognition may recognize an overlap between human regions (which may include hands).
The moving object area transfer recognition unit 3311 using the ceiling camera may be realized by a shelf camera or the like that can capture a wide area, instead of the ceiling camera.

  Thereafter, the object recognition unit 3312 transferred by the ceiling camera defines an area of the object from the captured image at the time when the transfer is recognized.

  Further, the commodity specifying unit 3313 passed by the ceiling camera passes the object recognized by the object recognition unit 3312 passed by the ceiling camera to the read moving object Mo (the person who has passed). Which of the products in the product list is specified, the respective moving objects Mo specified by the moving object area transfer recognition unit 3311 by the ceiling camera and the products specified by the product specification unit 3313 passed by the ceiling camera are linked. Then, the list of products of each moving object Mo is updated.

The object recognizing unit 3312 transferred by the ceiling camera may set the ceiling camera to a zoomable camera, zoom up a portion estimated to have been transferred, and define a region of the object.
A moving object area transfer recognition unit 3311 by the ceiling camera, an object recognition unit 3312 transferred by the ceiling camera, and a commodity specifying unit 3313 transferred by the ceiling camera are replaced by a shelf camera capable of imaging a wide area instead of the ceiling camera. And so on.

  As in the second embodiment, the position information management unit 340 includes an inter-camera information transfer unit 341, a position definition unit 342 of each camera, and a moving object display unit 343, as shown in FIG.

FIG. 38 is a functional block diagram illustrating a detailed functional configuration example of the shelf product recognition unit 360 provided in the counter device 3 according to the fourth embodiment.
The shelf product recognizing unit 360 includes an object recognizing unit 3602 using a shelf camera, a product specifying unit 3603 using a shelf camera, a moving object and a product linking unit 3604, and a product list managing unit 3606 linking the moving object shown in FIG. , An object entry / exit detection unit 3608 by a shelf camera, a product unspecified determination unit 3609, a label recognition unit 3610, and a discount seal recognition unit 3611.

  Among these, the shelf camera-based product specifying unit 3603 specifies which product in the shelf is recognized by the shelf camera-based object recognition unit 3602. The product specifying unit 3603 by the shelf camera lists product candidates by an image processing method such as specific object recognition, general object recognition, and deep learning. The listed product candidates are referred to as “product candidate list S”. After that, the product specifying unit 3603 by the shelf camera specifies the product with high accuracy by performing the verification function.

The verification function lists the “product candidate list P” using an algorithm different from the above-described method of listing the product candidates. If the results of the product candidate lists S and P are matched, and the value exceeds a predetermined threshold, the product is specified.
For example, the listing method of the “commodity candidate list” may be realized by a method of matching image information of an object obtained from an object whose existence has been recognized with image information stored in the product DB 131 or a memory. . That is, when the feature information of the two images match (exceeds the threshold), the object whose existence has been recognized by the object recognition unit 3602 by the shelf camera is a product registered in the product DB 131. The unit 3603 specifies that the product is a product registered in the product DB 131. By creating product candidates by deep learning, and then performing a verification function, products are specified with high accuracy.

  The product identification unit 3603 by the shelf camera does not identify the product in one frame of the image captured by the shelf camera 311 but also makes full use of the image captured by the ceiling camera 310 to identify the product over a plurality of captured images. You may. At that time, the product identification unit 3603 by the shelf camera gives a percentage to the product candidate, adds the percentage based on information such as purchase history, time, place, and personal preference, and exceeds a certain threshold. When the product is specified.

In the present embodiment, the specific object inspection mode in which the inspection is performed by the inspection terminal Q for a specific object, and the all object inspection mode in which the inspection is performed by the inspection terminal Q for all objects are provided. Can be set.
When the specific object inspection mode is set, the commodity identification unit 3603 by the shelf camera inspects the object captured image of the object identified as the commodity unspecified and related information via the inspection result acquisition unit 301a. To the terminal Q for use. On the other hand, when the all-object inspection mode is set, the product identification unit 3603 by the shelf camera outputs the product identification result and the product imaging image of the identified product, and the object imaging image of the object determined to be product unspecified. The related information and the like are transmitted to the inspection terminal Q via the inspection result acquisition unit 301a.
Then, the shelf camera-based product specifying unit 3603 specifies the product based on the inspection result obtained from the inspection terminal Q by the inspection result obtaining unit 301a. Specifically, for the product specified by the product identification unit 3603 by the shelf camera, if the specification result is approved or corrected by the inspection result, the product identification unit 3603 by the shelf camera according to the content indicated by the inspection result. , To specify the product. In addition, for an object that is determined to be unspecified by the product identification unit 3603 by the shelf camera, the product identification unit 3603 by the shelf camera identifies the object as a product indicated by the inspection result. When the product specifying unit 3603 by the shelf camera requests the specification of the product by inspection, the request may be made after complementing the product specification by the label recognition unit 376 of the basket product. In this case, as a result of the complementation of the product specification by the label recognition unit 3610, it is possible to specify the product by inspection, excluding products for which the product has been specified.
The other functional units in the shelf product recognition unit 360 are the same as in the second embodiment.

  In the CPU 501 of the settlement gate 5, a settlement unit 535, an input control unit 536, and an output control unit 537 function as shown in FIG.

When the moving object area that has been continuously tracked enters the settlement area 45, the settlement unit 535 receives merchandise information (commodity list) associated with one or more moving object areas from the server 1 via the communication unit 509. , Settlement amount and settlement target products are determined.
The input control unit 536 inputs a signal from an input unit 506 such as a payment button provided on the gate body or an information reading unit.
The output control unit 437 causes the output unit 507 to display the settlement amount, outputs information to the settlement machine 5a, and opens and closes the opening and closing member.

In addition, the settlement unit 535, based on the product information (commodity list) associated with one or more moving objects Mo, when the moving object Mo that has been continuously tracked, that is, when the person who performs settlement enters the settlement area 45, is set. , Settlement amount and settlement target products are determined.
Further, for example, when a father pays with a wallet, product information (a list of products) associated with a companion mother and child may be settled by the father using grouping.

  The settlement gate 5-1 using the settlement machine 5a is a device that calculates the total amount of the merchandise and performs settlement, and includes a settlement button (not shown). The payment machine 5a is installed closer to the exit 42 than the payment gate 5. The payment machine 5a is provided with a settlement means that can pay by cash, point payment, gift certificate, cash voucher, and the like.

  In the settlement gate 5-1, when the settlement person presses the settlement button, this press triggers reading of the product information associated with the moving object Mo, the settlement amount is determined, and the settlement gate 5 is set. -1 can be passed. Then, when the person who settles the account is settled by the payment machine, the store can be closed.

However, for example, when the product to be settled is an age-restricted product and an error state occurs such that the age cannot be confirmed, the information terminal 9a, the information terminal 9b, and the error display unit 151 are notified of the error state. . At that time, the settlement gate 5-1 keeps the passage impossible, for example, by keeping the opening / closing member closed. The settlement gate 5-1 may indicate that an error has occurred by means such as sound or light. When the clerk Mt is able to release the restriction, such as checking his age, the restriction is released by the operation of the error release unit 152 by the operation of the clerk Mt, and the payment gate 5-1 is allowed to pass. Note that the restriction can be released by remote control.
Also, when an object (an unspecified object) that cannot be identified as any product is linked to the moving object Mo, the flow is the same as that of the age-restricted product. The unspecified object may be specified by the clerk Mt in the manned checkout register 6 to specify the product.

  In addition, the settlement gate 5-2 using electronic money has a card reading unit (not shown), does not have a settlement button like the settlement gate 5-1 and does not use the settlement machine 5a. The card reading unit is adapted to handle any card such as a credit card, a David card, electronic money, a prepaid card, and the like.

Then, the person who settles enters the settlement area 45 of the settlement gate 5-2, and when the card is read by the card reading unit, the product information associated with the moving object Mo is read, and the settlement and settlement are completed. It can pass through the settlement gate 5-2.
When the payment target product is a trade-restricted product or an unspecified object is linked to the moving object Mo, the operation is the same as that of the above-described payment gate 5-1.

The settlement gate 5-3 needs to have the personal object authenticated for the moving object Mo that has been continuously tracked, and the settlement information must be specified.
When the settlement person enters the settlement area 45 of the settlement gate 5-3, the settlement gate 5-3 is automatically settled as long as there is no error, and can pass through the settlement gate 5-3.
When the payment target product is a trade-restricted product or an unspecified object is linked to the moving object Mo, the operation is the same as that of the above-described payment gate 5-1.

  The product recognition system according to the fourth embodiment may include a remote control unit (not shown) in addition to the functions described above.

The sales-restricted product determination unit 380 determines whether the specified product falls under the sales-restricted product. That is, the trading-restricted product determination unit 380 determines from the DB information that the specified product is a trading-restricted product due to the age restriction of alcohol and tobacco. The trade-restricted product determination unit 380 uses image recognition and the like to determine that the product is a trade-restricted product due to expiration of the expiration date and the expiration date. In addition, the sales-restricted product determination unit 380 associates the product with the personal information obtained by the personal authentication, and determines that the product is a sales-restricted product other than allergy, halal food, and the like to the shopper.
In addition, when it is determined that the product is a sales-restricted product, the sales-restricted product determination unit 380 presents that the product is a sales-restricted product. In addition, the sales-restricted product determination unit 380 may restrict sales based on personal information obtained from personal authentication. In addition, the sales-restricted product determination unit 380 may estimate the age and gender based on the recognition of the face and hands, and may restrict the sales.

When the remote operation unit receives a notification of an error state such as a system processing abnormality, an unspecified product, or trading restrictions, the remote operation unit eliminates the error state by remote control. The ceiling camera 310, the shelf camera 311 and the settlement gate 5 that have detected the error state transmit the error state and the captured image of the state to the information terminal 9a for remote operation inside or outside the store via the network N. 9b and the server 1 are notified. By operating these devices, an error state can be eliminated.
Naturally, the error state may be released at a manned cash register in the store, or the clerk Mt or the like may go to a place where the error state occurs and release the error state.

Next, a merchandise settlement method in the merchandise recognition system according to the fourth embodiment will be described with reference to FIG.
FIG. 39 and FIG. 40 are flowcharts for explaining the automatic settlement process executed by the server 1, the sales floor device 3, and the settlement gate 5 in FIG.

  In step S401, the shopper (moving object) enters the shop from the entrance 41 of the store 40 (FIG. 33), and the ceiling camera 310 installed near the entrance 41 starts imaging the shopper. When the shopper gets a basket or card and proceeds along the passageway 44, the ceiling camera 310 at the back starts imaging the shopper. In this manner, similarly to the third embodiment, the plurality of ceiling cameras 310 constantly capture an image of the entire store 30 including the shopper, the basket, and the cart. Note that, before step S401, the personal authentication unit 320 may perform personal authentication of the shopper and acquire personal information of the shopper.

  In step S402, the moving object detection unit 3302 using the ceiling camera finds the moving object Mo and assigns an individual ID. The ID is continuously used until a specific timing such as closing the store or completing payment. In addition, at this timing, the personal authentication unit 320 may perform personal authentication of the shopper and acquire personal information of the shopper. Further, the grouping unit 3308 may group the plurality of moving objects Mo as one unit, and the blood relationship determination unit 3309 determines the blood relationship of the plurality of shoppers (moving objects Mo), and complements the grouping unit 3308. You may do.

  In step S403, the moving object area definition unit 3304 using the ceiling camera defines a predetermined area including the moving object Mo discovered by the moving object finding unit 3302 using the ceiling camera. Further, when the moving object Mo moves within the range imaged by the ceiling camera 310, the position of the area of the moving object Mo after the movement is defined again. The position information is managed in the position information management DB 132, a memory, and the like in association with the ID of the moving object Mo, and is updated for each area definition. This defined position is also recognized at a position where another ceiling camera 310 is capturing an image.

  In step S404, the moving object area tracking unit 3305 using the ceiling camera estimates the position where the moving object Mo moves in the image captured by the ceiling camera 310. Further, the moving object area definition unit 334 defines an area of the moving object Mo with respect to the position estimated to have moved, and updates the position information of the moving object Mo stored in the position information management DB 132 and the memory.

  In step S405, similarly to the third embodiment, the object entry / exit detection unit 3608 using the shelf camera detects entry / exit of an object into / from the shelf.

In step S406, as in Embodiment 3, the activation of the object entry / exit detection unit 3608 by the shelf camera is used as a trigger, and the object recognition unit 3602 by the shelf camera causes the image before or after the image in which the object is taken or the image in which the object is placed. Are compared with each other to define an image area that is a target of product identification.
In addition, in step S406, the object acquisition recognition unit using the ceiling camera may recognize that the moving object Mo has acquired an object from inside a shelf or the like.

In step S407, as in the third embodiment, the product specifying unit 3603 based on the shelf camera specifies which product the object is. If the specified product is a sales-restricted product, the sales-restricted product is linked to the moving object Mo. Also, when it is not possible to specify which product the object is, the unspecified product is linked to the moving object Mo. In addition, the information terminal 9a, the information terminal 9b, and the server 1 may be notified of the error state due to the unspecified product, and the clerk Mt who recognizes the error state may resolve the error state by remote control (of course, the clerk Mt). May go directly to clear the error state).
Further, at the time of step S407, the label recognition unit may recognize the label linked according to the specified product.
Further, at the time of step S407, the discount seal recognition unit may recognize the attached discount seal according to the specified product.
Further, in step S407, the product specifying unit using the ceiling camera may specify any product in the object area acquired by the object acquisition recognizing unit using the ceiling camera.

In step S408, the product identification unit 3603 using the shelf camera determines whether the mode is set to the specific object inspection mode or the all object inspection mode.
If the specific object inspection mode has been set, the process proceeds to step S409.
If the all-object inspection mode has been set, the process proceeds to step S410.
In step S410, the inspection result acquisition unit 301a requests an inspection (identification of a product) at the inspection terminal Q for the target object, and acquires the inspection result.
After step S409, the process proceeds to step S411.
In step S410, the product identification unit 3603 based on the shelf camera determines whether the added one object is a product that cannot be identified. If the product identification unit 3603 by the shelf camera cannot identify which product the product is (YES in step S410), the process proceeds to step S409.

If the product identification unit 3603 by the shelf camera can identify which product is the product in step S410 (NO in step S410), the product identification unit 3603 by the shelf camera determines the product name or the product name stored in the storage unit of the server 1. The product is specified including information such as the price and the sale-restricted product. Thereby, the process proceeds to step S411. The specified product information may be output to the output unit 507.
Also, in step S410, if the product identification unit 3603 by the shelf camera could not identify any of the products, a screen display on a display unit (not shown) provided in the sales floor device 3, voice guidance by a speaker (not shown), etc. Thus, a configuration in which the purchaser is requested to remove the product from the shelf may be adopted. Further, a configuration may be adopted in which the purchaser and the inspector communicate with each other via a microphone (not shown) provided in the sales floor device 3.

  In step S411, the sales-restricted product determination unit 380 determines whether the product specified by the product specification unit 3603 by the shelf camera is a product for which age confirmation is required.

If it is determined in step S411 that the product specified by the product specifying unit 3603 by the shelf camera is a product for which age confirmation is required, that is, if it is determined as YES, the process proceeds to step S412.
If it is determined in step S411 that the product specified by the product specifying unit 3603 by the shelf camera is not a product for which age confirmation is required, that is, if it is determined as NO, the process proceeds to step S417.
In step S412, the output control unit 437 causes the output unit 507 of the settlement gate 5 to display a screen for age confirmation. However, if the shopper's personal information is acquired and there is no need to confirm the age here, step S412 is skipped and the process proceeds to step S416. Here, when there is a problem with the age authentication of the shopper, the purchaser and the inspector may communicate with each other via a speaker (not shown) and a microphone (not shown) provided in the checkout gate 5.
In step S413, the inspection result acquisition unit 301a requests an inspection (determination of a trade-restricted item) at the inspection terminal Q for the target product, and acquires the inspection result.
In step S414, the trading restriction product determination unit 380 determines whether an instruction to cancel the trading restriction has been received. The sale restriction of the sale-restricted product is released by a method in which the clerk responds on the spot in response to the result of the determination, or employs the result of the inspection as a system. Here, a case where the result of the inspection is adopted as the system will be described.

If it is determined in step S414 that an instruction to cancel the trading restriction has not been received, the process proceeds to step S415.
If it is determined in step S414 that a release instruction to release the trading restriction has been received, that is, if it is determined as YES, the process proceeds to step S416.
In step S415, the inspection result transmitting unit 412 of the inspection terminal Q transmits a warning that the trading restriction has not been released by the inspection result. Upon receiving this warning, for example, the output control unit 537 of the settlement gate 5 presents, via the output unit 507, a warning that the trade restriction has not been released by the inspection result. After step S416, the process ends, and the settlement stops.
In step S416, the trading restriction product determination unit 380 releases the trading restriction.
In this way, if step S416 is completed, or if it is determined in step S411 that the product is not an age-restricted product (if NO is determined), the process proceeds to step S417.

  In step S417, the moving object and the product association unit 3604 associate the moving object Mo with the identified product.

In step S418, the product list management unit 3606 linked to the moving object continues to manage the product list linked to the person until settlement.
In step S419, the settlement gate 5 performs settlement or settlement based on the product information associated with the moving object Mo.
Note that the error display unit 151, the information terminal 9a, the information terminal 9b, and the output unit 507 of the settlement gate 5 may notify a store clerk or the like of the error display unit 151, the information terminal 9a, the information terminal 9b, or the output unit 507 during or before step S419. .
Further, in step S410, when the trade-restricted product is linked to the moving object Mo, the error state may be displayed by keeping the gate closed without opening the gate.

  In the fourth embodiment, the errors assumed in tracking the moving object Mo (the shopper) include (A) when the shopper enters the store 40 from the entrance 41, the moving object detection unit 3302 using the ceiling camera moves. When no object is detected, (B) when the moving object area tracking unit 3305 using the ceiling camera loses track of the moving object Mo being tracked, (C) the moving object Mo is tied to each of two or more different moving objects Mo. This includes, but is not limited to, a case where the ID is changed during the tracking.

In accordance with each error state, the system adopted in this embodiment performs various measures including the following examples.
(A) An image including a shopper as a subject imaged by the ceiling camera 310 is transmitted to the inspection terminal Q, and a request for detection of the shopper by the inspection is made. When the examiner detects a shopper, a new moving object Mo is defined, an ID is issued, and tracking by the moving object region tracking unit 3305 by the ceiling camera is started. If the patron does not detect the shopper, it informs the clerk to that effect.
(B) When a shopper who is not associated with any ID registered in the location information management DB 132 is recognized, a captured image of the shopper and a list of ID candidates to be associated are displayed on the inspection terminal Q. Sent to. The examiner links the most appropriate ID to the shopper based on the past linking information between the shopper and the ID, and starts re-tracking by the moving object area tracking unit 3305 using the ceiling camera. If re-tracking could not be started for any reason, a clerk is notified to that effect.
In addition, when an ID that has lost the association with the shopper is detected, the ID and a list of captured images of the shopper who are not associated with any ID are transmitted to the inspection terminal Q. The examiner attempts to link the shopper to the ID based on the link information between the shopper and the ID in the past, and when the link is successful, the moving object area tracking unit 3305 by the ceiling camera Start retracking. If the start of re-tracking fails for any reason, the clerk is informed accordingly.
(C) The image list of each shopper and the list of IDs to be associated with each shopper are transmitted to the inspection terminal Q. The examiner attempts to allocate the ID to the shopper most appropriately based on the past association information between the shopper and the ID. When the ID can be appropriately assigned to the shopper, re-tracking by the moving object area tracking unit 3305 by the ceiling camera is started. If the ID cannot be appropriately assigned to the shopper, the ID is abandoned and the clerk is notified that the link has been changed.

[Other embodiments]
The present invention is not limited to the above-described embodiment, and includes modifications, improvements, and the like as long as the object of the present invention can be achieved.

For example, the present information processing system uses a method in which an examiner visually confirms an image including an object in a subject, and obtains a result of an attempt to identify the object as a product by using a method for visually confirming the object. And a payment means for performing a payment process on the product specified based on the result of the product specifying means.
Thereby, when the purchaser purchases the product, it is possible to automate the settlement of the price of the product while maintaining the same product specification accuracy as the manned cash register.

Further, for example, the present information processing system may include a shelf inventory management function. The shelf inventory management function includes a clerk discriminating unit and a stock information updating unit.
The clerk discriminating means discriminates between the clerk and the shopper. For example, the clerk discriminating unit assigns a physical marker that can identify the clerk to something that the clerk wears, such as a hat or clothes. The clerk is identified by imaging the physical marker with the ceiling camera and the shelf camera. This clerk discriminating means can be effectively used especially in the third and fourth embodiments.

  The clerk determining means may include a button provided in a predetermined area such as a backyard of the store. What is this clerk discriminating means? When a button is pressed by a clerk present in a predetermined area, an ID is assigned to the clerk. This clerk discriminating means can be effectively used particularly in the fourth embodiment. The clerk discriminating unit keeps track of the area of the person recognized as the clerk by the ceiling camera.

The inventory information updating means uses the object recognition unit and the product identification unit to update the inventory information of the shelf when the store clerk refills (adds) or removes or discards (subtracts) the product from the shelf. Update in the product DB of the server.
The stock information updating means updates, when a product is purchased, the stock on the shelf where the product is located and the stock information of the entire store.
The stock information updating unit manages the stock of the shelf and the stock information of the entire store, and notifies the remote operation unit or automatically places an order when the stock quantity falls below the threshold value.

  Further, the present invention may include a function of estimating the attribute of a shopper. The shopper's attribute estimation function estimates attributes of the shopper, such as rough age, age, and gender, for example, from face recognition. The function of estimating the attribute of the shopper is provided in the cashier terminal in the first embodiment, and is provided in the counter in the second to fourth embodiments. In the second embodiment, the cashier terminal may be provided. In addition, the present invention associates purchase information and shopper attribute information with a POS system. As the purchase information, information such as a product name and an amount is converted into data, and the information is linked to the POS system after the settlement is completed. As the shopper attribute information, information obtained by the shopper attribute estimation function described above is linked to the POS system.

  The function of estimating the attribute of the shopper may be linked with, for example, the sale-restricted product determination unit. As a result of estimating attributes, if the shopper apparently does not need age confirmation, the shopper's attribute estimation function may control the sales-restricted product determination unit not to confirm the age if the shopper is in the 30s or older. Good.

Here, the problems relating to the above-described first to fourth embodiments and other embodiments will be supplemented.
As a product recognition method in the product recognition system in each embodiment, image recognition using deep learning is promising. However, deep learning requires re-learning when a product cannot be identified or misrecognized, and in that case, it is necessary to give the image of the correct product manually, which requires a great deal of time and effort. Take it.

As a solution to the above-mentioned problem, the product recognition system of the present invention comprises:
At the time of product purchase, a function to correct the product recognition by performing a visual check by an inspector, a barcode scan, etc.,
A function of linking and recording the correct product image and necessary information (corrected product number, specific code, etc.) when the product recognition is corrected,
A function of automatically executing learning of deep learning based on the correct answer product image and necessary information,
A function to automatically deploy the learning results to the product recognition system,
By implementing the above, automation of the re-learning process of the above series of deep learning can be realized.
Here, the deployment of the learning result refers to a process of updating, for example, a deep learning model of the product recognition system based on the learning result so that the learning result can be used for product recognition.
The above-described re-learning process may be configured to be started with a predetermined condition such as a failure in specifying a product as a trigger, or may be configured to be started with an explicit operation by a person in charge, for example, pressing a button or the like as a trigger. Good.

  According to the re-learning process of deep learning as described above, it is possible to provide a product recognition system that can simplify complicated work for re-learning deep learning, reduce labor, and keep improving product recognition accuracy.

In each of the embodiments including the above-described first to fourth embodiments and other embodiments, as an image recognition method applied when specifying an object as a product, for example, object recognition using AI (Artificial Intelligence), a feature, Object recognition using points / features, shape recognition of product logos, character recognition (character shape recognition using AI, character recognition using OCR (Optical Character Recognition), etc.) can be adopted. is there. In addition to image recognition methods, weight recognition (weight recognition of an object using a weight sensor), product recognition by scanning identification information such as a bar code or a QR code (registered trademark) (bar recognition using an infrared sensor or image reading, etc.) Code, QR code (registered trademark), etc., and one or more recognition methods of product recognition by reading electronically recorded identification information such as RFID, to recognize an object as a product It is. In this case, it is possible to score the recognition result of the object by each recognition method, and to estimate that the object is a product having the highest score.
Further, the above-described recognition methods such as weight recognition, product recognition by scanning identification information such as barcodes and QR codes (registered trademark), and product recognition by reading electronically recorded identification information are combined with image recognition methods. Instead, it is possible to recognize an object as a product by combining one or more.

In each of the above embodiments, the inspection of the object or the commodity is performed by the inspection terminal Q, but the invention is not limited to this. That is, the inspection by the inspector may be performed by another device such as the cashier terminal 2, the checkout machine 4, and the information terminal 9.
Further, in each of the above-described embodiments, one inspection terminal Q is provided for one store or one cashier terminal 2, or one inspection terminal Q is provided for a plurality of stores or a plurality of cashier terminals. It is possible to provide.

  Further, the hybrid check system (inspection) in the above-described embodiment is not necessarily used only for product identification. That is, for example, since the product contains (A) a product such as tobacco and liquor which cannot be purchased until a certain age is reached, (B) a product whose consumption expires and the expiration date has expired, and (C) an allergic component, It can also be used to determine whether or not it is a product that should not be ingested, (D) a product other than halal food, etc., or a product that is restricted by religion.

In the present invention, the image processing by the CPU may be processing by a GPU (Graphic Processing Unit).
Further, the illustrated hardware configuration and block diagram are merely examples for achieving the object of the present invention, and the present invention is not limited to the illustrated example.
Further, the location of the functional block is not limited to the illustrated one, and may be arbitrary. For example, one functional block may be constituted by hardware alone, may be constituted by software alone, or may be constituted by a combination thereof. When the processing of each functional block is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium.

The computer may be a computer incorporated in dedicated hardware, or may be a smartphone or a personal computer.
Further, the processing of the trading restricted product described in the first embodiment can be applied to the second to fourth embodiments. However, the trading restricted product in the second embodiment is applied to an age restricted product.

In this specification, the steps of describing a program recorded on a recording medium may be performed in chronological order according to the order, or may be performed in parallel or individually. Also includes the processing executed in
Also, in this specification, the term system refers to an entire device including a plurality of devices and a plurality of means.

In summary, the information processing system to which the present invention is applied only needs to have the following configuration, and can adopt various embodiments other than the above-described first to fourth embodiments.
That is, the information processing system
A first specifying unit (for example, a visual inspection method) that obtains a result of an attempt to identify the object as a product by using a first method in which an examiner visually confirms an image including an object in a subject. A result acquisition unit 239 and an inspection terminal Q),
A settlement unit (for example, a settlement unit 237) that performs a settlement process on the product specified based on the result of the first specification unit;
Is provided.
Thereby, when the purchaser purchases the product, it is possible to automate the settlement of the price of the product while maintaining the same product specification accuracy as the manned cash register.
Furthermore, such an automatic payment processing system enables a procedure for purchasing a product to be completed between the purchaser and the system, and can lower the hurdle for realizing an unmanned store.

A second specifying unit (for example, an object recognizing unit 233) that attempts to specify the object as a product using a second method other than the first method,
The settlement unit may be configured to perform a settlement process on a product specified based on at least one of the first specifying unit and the second specifying unit.

  The second method may be configured to include at least a predetermined image recognition method (for example, deep learning).

For the object not identified as a product by the predetermined image recognition method,
The first specifying unit and the second specifying unit associate product information of the object specified as a product using one or more methods not including the predetermined image recognition method with the image,
Using information including the linked product information and the image, re-learning the predetermined image recognition method,
The image processing apparatus may further include an image recognition re-learning unit that deploys a learning result obtained by the re-learning to the second specifying unit.

  The second method may include at least one of weight recognition, scanning of identification information, and reading of identification information recorded electronically.

  For the object not specified as a product by the second method, the product specified based on the result of the first specifying unit may be a target of the payment processing by the payment unit.

  The product specified based on the respective results of the first specifying unit and the second specifying unit may be configured to be a target of the settlement processing by the settlement unit.

  DESCRIPTION OF SYMBOLS 1 ... Server, 2,2-1,2-n ... Cashier terminal, 3 ... Sales counter, 4,4-1,4-n, 5a ... Checkout machine, 5,5-1 , 5-2, 5-3, 5-n: payment gate, 6: payment register, 9, 9a, 9b: information terminal, 10, 20, 30, 40: store, 11, 22 ... entrance, 12, 25 ... cash register counter, 13, 26 ... manned register, 14, 23, 33, 43 ... shelf rack, 15, 24, 34, 44 ... passage, 21 gate, 31, 41 entrance, 32, 42 exit, 35, 45 settlement area, 36 register unit, 101, 201, 301, 401, 501, etc. CPU, 102, 202, 302, 502 ... ROM, 103, 203, 303, 503 ... RAM, 104, 204, 04, 504: bus, 105, 205, 305, 505: input / output interface, 106, 206, 406, 506: output unit, 107, 207, 407, 507: input unit, 108, 208, 508 ... storage unit, 109, 213, 315, 409, 509 ... communication unit, 110, 214 ... drive, 120, 220 ... removable media, 131 ... product DB, 132 ··· Position information management DB, 141 ··· DB management unit, 150 ··· error determination unit, 151 ··· error display unit, 152 ··· error release unit, 209 ··· light blocking unit, 210 ··· Presentation unit, 211: cash register camera, 212: external camera, 221: illumination unit, 228: light emission control unit, 229: light shielding control unit, 230: presentation control unit 231,320: Individual authentication unit, 232: Image acquisition unit, 233: Object recognition unit, 234: Object quantity recognition unit, 235: Product identification unit, 236: Trading restricted products Judgment unit, 237, 435, 535: settlement unit, 238: display control unit, 239, 301a: inspection result acquisition unit, 241: DB information holding unit, 270: surrounding unit, 271, top plate portion, 272, bottom plate portion, 273, side plate portion, 275, housing portion, 276, plate, 310, ceiling camera, 311, shelf camera, 312: basket camera, 321: personal information acquisition unit, 330: moving object tracking unit, 340: position information management unit, 341: inter-camera information transfer unit, 342: each camera Position definition unit, 343 ... moving object display unit, 3 50: Book number counting section, 351: Book number recognition section, 352: Moving object and book number linking section, 353: Book number unspecified determination section, 354: Book number management section linked to moving object 355: recognition unit for transfer between moving object areas by ceiling camera; 356: recognition unit for number of books transferred by ceiling camera; 360: shelf product recognition unit; 370: basket product recognition unit; ... Cargo in / out detection unit by basket camera, 373 ... Object recognition unit by basket camera, 374 ... Product specification unit by basket camera, 375 ... Product non-specification determination unit of basket product, 376 ... Cart product label recognition unit, 377: Basket product discount seal recognition unit, 380: Trading restricted product determination unit, 390: Remote operation unit, 391, D: Display unit, 411 ... Image display control 412, an inspection result transmission unit, 436, 536, an input control unit, 437, 537, an output control unit, 3302, a moving object detection unit using a ceiling camera, 3303, a ceiling camera Car discovery unit, 3304: Moving object region definition unit by ceiling camera, 3305: Moving object region tracking unit by ceiling camera, 3306: Basket region definition unit by ceiling camera, 3307: Basket by ceiling camera Area tracking section, 3308... Grouping section, 3309... Blood relation determination section, 3310... Basket area transfer recognition section by ceiling camera, 3311... Moving object area transfer recognition section by ceiling camera, 3312 ... Object recognition unit passed by ceiling camera, 3313 ... Product identification unit passed by ceiling camera, 360 ... Object recognition unit by shelf camera, 3603: Product identification unit by shelf camera, 3604: Moving object and product linking unit, 3605: Basket and product linking unit, 3606: Moving object Product list management unit linked to a basket, 3607: Product list management unit linked to a basket, 3608: Object entry / exit detection unit by shelf camera, 3609: Product unspecified determination unit, 3610: Label recognition unit .., 3611... Discount seal recognition unit, 3612... Basket detection unit by shelf camera or ceiling camera, A... Predetermined area, C... Card reader unit, Mo... Moving object, Mt.・ Sales clerk, M: Microphone, N: Network, Q: Visual inspection terminal, R: Receipt part, S: Speaker

Claims (13)

In an information processing system used in a store where one or more people including a product purchaser exist,
For an image including the object in the subject, the examiner who visually checks the object and attempts to specify the object as a product to be purchased by the purchaser uses the first method of visually checking the object. First specifying means for obtaining a result of an attempt to specify as a product,
When the object is specified as a product based on the result of the first specifying unit, a payment unit that performs a payment process on the specified product;
Presenting means for presenting predetermined information including information contributing to specifying the object as a product when the object is not specified as a product based on the result of the first specifying device;
An information processing system comprising: The predetermined information presented by the presentation means is:
If the object is determined as a trade-restricted product, and if the image regarding the purchaser of the trade-restricted product is determined to be unclear by the first method, including information for contacting the purchaser,
The information processing system according to claim 1. Further comprising a cashier terminal or a sales floor device used by the purchaser for the product corresponding to the object,
The predetermined information presented by the presentation means is:
Includes information on how to use the cashier terminal or the counter device to the purchaser ,
The information processing system according to claim 1. Using a second method other than the first method, the apparatus further includes a second specifying unit that attempts to specify the object as a product purchased by the purchaser,
The settlement unit, when the object can be specified as a product based on at least one of the first specification unit and the second specification unit, performs a payment process on the specified product. Do,
The information processing system according to claim 1. The predetermined information presented by the presentation means is:
If the object is determined as a trade-restricted product, and if the image of the purchaser of the trade-restricted product is determined to be unclear by the first method, including information for contacting the purchaser,
The information processing system according to claim 4. Further comprising a cashier terminal or a sales floor device used by the purchaser for the product corresponding to the object,
The predetermined information presented by the presentation means is:
Includes information on how to use the cashier terminal or the counter device to the purchaser ,
The information processing system according to claim 4. The second method includes at least a predetermined image recognition method,
The information processing system according to claim 4. For the object not identified as a product by the predetermined image recognition method,
The first specifying unit and the second specifying unit specify the object as a product using one or more methods different from the predetermined image recognition method, and determine the product information of the specified object and the image. Tied,
Using information including the linked product information and the image, re-learning the predetermined image recognition method,
An image recognition re-learning unit that deploys a learning result obtained by the re-learning in the second specifying unit;
The information processing system according to claim 7, further comprising: The second method includes at least one of weight recognition, scanning of identification information, and reading of electronically recorded identification information.
The information processing system according to claim 4. For the object not specified as a product by the second method, a product specified based on a result of the first specifying unit is a target of the payment process by the payment unit.
The information processing system according to claim 4. The product specified based on the respective results of the first specifying unit and the second specifying unit is a target of the payment process by the payment unit.
The information processing system according to claim 4. In an information processing system used in a store where one or more people including a product purchaser exist,
An information processing method executed by an information processing system,
For an image including the object in the subject, an examiner who visually checks the object and attempts to specify the object as a product to be purchased by the purchaser uses a method of visually checking the object, and Identifying a result of attempting to identify as
When the object is specified as a product based on the result of the process of the specifying step, a payment step of performing a payment process for the specified product,
A presentation step of presenting predetermined information including information that contributes to specifying the object as a product when the object is not specified as a product based on the result of the processing of the specifying step ;
An information processing method including: In an information processing system used in a store where one or more people including a product purchaser exist,
For an image including the object in the subject, an examiner who visually checks the object and attempts to specify the object as a product purchased by the purchaser uses a method of visually checking the object to determine the product. Identifying means for obtaining a result of an attempt to identify as;
When the object is specified as a product based on the result of the specifying unit, a payment unit that performs a payment process for the specified product,
Presenting means for presenting predetermined information including information that contributes to specifying the object as a product when the object is not specified as a product based on the result of the specifying unit,
A computer that controls an information processing system including
A specifying step of causing the specifying means to function;
A settlement step of function the fine calculation means,
A presentation step for causing the presentation means to function;
A program for executing a control process including

Images