JP6283806B2 - Information processing system - Google Patents

Description

  The present invention relates to an information processing system.

  Traditionally, in stores such as convenience stores, supermarkets, shopping centers, and various mass retailers, buyers pick up products from the product shelves, put them in shopping carts and shopping baskets, and pay for the products near the store exits. To the cash register terminal for paying out the goods and paying the price of the goods using a self cash register or the like (for example, see Patent Document 1).

However, when the cash register terminal is used to pay for the product in the shopping cart or shopping basket, even if the self-cash register terminal is used for payment, the barcode for each product is used to identify one of the products. Will be read. For this reason, it is not possible to eliminate the queue of purchasers waiting for settlement, and the purchaser waits for payment for a long time by the cash register terminal.
In addition, when a purchaser purchases a product that he / she wants to purchase at a store, if there are many other purchasers, the purchaser does not want to wait for the cash register terminal to settle for a long time, and as a result, the purchase itself May give up.
In consideration of the above circumstances, when a purchaser purchases a product displayed in a store, it is possible to automate the settlement of the price of the product and shorten the time required for the settlement of the price of the product. The system is requested.
Therefore, when a product is placed in a predetermined area such as a cash register, the present applicant images the predetermined area with a camera, recognizes the object of the product candidate from a captured image obtained as a result, and the object (product candidate) ) Is specified, and a patent application has already been filed for a next-generation cash register system that can execute a series of processing until the payment for the specified product is settled (Japanese Patent Application No. 2006-080624). .

JP 2001-76261 A

  However, in the next-generation cash register system for which a patent application has been filed by the applicant, there is further support for recognizing stacked products, support for recognizing products without features on the top, support for monitoring fraud, and support for tracking fraud. There was a desire to realize a technology that would be possible.

  The present invention has been made in view of such circumstances, and enables recognition support for stacked products, support for recognition of products without features on the top, support for fraud monitoring, and support for tracking fraud. The purpose is to make this technology feasible.

In order to achieve the above object, an information processing system according to one embodiment of the present invention includes:
An information processing system functioning as a system including a cash register terminal,
Acquisition means for acquiring, as an object captured image, a plurality of captured images obtained as a result of imaging each of one or more objects placed in a predetermined area from a plurality of directions;
Management means for storing and managing the object captured image in a predetermined database.

  In addition, the information processing system counts the number of the recognized one or more objects, the recognition means for recognizing the existence of the one or more objects based on the acquired object captured image, and recognizes the result. Recognized quantity counting means for generating as quantity information, a settlement quantity counting means for counting the quantity of the one or more objects settled as products, and generating the result as adjusted quantity information, recognized quantity information and adjusted quantity information Judgment means for judging whether or not they are the same and presentation means for presenting a warning when the recognized quantity information and the adjusted quantity information are different can be further provided.

  In addition, the information processing system may be configured such that a recognition unit that recognizes the presence of the one or more objects based on the acquired object captured image and the one or more objects for which the existence is recognized are any product. A specifying means for specifying whether there is a checkout means for paying out the specified product, a judging means for determining whether or not all of the specified product has been settled, and when all of the products are not settled And a presenting means for presenting a warning.

Further, the acquisition means further acquires a captured image obtained by capturing a predetermined position where a purchaser is assumed to exist for the predetermined area as a human captured image,
The management unit can store and manage the object captured image in association with the person captured image in a predetermined database.

  In addition, when the warning by the presenting unit is presented as a result of the determination by the determining unit, the managing unit can separately manage the acquired human captured image in a predetermined database. .

  In addition, the information processing system includes a person who is a subject included in the person captured image acquired by the acquiring unit, and a person who is a subject included in the person captured image separately stored and managed by the management unit. It is possible to further provide a comparison means for causing the presenting means to present the fact when the results of the comparison match.

  Further, the acquisition unit can acquire a captured image of one or more objects passing through the predetermined area as a further object captured image.

  An information processing method and program of one embodiment of the present invention are a method and program corresponding to the above-described information processing system of one embodiment of the present invention.

  According to the present invention, it is possible to provide support for recognizing products placed on top of each other, support for recognizing products having no upper features, support for monitoring fraud, and support for tracking fraud.

It is a lineblock diagram of a goods recognition system as a 1st embodiment of an information processing system of the present invention. It is a block diagram which shows the hardware constitutions of the server which comprises the goods recognition system of FIG. It is a block diagram which shows the hardware constitutions of the cash register terminal which comprises the goods recognition system of FIG. It is an external appearance top view of the cash register terminal of FIG. It is a functional block diagram which shows an example of a functional structure with the server of FIG. 2, and the cash register terminal of FIG. It is a flowchart explaining the quantity collation process which the server and cash register terminal of FIG. 5 perform. It is a flowchart explaining the quantity collation process which the server and cash register terminal of FIG. 5 perform. It is a block diagram of the modification of 1st Embodiment of the information processing system of this invention.

First, the outline of the information processing system of the present invention will be described.
The product recognition system (next generation cash register system) to which the information processing system of the present invention is applied can solve the following problems (1) to (3).
(1) When imaging is performed only from the top of the cash register, it is not possible to image the existence of all products (objects) such as products (objects) placed on top of each other and products (objects) without features on the top. Here, as a product having no features in the upper portion, for example, a product such as a can, a plastic bottle, a soup, or the like whose presence can be clarified when imaged from another direction (lateral direction, etc.) is applicable.
(2) Not aware of shoppers' fraud (shoplifting, etc.) and cannot confirm later. In particular, they do not notice the fraud called Sweet Hearting by the cashier clerk (a fraudulent act that conspires with family, friends, and colleagues and does not intentionally scan the product).
(3) In the case of self-checkout, shoppers think that shoppers are more likely to cheat psychologically than manned cashiers because they are not looking at it.

In this way, simply imaging a product (object) from one direction cannot capture images of products (objects) placed on top of each other or products hidden behind other products (objects). You may not notice.
In this specification, what is not specified as a product is called an “object”. In the present specification, a product in which “object” is specified as a product is referred to as a “product”.
In this specification, a plurality of captured images obtained by capturing an object are referred to as “object captured images”. A captured image obtained by capturing a person (purchaser) who purchases an object (product) is referred to as a “person captured image”. Further, in this specification, the result of counting the object (product) before settlement is referred to as “recognized quantity information”. Furthermore, the result of counting the products after settlement is called “settlement quantity information”.
In this specification, a database in which information for specifying a product is stored is referred to as a “product DB”. A database in which “object captured image” and “person captured image” are stored is referred to as “image DB”.

An information processing system to which the present invention is applied has at least the following functions.
That is, the information processing system to which the present invention is applied has an acquisition function and a management function.
The acquisition function is a function of acquiring a plurality of captured images obtained as a result of imaging one or more objects placed in a predetermined area (product recognition area) from a plurality of directions as an object captured image. The acquisition function is a function of acquiring a captured image obtained by capturing a predetermined position where a purchaser is assumed to exist in a predetermined area as a human captured image.
The management function is a function of storing and managing the acquired “object captured image” and “person captured image” in a database (hereinafter referred to as “image DB”).
Here, the “predetermined area” is an area where one or more objects are placed for settlement, and is a “product recognition area” where imaging of one or more objects is performed.

Such an information processing system to which the present invention is applied can execute the following process (A).
Process (A) refers to the following process.
That is, an information processing system to which the present invention is applied exhibits an acquisition function, and picks up a plurality of captured images acquired as a result of capturing images of one or more objects placed in a predetermined area from a plurality of directions. Obtain as an image.
Next, the information processing system to which the present invention is applied exhibits a management function, and stores and manages the acquired object captured image in the image DB.
In addition, the information processing system to which the present invention is applied may acquire the captured image obtained by capturing the predetermined position where the purchaser is assumed to be present in the predetermined area as the human captured image by performing the acquisition function. Good. At this time, the information processing system to which the present invention is applied may exhibit a management function and associate and store the acquired object captured image and person captured image in the image DB for management.
As a result, it becomes possible to provide support for recognizing stacked products, support for recognizing products without features on the top, support for monitoring fraud, and support for tracking fraud.

Here, there are one or more sensing devices (cameras or the like) that capture the product. That is, a product may be imaged from a plurality of directions with a plurality of devices, or a product may be imaged from a plurality of directions by moving one device or the like.
“Imaging” means that a predetermined area is divided into a plurality of areas, and each pixel corresponding to each of the plurality of divided areas has a pixel value. The pixel value may be any value such as temperature and distance in addition to the amount of light. That is, the sensing device may be an image sensor (such as a camera) that accumulates (detects) the amount of light for each pixel, a temperature sensor that detects a temperature for each pixel, a distance sensor that detects a distance for each pixel, and the like.

In addition to the functions described above, the information processing system to which the present invention is applied can further include a recognition function, a recognized quantity counting function, a settlement quantity counting function, a determination function, and a presentation function.
The recognition function is a function for recognizing the presence of one or more objects based on the acquired object captured image.
The recognized quantity specifying function is a function that counts the quantity of one or more recognized objects and generates the result as recognized quantity information.
The settlement quantity counting function is a function for counting the quantity of one or more objects settled as a product and generating the result as recognition quantity information.
The determination function is a function for determining whether or not the recognized quantity information and the adjusted quantity information are the same.
The presentation function is a function that presents a warning when the recognized quantity information and the adjusted quantity information are different.

Such an information processing system to which the present invention is applied can execute the following process (B).
Process (B) refers to the following series of processes.
That is, the information processing system to which the present invention is applied recognizes an object by exerting a recognition function. Here, recognizing an object means specifying an area by an image processing method such as background difference.
Next, the information processing system to which the present invention is applied exhibits the recognized quantity counting function, counts the number of recognized objects, and generates the result as recognized quantity information.
Next, the information processing system to which the present invention is applied exhibits the settlement quantity counting function, counts the quantity of one or more objects settled as a product, and generates the result as settlement quantity information.
Next, the information processing system to which the present invention is applied causes a determination function to determine whether the recognized quantity information and the adjusted quantity information are the same.
Next, the information processing system to which the present invention is applied exhibits a presentation function and presents a warning when the recognized quantity information and the adjusted quantity information are different.
Accordingly, a warning can be presented when the quantity of objects placed in a predetermined area before settlement is different from the quantity of merchandise after settlement.
Here, “presenting a warning” means notifying using a display, sound, light or the like that the quantity of recognized objects is different from the quantity of merchandise that has been settled.

On the other hand, the information processing system to which the present invention is applied can also include a recognition function, a specific function, a settlement function, a determination function, and a presentation function.
The recognition function is a function of recognizing the presence of one or more objects based on the acquired object captured image.
The specific function is a function for specifying which product is one or more recognized objects.
The checkout function is a function for checking out the specified product.
The determination function is a function for determining whether or not all products have been settled.
The presentation function is a function that presents a warning when all products are not settled.

Such an information processing system to which the present invention is applied can execute the following process (C).
Process (C) refers to the following series of processes.
That is, the information processing system to which the present invention is applied recognizes an object by exerting a recognition function. Here, recognizing an object means defining an area by an image processing method such as background difference.
Next, the information processing system to which the present invention is applied causes a specific function to be specified, and specifies which commodity each of the one or more objects whose existence is recognized is. Note that image recognition techniques such as specific object recognition and general object recognition can be used for specifying an object.
Next, the information processing system to which the present invention is applied exhibits the settlement function and settles the specified product.
Next, the information processing system to which the present invention is applied makes a determination function to determine whether or not all the specified products have been settled. This determination is made based on whether or not the specified product is included in the settled product.
Next, the information processing system to which the present invention is applied exhibits a presentation function and presents a warning when all products are not settled.
Thereby, a warning can be presented when the product before settlement is not included in the product after settlement.
Here, “presenting a warning” means notifying using a display, sound, light or the like that the quantity of recognized objects is different from the quantity of merchandise that has been settled.

Further, in the information processing system to which the present invention is applied, the management function described above includes the acquired human captured image when a warning using the presentation function is presented as a result of the determination using the determination function. A function of separately storing and managing in the image DB can be included.
Such an information processing system can execute the following process (D).
Process (D) refers to the following process.
That is, the information processing system to which the present invention is applied allows the management function to be exhibited, and when the warning using the presentation function is presented as a result of the determination using the determination function, the acquired human captured image is displayed. , Separately stored in the image DB and managed.
Thereby, it is possible to distinguish and manage a purchaser who seems to have performed fraud.
Here, “separately manage” means storing a person captured image acquired when a warning is presented, distinguishing it from other person captured images, or setting a flag for management.

The information processing system to which the present invention is applied can have a display control function.
The display control function sequentially displays at least a part of the object captured image acquired using the acquisition function and the image processed or created based on the object captured image to the purchaser and the store clerk. This function executes control.
Such an information processing system can execute the following process (E).
Process (E) refers to the following process.
That is, the information processing system to which the present invention is applied exhibits at least the object captured image acquired using the acquisition function and the image processed or created based on the object captured image by performing the display control function. Control is performed to sequentially display some images to the purchaser and the store clerk.
Thus, fraud can be suppressed. In particular, the psychological effect of hesitating or giving up cheating by displaying the purchaser's face together is also increased.

The acquisition function described above can include a function of acquiring a captured image of one or more objects passing through a predetermined area as a further object captured image.
Such an information processing system can execute the following process (F).
Process (F) refers to the following process.
That is, the information processing system to which the present invention is applied exhibits an acquisition function to acquire a captured image of one or more objects passing through a predetermined area as a further object captured image.
Thereby, since one or more objects can be imaged for a long time, the number of object captured images with different viewpoints of one or more objects can be increased, and the recognition rate of one or more objects can be improved. Further, when one or more objects pass through a predetermined area, it becomes possible to place the objects without being affected by the size of the predetermined area, thereby preventing one or more objects from being placed on top of each other. it can.
Here, “passing” means moving through a predetermined area using an automatic conveyance mechanism such as a belt conveyor.

Next, an embodiment of an information processing system of the present invention will be described with reference to the drawings.
FIG. 1 is a configuration diagram of an embodiment of an information processing system of the present invention.
The information processing system of the present embodiment is capable of recognizing (supporting) stacked products, recognizing (supporting) products without features on the top, monitoring fraud (support), and tracking (support) fraud. In order to realize this technology, the configuration shown in FIG. 1 is provided.
That is, the information processing system as an embodiment of the present invention includes a server and n (n is an arbitrary integer) cash register terminals 2-1 to 2 -n.
The server 1 and the cashier terminals 2-1 to 2-n are connected to each other via a network N such as the Internet line.
For convenience of explanation, only one server in FIG. 1 is drawn, but there may actually be a plurality of servers.
Hereinafter, when it is not necessary to distinguish the cashier terminals 2-1 to 2-n individually, these are collectively referred to as “the cashier terminal 2”.

The server 1 executes various processes to manage each operation of the cash register terminal 2.
The cash register terminal 2 includes a product recognition area, one or more cameras, and two display units. The product recognition area is an area where an object to be recognized is placed. One or more cameras image an object placed in the product recognition area, or image a purchaser (a predetermined position where a purchaser is estimated to be present). One of the two display units displays a purchaser's screen for presenting information relating to the product quantity (score), information relating to payment, and the purchaser's face image to the purchaser. The other of the two display units displays a seller screen for presenting information relating to the product quantity (score), information relating to payment, and the purchaser's face image to the seller. A specific example of the external configuration of the cash register terminal will be described later with reference to FIG.

In the information processing system of the present embodiment, the following operation (processing) is executed.
That is, in the information processing system of the present invention, the server 1 captures a captured image (person captured image) obtained by capturing a predetermined position where a purchaser is assumed to exist and an object placed in the product recognition area from a plurality of directions. A plurality of captured images (object captured images) obtained as a result are associated with each other and stored and managed in the image DB 401 (FIG. 5).

When the object is placed in the product recognition area of the cash register terminal 2, the cash register terminal 2 captures the placed object and acquires a plurality of captured images obtained as a result of the imaging as an object captured image.
The cash register terminal 2 captures a predetermined position (predetermined position where it is estimated that a purchaser is present) in the product recognition area, and acquires a captured image obtained as a result of the imaging as a human captured image.
The cash register terminal 2 stores the object captured image and the person captured image in association with each other in the image DB 401 and manages them.
The cash register terminal 2 recognizes the presence of an object placed in the product recognition area by using the predetermined image recognition method described above for the acquired captured image.
The cash register terminal 2 counts the number of recognized objects (number of object points) and generates the result as recognition quantity information.
The cash register terminal 2 counts the quantity of goods that have been settled (the number of goods) and generates the result as the settlement quantity information.
The cash register terminal 2 determines whether the recognized quantity information and the adjusted quantity information are the same.
The cash register terminal 2 presents a warning when the recognized quantity information and the adjusted quantity information are different.
In this case, the cashier terminal 2 can prevent fraud by presenting a warning that there is something that has not been settled.

The cash register terminal 2 can also separately store a captured image obtained by capturing a predetermined position in the product recognition area in the image DB 401 when the recognized quantity information and the adjusted quantity information are different. Then, the cash register terminal 2 can present a warning when the same person is imaged as a subject after the next time. Specifically, when the same person is imaged as a subject after the next time, the cashier terminal 2 can display that fact on the display unit, assuming that the person who performed the fraud is trying to settle again.
Details of these processes executed by the server 1 will be described later with reference to the functional block diagram of FIG.

  FIG. 2 is a block diagram illustrating a hardware configuration of the server 1 configuring the information processing system of FIG.

  The server 1 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a bus 104, an input / output interface 105, an output unit 106, an input unit 107, A storage unit 108, a communication unit 109, and a drive 110.

The CPU 101 executes various processes according to a program recorded in the ROM 102 or a program loaded from the storage unit 108 to the RAM 103.
The RAM 103 also stores data necessary for the CPU 101 to execute various processes as appropriate.

  The CPU 101, ROM 102, and RAM 103 are connected to each other via a bus 104. An input / output interface 105 is also connected to the bus 104. An output unit 106, an input unit 107, a storage unit 108, a communication unit 109, and a drive 110 are connected to the input / output interface 105.

The output unit 106 includes a display, a speaker, and the like, and outputs various types of information as images and sounds.
The input unit 107 includes a keyboard, a mouse, a touch panel, and the like, and inputs various information.

The storage unit 108 includes a hard disk, a DRAM (Dynamic Random Access Memory), and the like, and stores various data.
The communication unit 109 communicates with other devices (the cashier terminals 2-1 to 2-n in the example of FIG. 1) via the network N including the Internet.

A removable medium 120 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 110. The program read from the removable medium 120 by the drive 110 is installed in the storage unit 108 as necessary.
The removable medium 120 can also store various data stored in the storage unit 108 in the same manner as the storage unit 108.

  FIG. 3 is a block diagram showing a hardware configuration of the cash register terminal 2 constituting the information processing system of FIG.

The cash register terminal 2 includes a CPU 201 to a drive 211.
The output unit 206 includes a display (display unit D), a speaker, and the like, and outputs various types of information as images and sounds.
The imaging unit 208 includes one or more cameras and the like, and images the product and the purchaser of the product.
Each of the CPU 201 through the input / output interface 205, the input unit 207, the storage unit 209, and the drive 211 has basically the same functions and configurations as the CPU 101 through the drive 110 in FIG. Therefore, these descriptions are omitted.

A removable medium 220 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 211. The program read from the removable medium 220 by the drive 211 is installed in the storage unit 209 as necessary.
The removable medium 220 can also store various data stored in the storage unit 209 in the same manner as the storage unit 209.

  FIG. 4 is a diagram showing an example of an external configuration of the cash register terminal 2 of FIG.

As shown in FIG. 4, the cash register terminal 2 includes a product recognition area A, seven cameras (imaging unit 208), and two display units D (output unit 206). The product recognition area A is an area in which a product to be recognized is placed.
One of the two display units D is a captured image (person) that captures information on the quantity of recognized products (recognized quantity) and information on the quantity of paid goods (settlement quantity) and the position where the purchaser is assumed to exist. A purchaser screen for presenting the captured image) to the purchaser is displayed. The other of the two display units D is a captured image (person captured image) obtained by capturing information regarding the recognized quantity (recognized quantity) and the settled quantity (settlement quantity) and the position where the purchaser is assumed to exist. A screen for the seller for presenting to the seller is displayed.
Note that one of the seven cameras shown in FIG. 4 is installed so that the product recognition area A is looked down from above. Of the seven cameras, one is a captured image that captures information regarding the number of recognized objects (number of objects) and the number of items that have been settled (number of products) and the position where the purchaser is assumed to be present ( The person (captured person image) is installed on the upper part of the display unit D for presenting the purchaser, and the purchaser (position where the purchaser is assumed to exist) is imaged. Of the seven cameras, each of the four cameras is arranged by shifting by a predetermined angle (90 degrees in the present embodiment) in the circumferential direction, and images the product placed in the product recognition area A from the four directions on the side surface. Of the seven cameras, one is arranged below the product recognition area A and facing upward.

  FIG. 5 is a functional block diagram showing an example of a functional configuration of the server 1 of FIG. 2 and the cash register terminal 2 of FIG.

In the CPU (FIG. 2) of the server 1, a DB management unit 301 functions as shown in FIG.
An image DB 401 and a product DB 402 are provided in one area of the storage unit 108 (FIG. 2).

  The DB management unit 301 presumes that there is a purchaser for the object captured image obtained as a result of imaging one or more objects placed in the product recognition area A from a plurality of directions and the product recognition area A. The image is stored and managed in the image DB 401 in association with the person-captured image captured at the predetermined position. Further, the DB management unit 301 separately stores and manages the person captured image in the image DB 401 when the recognized quantity information and the settlement quantity information are different. Further, the DB management unit 301 manages product information stored in the product DB 402. In addition, the DB management unit 301 transmits a human captured image stored separately to the cash register terminal 2 via the communication unit 109. Note that the DB management unit 301 may store and manage only the object captured image in the image DB 401. When the DB management unit 301 manages only the object captured image, the process using the person captured image is not performed below.

  In the CPU 201 (FIG. 3) of the cash register terminal 2, as shown in FIG. 5, a recognition button determination unit 230, an image acquisition unit 229, a comparison unit 231, a presence recognition unit 232, a display control unit 234, and a recognition The quantity counting unit 233, the product specifying unit 239, the settlement unit 235, the settlement quantity counting unit 236, the determination unit 237, and the presentation unit 238 function.

  The recognition button determination unit 230 determines whether the recognition button has been pressed. Specifically, the recognition button determination unit 230 determines whether or not the product recognition button on the screen displayed on the display unit D of the cash register terminal 2 has been pressed.

  The image acquisition unit 229 acquires a captured image (object captured image) of one or more objects placed in the product recognition area A, which is captured by the imaging unit 208 (FIG. 3). In addition, the image acquisition unit acquires a captured image (person captured image) obtained by capturing an image of a person who is assumed to be present at a predetermined position (before the product recognition area A) with respect to the product recognition area A.

  The comparison unit 231 compares the purchaser by referring to a person captured image separately stored and managed by the DB management unit 301. Specifically, the comparison unit 231 extracts a face part from a person captured image that is separately stored and managed and a person captured image acquired by the image acquisition unit 229. The comparison unit 231 compares whether or not both extracted faces match.

The presence recognition unit 232 recognizes the presence of an object placed in the product recognition area A using the predetermined image recognition method described above.
The display control unit 234 executes control for sequentially displaying at least a part of the acquired object captured image and an image processed or created based on the object captured image to the purchaser and the store clerk. . Specifically, the display control unit 234 sequentially outputs at least a part of the acquired object captured image and an image processed or created based on the object captured image, and the output unit 206 (display unit D). Executes the control displayed on the screen. In addition, the display control unit 234 executes control to display the person captured image on the output unit 206 (display unit D).
The recognized quantity counting unit 233 counts the quantity of objects whose existence is recognized, and generates the result as recognized quantity information. Accordingly, the recognized quantity counting unit 233 counts the number of products (objects) that the purchaser plans to purchase.

  The product specifying unit 239 specifies which product the object placed in the product recognition area A is. For example, the product specifying unit 239 matches information obtained from an object whose existence is recognized with product information held by the DB information holding unit 241. As a result, the product whose presence is recognized by the presence recognition unit 232 is identified as which product among the product information held in the DB information holding unit 241.

  The settlement unit 235 settles the product placed in the product recognition area A. For example, the settlement unit 235 settles the product by individually identifying the product placed in the product recognition area A.

  The settlement quantity counting unit 236 counts the quantity (product points) of the settled products and generates the result as the settlement quantity information. As a result, the settlement quantity counting unit 236 counts the quantity of the merchandise that has been settled by the purchaser. Specifically, the checkout quantity counting unit 236 can generate the checkout quantity information using the object captured image after the checkout unit 235 finishes the checkout of the product. Further, the settlement quantity counting unit 236 can generate settlement quantity information by counting the number of products actually specified by the settlement unit 235.

  The determination unit 237 determines whether the recognized quantity information and the adjusted quantity information are the same. Specifically, the determination unit 237 compares the quantity of the commodity scheduled to be purchased indicated by the recognized quantity information with the quantity of the actually settled commodity indicated as the settlement quantity information.

  The presentation unit 238 presents a warning when the recognized quantity information and the adjusted quantity information are different. Specifically, the presentation unit 238 receives a determination result by the determination unit 237, and presents a warning when the quantity indicated by the recognized quantity information is different from the quantity indicated by the adjusted quantity information. For example, the presentation unit 238 can display a warning on the display unit D, or can present the warning using sound, light (lamp), or the like.

  The DB information holding unit 241 holds a person captured image that is separately stored in the image DB 401. Further, the DB information holding unit 241 holds product information. The human captured image and the product information are transmitted from the DB management unit 301 of the server 1 to the DB information holding unit 241. As described above, the human captured image held by the DB information holding unit 241 is used as a material for comparison by the comparison unit 231.

Next, in the information processing system of the present invention, a series of processing (hereinafter referred to as “quantity matching processing”) from when a product is placed in the product recognition area A until the product is settled will be described.
6 and 7 are flowcharts for explaining the quantity matching process executed by the server 1 and the cash register terminal 2 in FIG.

  In step S1, it is determined whether or not the recognition button has been pressed after the product to be purchased is placed in the product recognition area A by the purchaser. The recognition button determination unit 230 determines whether or not the purchase button has been pressed by the purchaser.

  If it is determined in step S1 that the recognition button has been pressed, YES is determined in step S1, and the process proceeds to step S2. On the other hand, if it is not possible to confirm that the recognition button is pressed, NO is determined in step S1, and the process waits until the recognition button is pressed.

  In step S <b> 2, the imaging unit 208 images a predetermined position (for example, before the product recognition area A) with respect to the product recognition area A of the cash register terminal 2. The imaging unit 208 images the product recognition area A of the cash register terminal 2 from a plurality of directions (up / down / left / right / front / back).

  In step S <b> 3, the image acquisition unit 229 acquires a captured image obtained by capturing a predetermined position with respect to the product recognition area A of the cash register terminal 2. Further, the image acquisition unit 229 acquires a captured image of the product recognition area A of the cash register terminal 2.

  In step S <b> 4, the image acquisition unit 229 sends the captured images (object captured image and person captured image) to the display control unit 234. The display control unit 234 sequentially displays at least a part of images processed or created based on the object captured image and the person captured image sent from the image acquisition unit 229 to the purchaser and the store clerk. The control to be executed is executed. For example, the display control unit 234 executes control to display images (object captured image and person captured image) captured as a moving image.

  In step S <b> 5, the comparison unit 231 extracts a person who is estimated to be included as a subject from an image obtained by capturing a predetermined position with respect to the product recognition area A. Further, the comparison unit 231 extracts a person who is estimated to be included as a subject from a person captured image stored separately in the image DB 301. In particular, the comparison unit 231 extracts a human face included as a subject. For example, the comparison unit 231 may identify a human eye using luminance and extract a contour portion of the human face.

  In step S <b> 6, the comparison unit 231 is included in a person extracted from the person captured image (a person who is a subject included in the person captured image) and a person captured image separately stored in the DB information holding unit 241. The person who is the subject (the person who is the subject included in the person captured image) is compared. Here, the comparison unit 231 is included in a facial feature portion (extracted facial feature portion) included in a captured image obtained by capturing a person as a subject and a captured image stored in the DB information storage unit 241. This may be done by comparing the facial feature (extracted facial feature) of the person who is the subject.

  When the comparison unit 231 determines that the imaged person is registered (the same person is stored separately) as a result of the comparison, YES is determined in step S7. Thereby, in step S7, the comparison unit 231 causes the presentation unit 238 to present the fact (warning). For example, the comparison unit 231 may display a warning on the display unit D on the seller side and the purchaser side, or warn with sound or light (lamp). On the other hand, if the comparison unit 231 determines that the imaged person is not separately stored (does not match) as a result of the comparison, it is determined NO in step S6, and the process skips step S7 and proceeds to step S8. move on.

  In step S <b> 9, the presence recognition unit 232 recognizes the presence of an object from an object captured image obtained by capturing the product recognition area A among a plurality of acquired captured images. Further, the presence recognizing unit 232 specifies an image area for each product from the object captured image obtained by capturing the product recognition area A.

  In step S10, the recognized quantity counting unit 233 counts the number of recognized objects and generates the result as recognized quantity information. Specifically, the recognized quantity counting unit 233 counts the image area for each object, thereby obtaining the quantity of the object. For example, the recognized quantity counting unit 233 discriminates an object such as a rice ball shown in FIG. 4 that is placed in an overlapping manner or an object such as a plastic bottle that does not have an upper part from a plurality of captured images. Count the number of objects. As a result, the recognized quantity counting unit 233 can accurately count the number of objects that are overlaid or objects that do not have an upper feature.

  In step S <b> 11, the product specifying unit 239 specifies which product is the object placed in the product recognition area A. For example, the product specifying unit 239 matches information obtained from an object whose existence is recognized with product information held by the DB information holding unit 241. Thereby, the product specifying unit 239 specifies which product of the product information held in the DB information holding unit 241 is the object whose presence is recognized by the presence recognition unit 232.

  In step S12, the settlement unit 235 settles the product placed in the product recognition area A. Specifically, the settlement unit 235 settles all the products placed in the product recognition area A by individually settled the specified products placed in the product recognition area A.

  In step S13, the settlement quantity counting unit 236 counts the quantity of the merchandise that has been settled, and generates the result as the settlement quantity information. The determination unit 237 determines whether the recognized quantity information and the adjusted quantity information are the same.

  As a result of the determination by the determination unit 237, if it is determined that the recognized quantity information and the adjusted quantity information are the same, YES is determined in step S14, and the process ends. On the other hand, as a result of the determination by the determination unit 237, if it is determined that the recognized quantity information and the adjusted quantity information are not the same, NO is determined in step S14, and the process proceeds to step S15.

  In step S15, the determination unit 237 causes the presentation unit 238 to present that the recognized quantity information and the settlement quantity information are different (warning). The presentation unit 238 presents that the recognized quantity information and the adjusted quantity information are different. The presentation unit 238 may display the display unit D by displaying that the recognized quantity information and the adjusted quantity information are different. The presentation unit 238 may present a warning using sound or light (lamp).

  In step S <b> 16, the determination unit 237 transmits, to the DB management unit 301, a person captured image obtained by capturing a predetermined position where a purchaser is estimated to exist for a predetermined area via the communication unit 210. The DB management unit 301 stores the transmitted captured image in the image DB 401. Thereby, the process ends.

  Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and modifications, improvements, and the like within the scope that can achieve the object of the present invention are included in the present invention. It is.

For example, the above-described image processing by the CPU may be processing by a GPU (Graphics Processing Unit).
Moreover, the external appearance structure of the cash register terminal 2 shown by FIG. 4 is an illustration, and is not limited to this external appearance. It is sufficient that at least the product recognition area A, the imaging unit 208, and the output unit 206 are included, and other components may be added. Further, the number of imaging units 208 is not limited to the above embodiment, and any number may be used as long as the present invention can be implemented.

For example, as shown in FIG. 8, a conveyance mechanism 501 (for example, a belt conveyor) that conveys one or more articles is provided, and a product recognition area A and a settlement area B are sequentially arranged along the conveyance direction of the conveyance mechanism 501. May be.
The product recognition area A is provided with an imaging unit 208 that captures a predetermined position (position where a purchaser is assumed to exist) with respect to the product recognition area A. Further, in the product recognition area A, a product recognition unit 502 capable of passing one or more articles conveyed by the conveyance mechanism 501 is provided. The product recognition unit 502 is formed by two side surfaces and an upper surface, and is arranged in an inverted U shape on the transport mechanism 501. In the product recognition unit 502, an imaging unit 208 is provided on the inner surfaces of the two side surfaces and the upper surface. Thereby, the imaging unit 208 can image one or more articles that pass through the product recognition area A from a plurality of directions.

One or more articles placed before the product recognition area A in the transport direction of the transport mechanism 501 are transported to the product recognition area A by the transport mechanism 501. One or more objects passing through the product recognition unit 502 (product recognition area A) by the transport of the transport mechanism 501 are imaged by a plurality of imaging units 208 provided in the product recognition unit 502. The number of one or more objects that have been imaged is counted by the recognition quantity counter 233. Then, the counted one or more objects are transported to the settlement area B by the transport mechanism 501. The determination unit 237 indicates that the quantity of the object recognized in the product recognition area A (recognized quantity information) is different from the quantity of the product settled in the settlement area B (settlement quantity information). May be presented. Note that the start of recognition of one or more objects is not limited to pressing the recognition button, and one or more objects or a specific object (for example, a bar) is conveyed to the product recognition area A and captured by the imaging unit 208. It may be time. The end of recognition of one or more objects may be when a predetermined time elapses after one or more objects are no longer imaged, or when a specific object (such as a bar) is imaged.
Further, the product specifying unit 239 (FIG. 5) specifies one or more recognized objects as products based on the object captured image captured by the imaging unit 208 provided in the product recognition unit 502, thereby specifying the product. May be performed. Then, the settlement of the product may be performed by the settlement unit 235 (FIG. 5) operating with the input to the display unit D provided in the settlement area B as a trigger.

  Further, in the above embodiment, the occurrence of fraud (sweet hearting) is determined based on whether or not the recognized quantity information generated by the recognized quantity counting unit 233 and the adjusted quantity information generated by the adjusted quantity counting unit 236 are different. However, it is not limited to this. For example, the determination unit 237 may determine whether the product specified by the product specifying unit 239 is different from the product settled by the settlement unit 235 (whether all products have been settled). And presentation part 238 may present a warning, when all the goods are not settled.

As a further modification, the image acquisition unit 229 may acquire the object captured image and the person captured image, and the DB management unit 301 may simply store the object captured image and the person captured image in association with each other and manage them.
This makes it possible for people to recognize (recognize) products that have been placed on top of each other or products that do not have features at the top in subsequent image confirmations, thereby enabling fraud monitoring support and fraud tracking support. Become.
2 and 3 are merely examples for achieving the object of the present invention, and are not particularly limited.

  Further, the functional block diagram shown in FIG. 5 is merely an example, and is not particularly limited. That is, it is sufficient that the information processing system has a function capable of executing the above-described series of processing as a whole, and what functional block is used to realize this function is not particularly limited to the example of FIG. .

Further, the location of the functional block is not limited to that shown in FIG.
In addition, one functional block may be constituted by hardware alone, software alone, or a combination thereof.
For example, at least a part of functions such as the comparison unit 231 of the cash register terminal 2 can be transferred to the server 1. More specifically, at least some of the functions of the comparison unit 231 and the presence recognition unit 232 may be transferred to the server 1 so that only the result of processing by the server 1 is transmitted to the cash register terminal 2. Conversely, at least a part of functions such as the DB management unit 301 of the server 1 can be transferred to the cash register terminal 2.

When the processing of each functional block is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium.
The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose smartphone or personal computer other than a server.

  A recording medium including such a program is not only constituted by a removable medium distributed separately from the apparatus main body in order to provide the program to each user, but also provided to each user in a state of being incorporated in the apparatus main body in advance. It is composed of a provided recording medium or the like.

In summary, the information processing system to which the present invention is applied only needs to have the following configuration, and can take various embodiments.
That is, the information processing system to which the present invention is applied (for example, the information processing system in FIG. 1)
An information processing system functioning as a system including a cash register terminal (for example, cash register terminal 2 in FIG. 1),
A plurality of captured images obtained as a result of imaging from one or more directions of one or more objects (for example, a product to be purchased by a purchaser) placed in a predetermined area (for example, the product recognition area A in FIG. 4) are acquired. Acquisition means (for example, the image acquisition unit 229 in FIG. 5);
Management means (for example, the DB management unit 301 in FIG. 5) for storing and managing object captured images in a predetermined database;
Is provided.
As a result, it becomes possible to provide support for recognizing stacked products, support for recognizing products without features on the top, support for monitoring fraud, and support for tracking fraud.

Further, the information processing system includes a recognition unit that recognizes the presence of one or more objects based on the acquired object captured image (for example, the presence recognition unit 232 in FIG. 5);
A recognized quantity counting means (for example, a recognized quantity counting unit 233 in FIG. 5) that counts the quantity of one or more recognized objects and generates the result as recognized quantity information;
A settlement quantity counting means (for example, a settlement quantity counting unit 236 in FIG. 5) that counts the quantity of one or more objects settled as a product and generates the result as settlement quantity information;
A determination means (for example, determination unit 237 in FIG. 5) for determining whether the recognized quantity information and the adjusted quantity information are the same;
Presenting means (for example, the presenting unit 238 in FIG. 5) for presenting a warning when the recognized quantity information and the adjusted quantity information are different;
Can further be provided.
Accordingly, a warning can be presented when the quantity of objects placed in a predetermined area (for example, the merchandise recognition area A in FIG. 4) before settlement is different from the quantity of merchandise after settlement.

Further, the information processing system includes a recognition unit that recognizes the presence of one or more objects based on the acquired object captured image;
A specifying means for specifying which product each of the one or more objects whose existence is recognized;
A settlement means (for example, the presence recognition unit 232 in FIG. 5) for settlement of the identified product;
A determination means (for example, the determination unit 237 in FIG. 5) for determining whether or not all of the identified products have been settled;
A presenting means (for example, the presenting unit 238 in FIG. 5) that presents a warning when all the products are not settled;
Can further be provided.
Thereby, a warning can be presented when the product before settlement is not included in the product after settlement.

In addition, the acquisition unit (for example, the image acquisition unit 229 in FIG. 5) acquires a captured image obtained by capturing a predetermined position where a purchaser is assumed to be present in a predetermined area as a human captured image,
The management means (for example, the DB management unit 301 in FIG. 5) can store and manage the object captured image in association with the person captured image in a predetermined database.

Further, the management unit (for example, the DB management unit 301 in FIG. 5) presents a warning from the presentation unit (for example, the presentation unit 238 in FIG. 5) as a result of the determination by the determination unit (for example, the determination unit 237 in FIG. 5). In this case, the acquired human captured image can be separately stored and managed in a predetermined database (for example, the image DB 401 in FIG. 5).
Thereby, it is possible to distinguish and manage a purchaser who seems to have performed fraud.

In addition, the information processing system includes a person who is a subject included in the captured person image acquired by the acquisition unit (for example, the image acquisition unit 229 in FIG. 5) and a management unit (for example, the DB management unit 301 in FIG. 5). It is compared whether or not a person who is a subject included in a human captured image that is separately stored and managed matches, and when the comparison result matches, a presentation means (for example, the presentation unit 238 in FIG. 5). ) Presents that fact (for example, the comparison unit 231 in FIG. 5).
Can further be provided.

Moreover, the display which performs the control which displays at least one part image sequentially toward a purchaser and a shop assistant among the several image acquired by the acquisition means and the image processed or created based on the said several image Control means (for example, the output unit 206 in FIG. 5)
Can further be provided.
Thus, fraud can be suppressed. In particular, the psychological effect of hesitating or giving up cheating by displaying the purchaser's face together is also increased.

Further, the acquisition unit (for example, the image acquisition unit 229 in FIG. 5) can acquire a captured image of one or more objects passing through a predetermined area (for example, the product recognition area A in FIG. 4) as a further object captured image. .
Thereby, since one or more objects can be imaged for a long time, the number of object captured images with different viewpoints of one or more objects can be increased, and the recognition rate of one or more objects can be improved. Further, when one or more objects pass through a predetermined area, it becomes possible to place the objects without being affected by the size of the predetermined area, thereby preventing one or more objects from being placed on top of each other. it can.

  In the present specification, the steps for describing the program recorded on the recording medium are not limited to the processing performed in time series according to the order, but may be performed in parallel or individually even if not necessarily performed in time series. The process to be executed is also included.

  Further, in the present specification, the term “system” means an overall apparatus composed of a plurality of devices, a plurality of means, and the like.

DESCRIPTION OF SYMBOLS 1 ... Server 2, 2-1, 2-2, 2-n ... Cash register terminal 3 ... Remote release terminal 101 ... CPU
102 ... ROM
103 ... RAM
104 ... Bus 105 ... Input / output interface 106 ... Output unit 107 ... Input unit 108 ... Storage unit 109 ... Communication unit 110 ... Drive 120 ... Removable media 201 ... CPU
202 ... ROM
203 ... RAM
204 ... Bus 205 ... I / O interface 206 ... Output unit 207 ... Input unit 208 ... Imaging unit 209 ... Storage unit 210 ... Communication unit 211 ... Drive 220 ... Removable media 229 ... Image acquisition unit 230 ... Recognition button determination unit 231 ... Contrast unit 232 ... Presence recognition unit 233 ... Recognized quantity counting unit 234 ... Display control unit 235 ... Settlement unit 236 ... Settlement quantity counting unit 237 ... Judgment unit 238 ... Presentation unit 239 ... Product identification unit 241 ... DB information holding unit 301 ... DB management unit 401 ... Image DB
402 ... Product DB
501 ... Conveying mechanism 502 ... Product recognition unit A ... Product recognition area B ... Settlement area D ... Display unit N ... Network

Claims (9)

An information processing system functioning as a system including a cash register terminal,
Acquisition means for acquiring a plurality of captured images obtained as a result of imaging each of one or more objects placed in a predetermined area from a plurality of directions as an object captured image;
Management means for storing and managing the object captured image in a predetermined database;
Recognition means for recognizing the presence of the one or more objects based on the acquired object captured image;
A specifying means for specifying which product each of the one or more objects whose existence is recognized based on the object captured image ;
Checkout means for checking out the identified product,
A determination means for determining whether or not all the identified products have been settled;
A presentation means for providing a warning when it is determined that all products are not settled;
An information processing system comprising: The specifying means matches any information obtained from an object whose presence is recognized with information for specifying a product, and each of the one or more objects whose presence is recognized is any product. To identify
The determination unit determines whether or not the product specified by the specifying unit is different from the product adjusted by the settlement unit, and determines whether or not all the specified products have been settled. Information processing system described in 1. The acquisition means further acquires, as a human captured image, a captured image obtained by capturing a predetermined position where a purchaser is estimated to exist for the predetermined area,
The information processing system according to claim 1, wherein the management unit stores and manages the object captured image in a predetermined database in association with the person captured image.   4. The management unit according to claim 3, wherein when the warning by the presentation unit is presented as a result of the determination by the determination unit, the acquired person captured image is separately stored in a predetermined database and managed. Information processing system. Whether or not a person who is a subject included in the person-captured image acquired by the acquisition unit matches a person who is a subject included in the person-captured image that is separately stored and managed by the management unit The information processing system according to claim 4, further comprising: a comparison unit that, when the comparison results match, causes the presentation unit to present the fact. Display for executing control to sequentially display at least a part of the image captured by the acquisition unit and the image processed or created based on the object captured image to the purchaser and the store clerk. Control means,
The information processing system according to claim 1, further comprising:   The information processing system according to claim 1, wherein the acquisition unit acquires a captured image of one or more objects passing through the predetermined area as a further object captured image. An information processing method executed by an information processing system functioning as a system including a cash register terminal,
An acquisition step of acquiring, as an object captured image, a plurality of captured images obtained as a result of imaging each of one or more objects placed in a predetermined area from a plurality of directions;
A management step of storing and managing the object captured image in a predetermined database;
A recognition step for recognizing the presence of the one or more objects based on the acquired object captured image;
A specifying step for specifying which product each of the one or more objects whose presence is recognized based on the object captured image ;
A checkout step to checkout the identified product;
A determination step for determining whether all of the identified products have been settled;
A presentation step for providing a warning when it is determined that all products have not been settled;
An information processing method including: In a computer that controls an information processing system that functions as a system including a cash register terminal,
An acquisition step of acquiring, as an object captured image, a plurality of captured images obtained as a result of imaging each of one or more objects placed in a predetermined area from a plurality of directions;
A management step of storing and managing the object captured image in a predetermined database;
A recognition step for recognizing the presence of the one or more objects based on the acquired object captured image;
A specifying step for specifying which product each of the one or more objects whose presence is recognized based on the object captured image ;
A checkout step to checkout the identified product;
A determination step for determining whether all of the identified products have been settled;
A presentation step for providing a warning when it is determined that all products have not been settled;
A program that executes control processing including

Images