JP6209716B1 - Information processing system, information processing method, and program - Google Patents

Abstract

An object of the present invention is to verify a recognition result such as object recognition by deep learning used in a next-generation cash register system and output that it is not understood (cannot be recognized) as a final result. An information processing system includes a verification unit 233 that verifies which product candidate an object is, and an output control unit 238 that executes control to output a verification result. When the number of similar feature points specifying unit 251 that specifies the number of similar feature points of each of a plurality of product candidates with respect to the feature point indicating the feature of the object and the maximum number of similar feature points does not exceed the predetermined number, the object And a verification result determination unit 257 that determines not to specify which product is the product. [Selection] Figure 5

Description

  The present invention relates to an information processing system, an information processing method, and a program.

  Traditionally, in stores such as convenience stores, supermarkets, shopping centers, and various mass retailers, buyers pick up products from the shelves, put them in shopping carts and shopping carts, and carry them to the cash register near the exit of the store. A cash register is used to settle the price of the product (for example, see Patent Document 1).

However, when checking out items in a shopping cart or basket at the cash register, barcodes will be read even at the self cash register, so the queue waiting for the cash register cannot be eliminated. The person waits for a cash register for a long time.
In addition, when a purchaser purchases a product that the purchaser wants to purchase at a store, if there are many other purchasers, the purchaser may give up shopping.
In consideration of the circumstances as described above, when a purchaser purchases a product displayed in a store, a system that can automate the settlement of the price of the product and shorten the time required for the settlement of the price of the product. It is requested.
Therefore, when a product is placed in a predetermined area such as a cash register, the applicant images the predetermined area with a camera, recognizes the object of the product candidate from the captured image obtained as a result, and the object (product candidate). A patent application has already been filed for a next-generation cash register system capable of executing a series of processing until the settlement of the specified product is specified (Japanese Patent Application No. 2006-080624).

JP 2001-76261 A

  However, in the object recognition by deep learning used in the next-generation cash register system, the recognition accuracy of the result of object recognition is low and the recognition rate is high, but if the recognition result is unmistakably wrong, or the recognition rate is low and the recognition result is lost. May be wrong. As a result, the object may be specified as one of the products while being wrong or lost.

  The present invention has been made in view of such a situation, and when a recognition result such as object recognition by deep learning used in a next-generation cash register system is verified, the authentication result is incorrect or in doubt. In this case, the fact that the user does not know (cannot be recognized) is output as a final result so that the person can correct the recognition result.

In order to achieve the above object, an information processing system according to one embodiment of the present invention includes:
Acquisition means for acquiring a captured image obtained as a result of imaging an object placed in a predetermined area;
Recognizing means for recognizing the presence of the object based on the acquired captured image;
A recognition rate calculating means for calculating a recognition rate of each of a plurality of product candidates for the object whose existence is recognized;
Verification means for verifying which product candidate the object is when the recognition rate satisfies a predetermined condition;
Output control means for executing control to output the verification result;
With
The verification means includes
Similar feature point number specifying means for specifying the number of similar feature points of each of the plurality of product candidates with respect to a feature point indicating the feature of the object;
A verification result determining means for determining not to specify which product the object is in the case where the number of the maximum similar feature points does not exceed a predetermined number;
Is provided.

The verification unit further includes a similar product determination unit that determines which product the object is from a predetermined similar product as a product similar to the product candidate having the most similar feature points,
The similar feature score specifying means determines whether there is a predetermined similar product,
The verification result determination means may determine not to specify which product is the object when there is the similar product and the object cannot be specified which product among the similar products. it can.

Further, the similar product determination means includes:
A different size product specifying means for specifying a different size product that is different in size from the similar products;
A size comparison means for comparing the size of the object and the size of the non-size product to determine which size range of the size of the object belongs to;
Further comprising
The verification result determining means can determine not to specify the product as an object when the size of the object does not belong to any size range of the different size products.

Further, the similar product determination means includes:
A different color product specifying means for specifying a different color product from the similar products,
Color comparison means for comparing the histogram of the object and the histogram of the different color product to determine which histogram of the object is similar to the histogram of the different color product;
Further comprising
The verification result determining means can determine not to specify which product is the object when the histogram of the object does not reach a predetermined similarity with respect to any histogram of different products. .

  An information processing method and program of one embodiment of the present invention are a method and program corresponding to the above-described information processing system of one embodiment of the present invention.

  According to the present invention, it is possible to verify a recognition result such as object recognition by deep learning used in a next-generation cash register system, output that it is not known (cannot be recognized) as a final result, and a person can correct the recognition result. it can.

It is a lineblock diagram of a goods recognition system as a 1st embodiment of an information processing system of the present invention. It is a block diagram which shows the hardware constitutions of the server which comprises the goods recognition system of FIG. It is a block diagram which shows the hardware constitutions of the cash register terminal which comprises the goods recognition system of FIG. It is an external appearance top view of the cash register terminal of FIG. It is a functional block diagram which shows an example of a functional structure with the server of FIG. 2, and the cash register terminal of FIG. It is a functional block diagram which shows an example of the verification part of FIG. It is a flowchart explaining the goods specific process which the server and cash register terminal of FIG. 5 perform. It is a flowchart explaining the verification process of FIG. It is a flowchart explaining the verification process of FIG. It is a table | surface which shows an example of merchandise information. It is a table | surface which shows an example which specified the number of the similar feature points of the goods candidate. It is a table | surface which shows another example which specified the number of the similar feature points of the goods candidate. It is a table | surface which shows another example which specified the number of the similar feature points of the goods candidate. It is a table | surface which shows an example of the similar goods contained in merchandise information. It is a screen figure which shows an example of the screen displayed on a display part.

First, the outline of the information processing system of the present invention will be described.
The product recognition system (next generation cash register system) to which the information processing system of the present invention is applied can solve the problem (2) of the following (1).
(1) In object recognition by deep learning, some results (product candidates) are output as a list.
That is, in the next-generation cash register system at the stage of the above-mentioned patent application (Japanese Patent Application No. 2006-080624), product candidates = product names are output.
(2) In this case, if the result of the object recognition is wrong or hesitated, there is a possibility that a result different from the substance is output.
In particular, the problem (2) becomes remarkable in the case of a series of products with different packages, products with different tastes, or products with different sizes. Also, the problem (2) becomes prominent even when the operation is different from the assumed operation such as placing the product in the cash register upside down.

In this specification, what is not specified as a product is called an “object”. In the present specification, a product in which “object” is specified as a product is referred to as a “product”. Also, in this specification, a result of object recognition that is output as a list of “product” candidates is referred to as a “product candidate”.
Further, in this specification, a database in which information for specifying a product (“product information”) is stored is referred to as “product DB”.
Further, in this specification, a plurality of points indicating the characteristics of the “object” are referred to as “feature points”. Further, a feature point of “object” similar to “feature point” included in “product information” of “product DB” is referred to as “similar feature point”. Further, in this specification, “similar” means that the local feature amount for each “feature point” included in the “product information” of the “product DB” and the local feature amount for each “feature point” of the “object”. Indicates that they are similar.
Further, in the present specification, among the “product information”, products having different sizes are referred to as “size different products”. In the present specification, among the “products”, products having different tastes and series are referred to as “colored products”.
Further, in this specification, “size different products” and “color different products” are collectively referred to as “similar products”.

An information processing system to which the present invention is applied has at least the following functions.
That is, an information processing system to which the present invention is applied includes an acquisition function, a recognition function, a recognition rate calculation function, a verification function, and an output control function. The verification function includes a similar feature number specifying function and a verification result determination function.
The acquisition function is a function of acquiring a captured image obtained as a result of imaging an object placed in a predetermined area (product recognition area).
The recognition function is a function for recognizing the presence of an object based on the acquired captured image.
The recognition rate calculation function is a function for calculating the recognition rate of each of a plurality of product candidates for an object whose existence is recognized.
The verification function is a function for verifying which product candidate the object is when the recognition rate satisfies a predetermined condition.
The output control function is a function for executing control for outputting the verification result.

The similar feature point number specifying function is a function for specifying the number of similar feature points of each of a plurality of product candidates with respect to a feature point indicating a feature of an object.
The verification result determination function is a function for determining not to specify which product an object is in the case where the number of maximum similar feature points does not exceed a predetermined number.
Here, the “predetermined area” is an area where an object is placed for settlement, and is a “product recognition area” where an image of the object is taken.

Such an information processing system to which the present invention is applied can execute the following process (A).
Process (A) refers to the following process.
That is, the information processing system to which the present invention is applied recognizes the presence of an object by exerting a recognition function. Here, recognizing the presence of an object means defining a region of the object existing in the captured image by an image processing method such as background difference.
Next, the information processing system to which the present invention is applied calculates the recognition rate of each of the plurality of product candidates for the object whose existence is recognized by using the recognition rate calculation function. Note that image recognition methods such as deep learning, specific object recognition, and general object recognition can be used to calculate the recognition rate of the product candidates.
The information processing system to which the present invention is applied exhibits a verification function, and verifies which product candidate is an object when the recognition rate satisfies a predetermined condition.

In particular, when the information processing system to which the present invention is applied exhibits the verification function, first, the similar feature number specifying function is exhibited, and each similar feature of a plurality of product candidates for the feature point indicating the feature of the object is displayed. Specify the number of points. Here, the number of similar feature points is specified by comparing (matching) the local feature amount of the feature point indicating the feature of the object with the local feature amount of the feature point indicating the feature of the product candidate, and similar features. To specify the number of points.
When the information processing system to which the present invention is applied exhibits the verification function, the verification result determination function is exhibited, and when the number of maximum similar feature points does not exceed the predetermined number, the object is This is a function that determines not to specify which product.

Finally, the information processing system to which the present invention is applied exhibits the output control execution function and outputs the verification result.
Thereby, the recognition result such as object recognition by deep learning used in the next generation cash register system is verified, and it is output as the final result that it is not understood (cannot be recognized), and the person can correct the recognition result.

In the information processing system to which the present invention is applied, the verification function may further include a similar product determination function in addition to the above-described function.
The similar product determination function is a function for determining which product is an object from similar products determined in advance as a product similar to a product candidate having the most similar feature points.
Here, the similar feature number specifying function determines whether there is a predetermined similar product. In addition, the verification result determination function determines that the product is not specified when there is a similar product and it is not possible to specify which product of the similar products is the object. Here, the “similar product” is a “product” that is determined in advance to be similar to the “product candidate” having the most similar feature points.

Such an information processing system to which the present invention is applied can execute the following process (B).
Process (B) refers to the following process.
In other words, when the information processing system to which the present invention is applied exhibits the verification function, the information processing system determines whether there is a predetermined similar product by causing the similar feature number specifying function to be exhibited.
Next, the information processing system to which the present invention is applied exhibits a similar product determination function, and the object is any product from similar products that are predetermined as products similar to the product candidate having the most similar feature points. Determine if there is.
The information processing system to which the present invention is applied exhibits the verification result determination function, and there is a similar product, and when it is not possible to identify which product among the similar products, which product is the object, Decide not to specify whether or not.
As a result, it is decided not to identify which product is the product when it is difficult to judge between the similar products determined in advance for the specified product, and finally it is not known (cannot be recognized) Output as a result. Therefore, the person can correct the recognition result.

Further, in the information processing system to which the present invention is applied, the similar product determination function can include a different size product specifying function and a size comparison function as the functions described above.
The different size product specifying function is a function for specifying different size products having different sizes from similar products. Here, “size different products” refers to “similar products” of different sizes among “similar products”.
The size comparison function is a function that compares the size of the object with the size of the different size product and determines which size range of the size of the object the object belongs to.
Here, the verification result determination function is a function for determining that the product is not specified when the size of the object does not belong to any size range of the different size products.

Such an information processing system to which the present invention is applied can execute the following process (C).
Process (C) refers to the following series of processes.
In other words, the information processing system to which the present invention is applied causes the similar feature point number specifying function to be exhibited, and then causes the different size product specifying function to be used to specify the different size product from the similar products.
Next, the information processing system to which the present invention is applied exhibits the size comparison function, compares the size of the object with the size of the different size product, and the size of the object is the size of any size difference product. It is judged whether it belongs to the threshold range.
The information processing system to which the present invention is applied exhibits the verification result determination function, and when the size of the object does not belong to any of the size ranges of the different size products, Decide not to specify if it exists.
This makes it possible to verify similar products of different sizes for the specified product, improve the accuracy of outputting the final result of not knowing (unrecognizable), and the person correcting the recognition result be able to.

Further, in the information processing system to which the present invention is applied, the similar product determination function can include a color-difference product specifying function and a color comparison function as the functions described above.
The different color product specifying function is a function for specifying a different color product with a different color from similar products. Here, “colored products” refers to “similar products” having different tastes and series among “similar products”.
The color comparison function is a function of comparing the histogram of the object and the histogram of the different color product to determine which color difference product histogram is similar to the histogram of the object.
Here, the verification result determination function is a function for determining not to specify which product is the object when the histogram of the object does not reach the predetermined similarity with respect to the histogram of any product of different colors.

Such an information processing system to which the present invention is applied can execute the following process (D).
Process (D) refers to the following series of processes.
In other words, the information processing system to which the present invention is applied causes the similar feature number specifying function to be exhibited, and then causes the different color product specifying function to specify the different color products that are different in color from the similar products.
Next, the information processing system to which the present invention is applied exhibits a color comparison function, compares the histogram of the object and the histogram of the different color product, and the histogram of the object is compared with the histogram of any different color product. Are similar to each other.
Then, the information processing system to which the present invention is applied exhibits the verification result determination function, and if the histogram of the object does not reach the predetermined similarity to the histogram of any differently colored product, Decide not to specify whether or not.
This makes it possible to verify similar products with different tastes and series for the specified product, improve the accuracy of outputting the final result of not knowing (unrecognizable), and human correcting the recognition result can do.

Next, an embodiment of an information processing system of the present invention will be described with reference to the drawings.
FIG. 1 is a configuration diagram of an embodiment of an information processing system of the present invention.
The information processing system according to the present embodiment is designed to realize a technology for verifying recognition results such as object recognition by deep learning used in the next-generation cash register system and outputting as a final result that it is not understood (cannot be recognized). 1 has the configuration shown in FIG.
That is, the information processing system as one embodiment of the present invention includes a server, n (n is an arbitrary integer) cash register terminals 2-1 to 2 -n, and a display terminal 3.
The server 1 and the cashier terminals 2-1 to 2-n are connected to each other via a network N such as the Internet line.
For convenience of explanation, only one server in FIG. 1 is drawn, but there may actually be a plurality of servers.
Hereinafter, when it is not necessary to distinguish the cashier terminals 2-1 to 2-n individually, these are collectively referred to as “the cashier terminal 2”.

The server 1 executes various processes to manage each operation of the cash register terminal 2.
The cash register terminal 2 includes a product recognition area, one or more cameras, and one or more display units. The product recognition area is an area where an object to be recognized is placed. One or more cameras image an object placed in the product recognition area. In the present embodiment, the cash register terminal 2 includes two display units, one of which is for the purchaser for presenting the purchaser with information on the identified product or that the object has not been identified as a product. Display the screen. The other of the two display units displays information on the specified product and a seller screen for presenting the seller that the object has not been specified as the product. A specific example of the external configuration of the cash register terminal 2 will be described later with reference to FIG.
The display terminal 3 is an information processing terminal operated by a store seller or the like, and includes, for example, a tablet terminal or a smartphone.

In the information processing system of this embodiment, the following operation (process) is executed.
That is, in the information processing system of the present invention, the cash register terminal 2 stores and manages the product information in the product DB 402 (FIG. 5).

When the object is placed in the product recognition area of the cash register terminal 2, the cash register terminal 2 images the placed object and acquires a captured image obtained as a result of the imaging.
The cash register terminal 2 recognizes the presence of an object placed in the product recognition area by using the predetermined image recognition method described above for the acquired captured image.
The cash register terminal 2 calculates the recognition rate of each of the plurality of product candidates for the object whose existence is recognized.
The cash register terminal 2 verifies which product candidate the object is when the recognition rate satisfies a predetermined condition.

Specifically, the cashier terminal 2 specifies the number of similar feature points of each of the plurality of product candidates with respect to the feature points indicating the feature of the object.
The cash register terminal 2 determines not to specify which product is the object when the maximum number of similar feature points does not exceed a predetermined threshold value.

Then, the cash register terminal 2 executes control for outputting the verification result. The verification result is displayed on the display unit D (FIG. 5) or the display terminal 3.
As a result, it is possible to verify the recognition result such as object recognition by deep learning used in the next generation cash register system and output that it is not understood (cannot be recognized) as a final result.

In order to further improve the verification accuracy, the cashier terminal 2 determines whether there is a predetermined similar product.
The cash register terminal 2 determines which product is the object from similar products that are determined in advance as products similar to the product candidate having the most similar feature points.
When there is a similar product and it is not possible to specify which of the similar products is the product, the cashier terminal 2 determines not to specify which product the object is.
As a result, it is decided not to identify which product is the product when it is difficult to judge between the similar products determined in advance for the specified product, and finally it is not known (cannot be recognized) Output as a result. Therefore, the person can correct the recognition result.

In order to specify which of the similar products is the product, the cashier terminal 2 specifies the different size product having a different size from the similar products.
The cashier terminal 2 compares the size of the object with the size of the different size product and determines which size range of the size of the object the object belongs to.
Then, when the size of the object does not belong to any size range of the different size products, the cashier terminal 2 determines not to specify which product the object is.
Thereby, the accuracy of outputting the fact that it is not known (cannot be recognized) as the final result is further improved, and the person can correct the recognition result.

Further, in order to specify which of the similar products is the product, the cash register terminal 2 specifies the different color product that is different in color from the similar products.
The cashier terminal 2 compares the histogram of the object and the histogram of the different color product to determine which color difference product histogram is similar to the histogram of the different color product.
The cash register terminal 2 determines not to specify which product is the object when the histogram of the object does not reach the predetermined similarity with respect to any of the different color products.
Thereby, the accuracy of outputting the fact that it is not known (cannot be recognized) as the final result is further improved, and the person can correct the recognition result.
Details of these processes executed by the server 1 will be described later with reference to the functional block diagram of FIG.

  FIG. 2 is a block diagram illustrating a hardware configuration of the server 1 configuring the information processing system of FIG.

  The server 1 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a bus 104, an input / output interface 105, an output unit 106, an input unit 107, A storage unit 108, a communication unit 109, and a drive 110.

The CPU 101 executes various processes according to a program recorded in the ROM 102 or a program loaded from the storage unit 108 to the RAM 103.
The RAM 103 also stores data necessary for the CPU 101 to execute various processes as appropriate.

  The CPU 101, ROM 102, and RAM 103 are connected to each other via a bus 104. An input / output interface 105 is also connected to the bus 104. An output unit 106, an input unit 107, a storage unit 108, a communication unit 109, and a drive 110 are connected to the input / output interface 105.

The output unit 106 includes a display, a speaker, and the like, and outputs various types of information as images and sounds.
The input unit 107 includes a keyboard, a mouse, a touch panel, and the like, and inputs various information.

The storage unit 108 includes a hard disk, a DRAM (Dynamic Random Access Memory), and the like, and stores various data.
The communication unit 109 communicates with other devices (the cashier terminals 2-1 to 2-n in the example of FIG. 1) via the network N including the Internet.

A removable medium 120 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 110. The program read from the removable medium 120 by the drive 110 is installed in the storage unit 108 as necessary.
The removable medium 120 can also store various data stored in the storage unit 108 in the same manner as the storage unit 108.

  FIG. 3 is a block diagram showing a hardware configuration of the cash register terminal 2 constituting the information processing system of FIG.

The cash register terminal 2 includes a CPU 201 to a drive 211.
The output unit 206 includes a display (display unit D), a speaker, and the like, and outputs various types of information as images and sounds.
The imaging unit 208 includes one or more cameras and the like, and images the product and the purchaser of the product.
Each of the CPU 201 through the input / output interface 205, the input unit 207, the storage unit 209, and the drive 211 has basically the same functions and configurations as the CPU 101 through the drive 110 in FIG. Therefore, these descriptions are omitted.

A removable medium 220 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 211. The program read from the removable medium 220 by the drive 211 is installed in the storage unit 209 as necessary.
The removable medium 220 can also store various data stored in the storage unit 209 in the same manner as the storage unit 209.

  FIG. 4 is a diagram showing an example of an external configuration of the cash register terminal 2 of FIG.

As shown in FIG. 4, the cash register terminal 2 includes a product recognition area A, two cameras (imaging unit 208), and one or more display units D (output unit 206). The product recognition area A is an area in which a product to be recognized is placed.
One of the two display units D displays the purchaser's screen for presenting the purchaser with information on the identified product or that the object has not been identified as a product. The other of the two display units D displays the seller screen for presenting the seller with information on the specified product or that the object has not been specified as a product.
One of the two cameras shown in FIG. 4 is installed so that the product recognition area A is looked down from above, and images the product placed in the product recognition area A from above. The other of the two cameras is installed on the upper part of the display unit D that displays information on the specified product or a purchaser screen for presenting the purchaser that the object was not specified as a product. The person is imaged. In the example of FIG. 4, only one camera that captures the product from above is described, but the angle of the camera that captures the product is not limited to the example of FIG. 4. For example, the product may be imaged from the side. Two or more cameras may be installed in order to take an image of a product from a plurality of angles.

  FIG. 5 is a functional block diagram showing an example of a functional configuration of the server 1 of FIG. 2 and the cash register terminal 2 of FIG.

In the CPU (FIG. 2) of the server 1, a DB management unit 301 functions as shown in FIG.
A product DB 402 is provided in one area of the storage unit 108 (FIG. 2).

  The DB management unit 301 manages product information stored in the product DB 402.

  In the CPU 201 (FIG. 3) of the cash register terminal 2, as shown in FIG. 5, a recognition button determination unit 230, an image acquisition unit 229, a presence recognition unit 232, a display control unit 234, a recognition rate calculation unit 239, The verification unit 233, the output control unit 238, and the settlement unit 235 function.

  The recognition button determination unit 230 determines whether the recognition button has been pressed. Specifically, the recognition button determination unit 230 determines whether or not the product recognition button on the screen displayed on the display unit D of the cash register terminal 2 has been pressed.

The image acquisition unit 229 acquires an object placed in the product recognition area A that has been imaged by the imaging unit 208 (FIG. 3).
The presence recognition unit 232 recognizes the presence of an object placed in the product recognition area A using the predetermined image recognition method described above.
The display control unit 234 executes control for sequentially displaying at least a part of the acquired captured image and an image processed or created based on the captured image to the purchaser and the store clerk. Specifically, the display control unit 234 sequentially displays at least a part of the acquired captured image and an image processed or created based on the captured image on the output unit 206 (display unit D). Execute control to

  The recognition rate calculation unit 239 calculates each recognition rate of a plurality of product candidates for an object placed in the product area A. The recognition rate calculation unit 239 may determine that the product cannot be specified when the recognition rates of all the product candidates do not exceed a predetermined condition (for example, a predetermined threshold).

The verification unit 233 verifies which product candidate is the object when the recognition rate satisfies a predetermined condition. The verification unit 233 will be described later.
The output control unit 238 executes control for outputting the verification result by the verification unit 233. For example, as illustrated in FIG. 11, the output control unit 238 executes control to output the specified product name on the display unit D. Further, for example, the output control unit 238 executes control to output that the product cannot be specified on the display unit D. The output control unit 238 may execute control for outputting the specified product name and the fact that it cannot be specified to the display terminal 3.

  The settlement unit 235 settles the product placed in the product recognition area A. For example, the settlement unit 235 can perform settlement by reading the price of the product placed in the product recognition area A from the product DB information holding unit 241, adding it, and displaying it on the display unit D. Note that the settlement unit 235 can settle the product that could not be identified as the identified product by receiving an input for identifying the product from the input unit 207 or the display terminal 3.

  In the CPU of the cash register terminal 2 (FIG. 3), as shown in FIG. 6, as the verification unit 233, a similar feature score specifying unit 251, a similar product determination unit 252, and a verification result determination unit 257 function.

  The similar feature point number specifying unit 251 specifies the number of similar feature points of each of a plurality of product candidates with respect to the feature point indicating the feature of the object. For example, the similar feature point number specifying unit 251 reads product information including feature points and local feature amounts for each product as illustrated in FIG. 10 from the DB information holding unit 241, and uses the product information regarding object feature points and product candidates. Compare feature points included. Then, as shown in FIGS. 11 to 13, the similar feature point number specifying unit 251 lists the number of similar feature points of the product candidates.

  Specifically, the similar feature point number specifying unit 251 is similar to the plurality of feature points of the product information and the local feature amounts of the product information regarding the product candidate with respect to the plurality of feature points of the object and the respective local feature amounts. To decide. Further, the similar feature point number specifying unit 251 compares the relative positional relationship (for example, coordinates and the like) of the determined feature points with the corresponding relationship with the feature points of the object by plane rotation and translation of the object. Then, the similar feature point number specifying unit 251 removes feature points that do not correspond to the positional relationship of the feature points of the object from the determined feature points, and specifies the number using the remaining feature points as similar feature points. Then, the similar feature score specifying unit 251 temporarily determines a product candidate having the largest number of similar feature points as a product candidate most similar to the product.

  Also, the similar feature score specifying unit 251 stores data of similar products (for example, product information indicated by the similar product ID 21 in FIG. 14) that are predetermined as products similar to the product candidates illustrated in FIG. Read from the information holding unit 241. The similar feature score specifying unit 251 creates a list of similar products to which product candidates having the most similar feature points belong. For example, as shown in FIG. 13 and FIG. 14, product information in which a similar product ID 21 determined in advance as a group of similar products including SP juice grapefruit (500 ml) (product A) having the most similar feature points is defined. A list to be sent to the similar product determination unit 252 can be used.

  The similar product determination unit 252 determines which product is an object from similar products predetermined as products similar to the product candidate having the most similar feature points. That is, the similar product determination unit 252 determines which product is the object among the predetermined similar products, using the list sent from the similar feature score specifying unit 251. As the similar product determination unit 252, a size different product specifying unit 253, a size comparing unit 254, a color different product specifying unit 255, and a color comparing unit 256 function.

  The different-size product specifying unit 253 specifies a different-size product having a size difference from the list of similar products sent to the similar feature score specifying unit 251. For example, the different size product specifying unit 253 includes SP juice grapefruit (500 ml), SP juice grapefruit (350 ml), SP juice orange (500 ml), and SP juice among the similar products indicated by the similar product ID 21 in FIG. An apple (500 ml) is identified as an off-size product.

  The size comparison unit 254 compares the size of the object and the size of the different size product to determine which size range of the size of the object the object belongs to. For example, the size comparison unit 254 compares the object size with any of the SP juice grapefruit (500 ml), SP juice orange (500 ml), SP juice apple (500 ml), and SP juice grapefruit (350 ml) shown in FIG. Judge whether it belongs to the size range of the product. The size comparison unit 254 determines whether or not the size of the object belongs within a predetermined error range (size threshold value) with the size set for each similar product as the center value. To do. Here, the error range (size threshold) may be determined in advance for each similar product based on a difference in size between similar products (size opening) or the like. The error range may be included in the product information and stored in the product DB 402.

  The different color product specifying unit 255 specifies a different color product that is a different color of the specified product from the similar products. For example, the color-specific product specifying unit 255 specifies similar products that have different series among the similar products indicated by the similar product ID 21 in FIG. The different color product specifying unit 255 acquires a histogram of the different color products from the product information shown in FIG.

  The color comparison unit 256 compares the histogram of the object and the histogram of the different color product, and determines which color difference product is similar to the color of the object. Specifically, the color comparison unit 256 acquires a histogram of the object from a captured image obtained by capturing the object. In addition, the color comparison unit 256 acquires a histogram of color-difference products from the color-difference product specification unit 255. Then, the color comparison unit 256 determines whether the histogram of the object has reached a predetermined similarity with respect to which of the different color products (is a similarity equal to or higher than a threshold). Note that the color comparison unit 256 determines whether the histogram of the object has reached a predetermined similarity threshold for the histogram set for each similar product. Here, the threshold value of similarity may be determined in advance for each similar product based on a difference in histograms between similar products (open difference in color) or the like. The similarity threshold value may be included in the product information and stored in the product DB 402.

  The verification result determination unit 257 determines that the maximum number of similar feature points among the number of similar feature points specified by the similar feature point number specifying unit 251 does not exceed a predetermined number (the threshold value of the number of similar feature points). (For example, refer to FIG. 12), it is determined not to specify which product the object is. Further, the verification result determination unit 257 does not specify which product is the object when there is a similar product in the similar product determination unit 252 and it is not possible to specify which product the similar product is. Determined as the verification result. On the other hand, the verification result determination unit 257 determines, as a verification result, that the object is a product specified by the recognition rate calculation unit 239 when there are no similar products (for example, different size products and different color products).

  Furthermore, when there is a size difference product, the verification result determination unit 257 determines that the size comparison unit 254 determines that the size of the object belongs to the size range of one size difference product. It is determined as a verification result that the product is different. On the other hand, when the size comparison unit 254 determines that the size of the object does not belong to any size range of the different size products, the verification result determination unit 257 does not specify which product the object is. Are determined as verification results.

  Furthermore, when there is a different color product, the verification result determination unit 257 determines that the object histogram belongs when the color comparison unit 256 determines that the histogram of the object has reached a predetermined similarity with respect to the histogram of one color difference product. It is determined as a verification result that the product is of different colors. On the other hand, if it is determined by the color comparison unit 256 that the histogram of the object has not reached the predetermined similarity with respect to the histogram of any different color product, the verification result determination unit 257 determines which product is the object. The verification result is determined not to be specified. The verification result determination unit 257 specifies an object as a product based on the determined result.

  Further, the DB information holding unit 241 holds product information. The product information is transmitted from the DB management unit 301 of the server 1 to the DB information holding unit 241. As described above, the product information held by the DB information holding unit 241 is used for calculation of the recognition rate by the recognition rate calculation unit 239 and verification by the verification unit 233.

Next, in the information processing system of the present invention, a series of processes (hereinafter referred to as “product specifying process”) from when a product is placed in the product recognition area A until the product is settled will be described.
FIG. 7 is a flowchart for explaining the product specifying process executed by the server 1 and the cash register terminal 2 of FIG.

  In step S1, it is determined whether or not the recognition button has been pressed after the object is placed in the product recognition area A by the purchaser. The recognition button determination unit 230 determines whether or not the purchase button has been pressed by the purchaser.

If it is determined in step S1 that the recognition button has been pressed, YES is determined in step S1, and the process proceeds to step S2. On the other hand, if it is not possible to confirm that the recognition button is pressed, NO is determined in step S1, and the process waits until the recognition button is pressed.
In step S <b> 2, the imaging unit 208 images the product recognition area A of the cash register terminal 2.
In step S <b> 3, the image acquisition unit 229 acquires a captured image of the product recognition area A of the cash register terminal 2. The image acquisition unit 229 sends the captured image to the display control unit 234. Based on the captured image sent from the image acquisition unit 229, the display control unit 234 executes control to display at least a part of the processed or created image sequentially to the purchaser and the store clerk. . For example, the display control unit 234 performs control to display an image captured as a moving image.

  In step S <b> 4, the presence recognition unit 232 recognizes the presence of an object from the captured image obtained by capturing the product recognition area A. Further, the presence recognizing unit 232 specifies an image area for each object from the object captured image obtained by capturing the product recognition area A.

  In step S <b> 5, the recognition rate calculation unit 239 calculates the recognition rate of each of the plurality of product candidates for the object placed in the product recognition area A.

  As a result of the recognition rate calculation unit 239 calculating the recognition rate of the product candidate, if the recognition rate satisfies the condition, YES is determined in step S6, and the process proceeds to step S7. On the other hand, if the recognition rate does not satisfy the condition, NO is determined in step S6, and the process proceeds to step S8. Here, the condition of the recognition rate means that the recognition rate of the highest product candidate is a predetermined value or more.

In step S7, the verification unit 233 verifies whether or not the object is a specified product.
As a result of verification by the verification unit 233, if it is not possible to specify which product the object is, it is determined YES in step S8, and the process proceeds to step S9. On the other hand, when the object is specified, the process proceeds to step S10. If the recognition rate does not satisfy the condition in step S6, the process proceeds to step S9.

  In step S <b> 9, the output control unit 238 outputs to the display unit D that it cannot be specified which product the object is. For example, as illustrated in FIG. 15, the output control unit 238 displays an unidentifiable object on the display unit D as “unidentifiable”. On the other hand, the output control unit 238 displays the product name and the like of the identified object on the display unit D. For example, as illustrated in FIG. 15, the output control unit 238 displays the identified object on the display unit D as a product “SP juice grapefruit (500 ml)”.

  In step S10, the output control unit 238 causes the display unit D to display the product name and the like of the specified product.

  In step S11, the settlement unit 235 performs settlement for the identified product. It should be noted that the settlement unit 235 can settle the product identified by accepting the input identifying the product from the input unit 207 or the display terminal 3 for the product that cannot be identified.

Next, a series of processing (hereinafter referred to as “verification processing”) for verifying whether or not an object is a specified product in the information processing system of the present invention will be described.
8 and 9 are flowcharts for explaining the verification processing executed by the server 1 and the cash register terminal 2 of FIG.

In step S <b> 61, the similar feature point number specifying unit 251 specifies the number of similar feature points of each of a plurality of product candidates with respect to the feature points indicating the features of the object.
If it is determined that the maximum number of similar feature points exceeds a predetermined number as a result of the specification by the similar feature point specifying unit 251, YES is determined in step S62, and the process proceeds to step S63. At this time, the similar feature score specifying unit 251 temporarily determines the product candidate having the most similar feature points as the product candidate most similar to the product. On the other hand, if it is determined that the maximum number of similar feature points is equal to or less than the predetermined number, NO is determined in step S62, and the process proceeds to step S73.

  In proceeding to step S63, the similar feature score specifying unit 251 determines whether or not the temporarily determined product candidate includes a similar product that has been determined in advance. As a result of the determination by the similar feature score specifying unit 251, if it is determined that there is a similar product, YES is determined in step S63, and the process proceeds to step S64. On the other hand, as a result of the determination by the similar feature score specifying unit 251, when it is determined that there is no similar product, NO is determined in step S 63, and the process proceeds to step S 72.

  In step S <b> 64, the different size product specifying unit 253 acquires a list of predetermined similar products that are similar to the provisionally determined product candidates from the similar feature point specifying unit 251. In addition, the different size product specifying unit 253 specifies a different size product that is different in size from similar products. For example, if there is a similar product and product information of a different size is determined for the provisionally determined product candidate, the size-different product specifying unit 253 includes the product information of the provisionally determined product candidate. The product candidate having the product information is identified as a different size product. In addition, if there is no product information of a different size in the product information of the temporarily determined product candidate, the different size product specifying unit 253 may specify that there is no different size product candidate.

  As a result of the specification by the wrong size product specifying unit 253, if the wrong size product can be specified, YES is determined in step S65, and the process proceeds to step S66. On the other hand, if the wrong size product cannot be specified, NO is determined in step S65, and the process proceeds to step S68.

  In step S66, the size comparison unit 254 compares the size of the object with the size of the different size product, and determines which size range of the different size product the object belongs to. The size comparison unit 254 identifies one different size product in which the size of the object belongs to the size range.

  If the size comparison unit 254 determines that the size of the object belongs to the threshold value range of the size of one non-size product, YES is determined in step S67, and the process proceeds to step S68. On the other hand, as a result of the determination by the size comparison unit 254, if the size of the object does not belong to any of the size range of the size of the different size product, NO is determined in step S67, and the process proceeds to step S73. .

  In step S <b> 68, the different color product specifying unit 255 specifies the candidate for the different color product that is the color difference of the specified product from the similar products. For example, if there is a similar product and product information of a different color is defined for the product information of the temporarily determined product candidate, the color-specific product specifying unit 255 includes the product information of the temporarily determined product candidate, The product candidate having the specified product information is identified as a different color product. The different-color product specifying unit 255 may specify that there is no different-color product if there is no product information of a different color in the product information of the temporarily determined product candidate.

  As a result of the specification by the color difference product specifying unit 255, if the color difference product can be specified, YES is determined in step S69, and the process proceeds to step S70. On the other hand, when a different color product cannot be specified, NO is determined in step S69, and the process proceeds to step S72.

  In step S <b> 70, the color comparison unit 256 compares the histogram of the object and the histogram of the different color product to determine which color difference product is similar to the color of the object. The color comparison unit 256 identifies one different color product with similar object histograms.

  If the result of determination by the color comparison unit 256 is that the histogram of the object has reached a predetermined similarity to the histogram of one differently colored product, YES is determined in step S71, and the process proceeds to step S72. On the other hand, if the result of determination by the color comparison unit 256 is that the histogram of the object does not reach the predetermined similarity with respect to any of the different color product histograms, NO is determined in step S71, and the process proceeds to step S73.

  In step S <b> 72, the verification result determination unit 257 determines that the object is the product candidate having the largest number of similar feature points when there is no different size product and when there is no different color product. In addition, the verification result determination unit 257 determines that the object is a different size product included in the size within the range when there is a different size product and no different color product. In addition, the verification result determination unit 257 is a different-size product that includes an object in a size range when there is a different-size product and a different-color product, and is a different-product that achieves a predetermined similarity in the histogram. Determine that there is. In addition, the verification result determination unit 257 determines that the object is a different color product that achieves the predetermined similarity of the histogram when there is no different size product and there is a different color product. The verification result determination unit 257 specifies which product is the object based on the determined result. As a result, the verification process ends.

  In step S <b> 73, the verification result determination unit 257 does not specify which product the object is when the maximum number of similar feature points specified by the similar feature point number specifying unit 251 does not exceed the predetermined number. To decide. In addition, the verification result determination unit 257 determines not to specify which product the object is in the case where the size of the object does not belong to the size range of the different size product determined by the size comparison unit 254. To do. Also, the verification result determination unit 257 determines not to specify which product the object is in the case where the histogram of the object has not reached the predetermined similarity of the histogram of the different color product determined by the color comparison unit 256. To do. The verification result determination unit 257 specifies which product is the object based on the determined result. As a result, the verification process ends.

  Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and modifications, improvements, and the like within the scope that can achieve the object of the present invention are included in the present invention. It is.

For example, the above-described image processing by the CPU may be processing by a GPU (Graphics Processing Unit).
Moreover, the external appearance structure of the cash register terminal 2 shown by FIG. 4 is an illustration, and is not limited to this external appearance. It is sufficient that at least the product recognition area A, the imaging unit 208, and the output unit 206 are included, and other components may be added. Further, the number of imaging units 208 is not limited to the above embodiment, and any number may be used as long as the present invention can be implemented.

  In the above embodiment, the determination by the color comparison unit 256 is performed after the determination by the size comparison unit 254. However, the present invention is not limited to this. That is, the determination of the color comparison unit 256 may be performed first, and then the determination of the size comparison unit 254 may be performed. In addition, the similar product determination unit 252 may include only one of the size comparison unit 254 and the color comparison unit 256.

2 and 3 are merely examples for achieving the object of the present invention, and are not particularly limited.
Further, the functional block diagram shown in FIG. 5 is merely an example, and is not particularly limited. That is, it is sufficient that the information processing system has a function capable of executing the above-described series of processing as a whole, and what functional block is used to realize this function is not particularly limited to the example of FIG. .

Further, the location of the functional block is not limited to that shown in FIG.
In addition, one functional block may be constituted by hardware alone, software alone, or a combination thereof.
For example, at least a part of functions such as the comparison unit 231 of the cash register terminal 2 can be transferred to the server 1. More specifically, the functions of the recognition rate calculation unit 239 and the verification unit 233 can be transferred to the server 1 so that only the result of processing by the server 1 is transmitted to the cash register terminal 2. Conversely, at least a part of functions such as the DB management unit 301 of the server 1 can be transferred to the cash register terminal 2.

When the processing of each functional block is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium.
The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose smartphone or personal computer other than a server.

  A recording medium including such a program is not only constituted by a removable medium distributed separately from the apparatus main body in order to provide the program to each user, but also provided to each user in a state of being incorporated in the apparatus main body in advance. It is composed of a provided recording medium or the like.

In summary, the information processing system to which the present invention is applied only needs to have the following configuration, and can take various embodiments.
That is, the information processing system to which the present invention is applied (for example, the information processing system in FIG. 1)
Acquisition means (for example, the image acquisition unit 229 of FIG. 5) for acquiring a captured image obtained as a result of imaging an object placed in a predetermined area (for example, the product recognition area A of FIG. 4);
Recognizing means for recognizing the presence of an object based on the acquired captured image (for example, the presence recognizing unit 232 in FIG. 5);
A recognition rate calculation means for calculating a recognition rate of each of a plurality of product candidates for an object whose existence is recognized, (for example, a recognition rate calculation unit 239 in FIG. 5);
A verification unit (for example, the verification unit 233 in FIG. 5) that verifies which product candidate the object is when the recognition rate satisfies a predetermined condition;
Output control means (for example, the output control unit 238 in FIG. 5) for executing control for outputting the verification result;
With
The verification means (for example, the verification unit 233 in FIG. 5)
A similar feature point number specifying unit (for example, a similar feature point number specifying unit 251 in FIG. 6) for specifying the number of similar feature points of each of a plurality of product candidates with respect to a feature point indicating a feature of an object;
A verification result determination means (for example, a verification result determination unit 257 in FIG. 6) for determining not to specify which product an object is in a case where the maximum number of similar feature points does not exceed a predetermined number;
Is provided.
Thereby, the recognition result such as object recognition by deep learning used in the next generation cash register system is verified, and it is output as the final result that it is not understood (cannot be recognized), and the person can correct the recognition result.

In addition, the verification unit (for example, the verification unit 233 in FIG. 6) determines a similar product that determines which product is an object from similar products that are predetermined as products similar to the product candidate having the most similar feature points. A determination unit (for example, a similar product determination unit 252 in FIG. 5);
The similar feature score specifying means (for example, the similar feature score specifying unit 251 in FIG. 6) determines whether there is a predetermined similar product,
The verification result determination means (for example, the verification result determination unit 257 in FIG. 6) specifies which product is the object when there is a similar product and it is not possible to specify which product is the similar product. You may decide not to.
As a result, when there are a plurality of product candidates having similar feature points, the fact that they do not know (cannot be recognized) is output as the final result, so that the person can correct the recognition result.

Moreover, the similar product determination means (for example, the similar product determination unit 252 in FIG. 6)
A different size product specifying means (for example, a wrong size product specifying unit 253 in FIG. 6) for specifying a different size product having a different size from similar products;
Size comparison means (for example, the size comparison unit 254 in FIG. 6) that compares the size of the object with the size of the different size product and determines which size of the different size product the size of the object belongs to ,
Further comprising
The verification result determination means (for example, the verification result determination unit 257 in FIG. 6) does not specify which product is the object when the size of the object does not belong to any size range of the different size product. Can be determined.
As a result, it is possible to verify similar products having different sizes with respect to the specified product, output that it is not known (cannot be recognized) as a final result, and a person can correct the recognition result.

Moreover, the similar product determination means (for example, the similar product determination unit 252 in FIG. 6)
A different color product specifying means (for example, a different color product specifying unit 255 in FIG. 6) for specifying different color products from different products,
Color comparison means for comparing the histogram of the object and the histogram of the different color product to determine which of the different color products is similar to the histogram of the color difference product (for example, the color comparison unit 256 in FIG. 6). When,
Further comprising
The verification result determination means (for example, the verification result determination unit 257 in FIG. 6) uses any product as the object when the histogram of the object does not reach the predetermined similarity with respect to the histogram of any color difference product. You may decide not to specify.
As a result, similar products with different colors can be verified for the specified product, and it is output as a final result that it is not known (cannot be recognized), and the person can correct the recognition result.

  In the present specification, the steps for describing the program recorded on the recording medium are not limited to the processing performed in time series according to the order, but may be performed in parallel or individually even if not necessarily performed in time series. The process to be executed is also included.

  Further, in the present specification, the term “system” means an overall apparatus composed of a plurality of devices, a plurality of means, and the like.

DESCRIPTION OF SYMBOLS 1 ... Server 2,2-1, 2-n ... Cash register terminal 3 ... Display terminal 101 ... CPU
102 ... ROM
103 ... RAM
104 ... Bus 105 ... Input / output interface 106 ... Output unit 107 ... Input unit 108 ... Storage unit 109 ... Communication unit 110 ... Drive 120 ... Removable media 201 ... CPU
202 ... ROM
203 ... RAM
204 ... Bus 205 ... I / O interface 206 ... Output unit 207 ... Input unit 208 ... Imaging unit 209 ... Storage unit 210 ... Communication unit 211 ... Drive 220 ... Removable media 229 ... Image acquisition unit 230 ... Recognition button determination unit 232 ... Presence recognition unit 233 ... Verification unit 234 ... Display control unit 235 ... Adjustment unit 238 ... Output control Unit 239 ... Recognition rate calculation unit 241 ... DB information holding unit 251 ... Similar feature point identification unit 252 ... Similar product determination unit 253 ... Size difference product identification unit 254 ... Size comparison unit 255 ... Color difference product specifying unit 256 ... Color comparison unit 257 ... Verification result determination unit 301 ... DB management unit 402 ... Product DB
A ... Product recognition area D ... Display unit N ... Network

Claims (6)

Acquisition means for acquiring a captured image obtained as a result of imaging an object placed in a predetermined area;
Recognizing means for recognizing the presence of the object based on the acquired captured image;
A recognition rate calculating means for calculating a recognition rate of each of a plurality of product candidates for the object whose existence is recognized;
Verification means for verifying which product candidate the object is when the recognition rate satisfies a predetermined condition;
Output control means for executing control to output the verification result;
With
The verification means includes
Among the plurality of feature points indicating the feature of the object, a feature point similar to the feature point of the product candidate is used as a similar feature point, and the number of similar feature points is specified for each of the plurality of product candidates. Means,
A verification result determining means for determining not to specify which product the object is in the case where the number of the maximum similar feature points does not exceed a predetermined number;
An information processing system comprising: The verification unit further includes a similar product determination unit that determines which product the object is from a similar product predetermined as a product similar to the product candidate having the most similar feature points,
The similar feature score specifying means determines whether there is a predetermined similar product,
The verification result determination unit determines that the product is not specified when the similar product is present and the product cannot identify which product among the similar products. 1. The information processing system according to 1. The similar product determination means includes:
A different size product specifying means for specifying a different size product that is different in size from the similar products;
A size comparison means for comparing the size of the object and the size of the non-size product to determine which size range of the size of the object belongs to;
Further comprising
3. The verification result determination unit according to claim 2, wherein when the size of the object does not belong to any size range of the different size products, the verification result determination unit determines that the product is not specified as the product. Information processing system. The similar product determination means includes:
A different color product specifying means for specifying a different color product from the similar products,
Color comparison means for comparing the histogram of the object and the histogram of the different color product to determine which histogram of the object is similar to the histogram of the different color product;
Further comprising
The verification result determination means determines that the product is not specified as a product when the histogram of the object does not reach a predetermined similarity with respect to a histogram of any product of different colors. Or the information processing system of 3. An information processing method executed by an information processing system,
An acquisition step of acquiring a captured image obtained as a result of imaging an object placed in a predetermined area;
A recognition step for recognizing the presence of the object based on the acquired captured image;
A recognition rate calculating step of calculating a recognition rate of each of a plurality of product candidates for the object whose presence is recognized;
A verification step of verifying which product candidate the object is when the recognition rate satisfies a predetermined condition;
An output control step for executing a control for outputting the verification result; and
Including
In the verification step,
Among the plurality of feature points indicating the feature of the object, a feature point similar to the feature point of the product candidate is used as a similar feature point, and the number of similar feature points is specified for each of the plurality of product candidates. Steps,
A verification result determining step for determining not to specify which product is the object when the maximum number of the similar feature points does not exceed a predetermined number;
An information processing method including: To the computer that controls the information processing system,
An acquisition step of acquiring a captured image obtained as a result of imaging an object placed in a predetermined area;
A recognition step for recognizing the presence of the object based on the acquired captured image;
A recognition rate calculating step of calculating a recognition rate of each of a plurality of product candidates for the object whose presence is recognized;
A verification step of verifying which product candidate the object is when the recognition rate satisfies a predetermined condition;
An output control step for executing a control for outputting the verification result; and
Including
In the verification step,
Among the plurality of feature points indicating the feature of the object, a feature point similar to the feature point of the product candidate is used as a similar feature point, and the number of similar feature points is specified for each of the plurality of product candidates. Steps,
A verification result determining step for determining not to specify which product is the object when the maximum number of the similar feature points does not exceed a predetermined number;
A program that executes control processing including

Images