JP2018097883A - Information display apparatus and guidance display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a commodity identification apparatus and a commodity recognition navigation method for registering a commodity without interfering with the flow of commodity registration by an operator.SOLUTION: A commodity identification apparatus 2 includes a camera that captures an image of an object held by an operator, and a CPU that recognizes the object from the image captured by the camera, and if it is similar to a plurality of commodities when comparing the recognized object with reference images of the respective commodities in a plurality of directions, compares the reference images of the plurality of commodities in the respective directions to specify a reference image whose difference in characteristics is significant, and guides the operator to rotate the object such that a direction related to the specified reference image is directed to the camera.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a product identification device and a product recognition navigation method for identifying a product to be registered for sales.

Conventionally, an electronic cash register (product registration device) reads a data code such as a barcode attached to a product by a barcode reading unit and identifies the product. The bar code assigned to the product includes maker code information, item code information, and check digit information. After determining that the barcode is correctly read based on the check digit information, the product registration apparatus identifies the product based on the item code information. By printing the barcode on the product packaging in advance, the burden on the cash register operator can be reduced.
However, it is extremely troublesome to individually attach barcodes to unpackaged products such as fruits and vegetables, which takes time and cost. Conventionally, for fruit and vegetable products, operations such as pressing a button indicating a product, touching a screen, or selecting a corresponding product code from a separately prepared bar code table and reading it with a scanner have been performed.

  In recent years, product registration apparatuses for recognizing and identifying fruits and vegetables and the like, and registering the identified products for sales are becoming popular. This merchandise registration apparatus extracts the feature quantity of the appearance from image information obtained by photographing fruit and vegetable products and compares it with a feature quantity database relating to the appearance of the merchandise, thereby identifying which merchandise the article is. Thereby, even if the product is not packaged in advance, the product registration device can recognize the product without sticking the barcode individually, so the burden on the store side can be reduced.

  Each product has a difference in appearance. When the difference between the target product and the appearance of the standard product is large, it is difficult to uniquely determine the target product due to the similarity during object recognition. In such a case, the product registration device having the object recognition function can display a confirmation screen of provisionally confirmed products that are candidates for the target product, or select a plurality of products that are candidates for the target product on the screen. It was displayed. The operator needs to perform an operation such as touching a product to be packaged from this option.

  For example, in paragraph 0068 of Patent Document 1, “In the product candidate presentation area 83, product candidate illustration images G1, G2, G3 and product names are displayed in descending order of registered products with high similarity. The images G1, G2, and G3 are configured to be selectable in accordance with a selection operation on the touch panel 105. A selection button 84 for selecting a product from the product list is provided below the product candidate presentation area 83. The product selected from the product list is processed as the above-mentioned provisionally confirmed product. "

JP2013-182323A (FIG. 8 and paragraph 0068)

  When purchasing a product in a supermarket, the customer puts the product into a shopping cart or the like, and then passes the shopping cart to an operator (clerk) of the cash register to request payment of the product price. At this time, the operator reads each item in the shopping cart (hereinafter referred to as the “first shopping basket”) brought by the customer in order in the product registration device, thereby selling the product information of the product. Register each product in the first shopping basket by registering the information temporarily and repeating the operation of putting the product into another shopping basket (hereinafter referred to as “second shopping basket”). .

At that time, the product registration apparatus may not be able to recognize the product immediately. For example, if the barcode portion of the product with the barcode cannot be photographed well, the product registration device cannot immediately recognize the product. In this case, the operator causes the merchandise registration apparatus to recognize the merchandise by an operation of bringing the merchandise as close as possible to the camera of the merchandise registration apparatus.
If the target product for object recognition cannot be photographed at an appropriate angle, the product registration device cannot immediately recognize the product. In this case, the operator may operate the product as close as possible to the camera of the product registration device with the same feeling as the barcode reading operation.

  However, the cause when the product registration apparatus cannot recognize the product as an object is often the direction of the product presented to the camera is not appropriate. For example, even if the direction of the product including many feature quantities that can identify the product is the front direction, the operator points the top surface of the product toward the camera, or conversely, The main factor is that the operator points the front of the product toward the camera even though the direction is the top direction. In such a case, the product registration device presents a plurality of registered product candidates.

  In the invention described in Patent Document 1, when there are a plurality of registered merchandise candidates, one registered merchandise candidate selected by the touch panel operation is set as a provisionally confirmed merchandise. However, this selection of candidate products hinders the flow of operator product registration. Therefore, it takes time and labor for the operator, and further causes waiting time for the customer.

  Therefore, an object of the present invention is to provide a product identification device and a product recognition navigation method for identifying a product without obstructing the flow of product registration by an operator.

In order to achieve the above object, the first invention provides
Photographing means for photographing an image of the object that the operator has deceived;
Object detecting means for detecting the object from an image taken by the photographing means;
Similarity determination means for determining similarity by comparing the object detected by the object detection means with a reference image in a plurality of directions of each product;
If the object is similar to a plurality of products, a reference image in each direction of the plurality of products is compared to identify a significant difference, and a direction related to the specified reference image is directed to the photographing unit. And guiding means for guiding the rotation of the object to the operator;
It is a goods identification device characterized by including.

In order to achieve the above object, the second invention
A shooting step for shooting an image of an object that the operator has deceived;
An object detection step of detecting the object from the image captured in the imaging step;
A similarity determination step of determining the similarity by comparing the detected object with a reference image in a plurality of directions of each product;
If the object is similar to a plurality of products, the reference images in each direction of the plurality of products are compared to identify the difference, and the direction of the specified reference image is photographed. A guidance step for guiding the rotation of the object to an operator;
The product recognition navigation method of the product identification apparatus characterized by including.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to provide the goods identification apparatus and goods recognition navigation method which identify goods, without disturbing the flow of an operator's goods registration.

It is a perspective view which shows the external appearance of the goods identification apparatus etc. in 1st Embodiment. It is a lineblock diagram showing an outline of a goods discernment device etc. in a 1st embodiment. It is a functional block diagram which shows the outline of the goods identification apparatus etc. in 1st Embodiment. It is a front view of the goods identification device in a 1st embodiment. It is a figure which shows the structure of a feature-value file. It is a figure which shows the reference | standard image of 6 directions of the stuffed rabbit. It is a figure which shows the reference | standard image of 6 directions of the stuffed bear. It is a figure which shows the structure of the similar goods database in 1st Embodiment. It is a flowchart which shows the provisional registration process of the goods in 1st Embodiment. It is a screen which shows the example of operation of 1st Embodiment. It is a figure which shows the structure of the similar goods database in 2nd Embodiment. It is a flowchart which shows the provisional registration process of the goods in 2nd Embodiment. It is a screen which shows the example of operation of 2nd Embodiment.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.
(First embodiment)
FIG. 1 is a perspective view showing the external appearance of a product identification device 2 and the like according to the first embodiment.
As shown in FIG. 1, the product registration device 1 includes a product identification device 2 that registers each product, and a POS terminal 3 that performs sales registration and settlement of products related to each transaction. The product identification device 2 is a POS connection scanner that reads and registers information about products.
The merchandise item identification device 2 is installed at the center in the longitudinal direction of the counter table 5 having a horizontally long table shape. The commodity identification device 2 includes a thin rectangular parallelepiped housing 51. A camera 27 (see FIG. 2) is arranged on the front surface of the housing 51 through a reading window 52.

  A display / operation unit 22 is attached to the upper portion of the housing 51. The display / operation unit 22 is provided with a display 221. A touch panel 222 (see FIG. 2) is laminated on the surface of the display 221. A keyboard 23 is arranged on the right side of the display 221. A card reading groove of a card reader (not shown) is provided on the right side of the keyboard 23. On the left back side of the display / operation unit 22, a customer display 24 for providing information to the customer is installed face down.

The POS terminal 3 is placed on the upper surface of the drawer 37 on the checkout table 6. The drawer 37 is controlled by the POS terminal 3 for the opening operation.
On the upper surface of the POS terminal 3, a keyboard 33 for operation by an operator (clerk) is arranged. A display 321 for displaying information is provided on the upper back side of the keyboard 33 as viewed from the operator. A touch panel 322 (see FIG. 2) is stacked on the surface of the display 321. Further on the back side of the display 321, a customer display 34 for displaying information is installed so as to be rotatable in the left-right direction. The customer display 34 shown in FIG. 1 faces the front side of the figure. This customer display 34 displays information toward the customer by rotating it toward the far side of the figure.

A horizontally long table-like counter table 5 is arranged so as to form an L shape with the checkout table 6 on which the POS terminal 3 is placed. A load receiving surface is formed on the upper surface of the counter table 5. The customer undergoes a product registration process while moving from the left side to the right side in FIG. That is, the first shopping basket 4L and the second shopping basket 4R for storing products are placed on the load receiving surface of the counter table 5. Hereinafter, when there is no particular distinction between the first shopping basket 4L and the second shopping basket 4R, the shopping basket 4 may be simply described. These shopping baskets 4 are not limited to the so-called basket shape, and may be trays or the like. The shopping basket 4 is not limited to a so-called basket shape, and may be a box shape or a bag shape.
The first shopping basket 4L is brought in by a customer and stores products related to one transaction. The second shopping basket 4R is placed at a position sandwiching the merchandise item identification device 2 from the first shopping basket 4L. The product in the first shopping basket 4L is taken out by an operator who operates the product identification device 2 and moved to the second shopping basket 4R. In this moving process, the product is picked up by the reading window 52 of the product identification device 2. At this point, the camera 27 (see FIG. 2) arranged in the reading window 52 images the product.

In the product identification device 2, the product included in the image captured by the camera 27 is identified on the display 221 as to which product recorded in a feature value file 361 (see FIG. 2) described later corresponds. The product identification device 2 further notifies the POS terminal 3 of the product ID of the identified product. Based on the product ID notified from the product identification device 2, the POS terminal 3 stores information related to sales registration such as the product classification, product name, and unit price of the product corresponding to the product ID, as a sales master file (not shown), and the like. And record sales.
The present invention may be applied to a self-registration in which a customer himself operates a cash register as an operator, and is not limited.

FIG. 2 is a configuration diagram showing an outline of the product identification device 2 and the like in the first embodiment.
The product registration apparatus 1 includes a product identification device 2 and a POS terminal 3.
The product identification device 2 includes a microcomputer 21, a display / operation unit 22, an interface 25, a camera 27, a speaker 28, and a power source 29.
The microcomputer 21 is a control unit configured by connecting a ROM (Read Only Memory) 212 and a RAM (Random Access Memory) 213 to a CPU (Central Processing Unit) 211 via a bus. The ROM 212 stores a program executed by the CPU 211.
The CPU 211 is connected to the display / operation unit 22, the interface 25, the camera 27, and the speaker 28 via an internal bus and input / output circuits (not shown).

The display / operation unit 22 includes a display 221 (see FIG. 1), a touch panel 222, a customer display 24, and a keyboard 23, and the operation is controlled by the CPU 211.
The display 221 is a display unit that displays information for the operator according to an instruction from the CPU 211. The touch panel 222 receives an operation input for information displayed on the display 221. The customer display 24 displays information for the customer in accordance with an instruction from the CPU 211.
The keyboard 23 is composed of a plurality of operation keys and accepts operator input.
The interface 25 is connected to the interface 35 of the POS terminal 3 to enable data transmission / reception with the POS terminal 3.

  The camera 27 is a color CCD image sensor, a color CMOS image sensor, or the like, and is an imaging unit that performs imaging from the reading window 52 (see FIG. 1) under the control of the CPU 211. The camera 27 captures a moving image of 30 [fps], for example. Frame images (captured images) sequentially captured by the camera 27 at a predetermined frame rate are stored in the RAM 213.

The speaker 28 generates a preset warning sound or the like. The speaker 28 performs warning sound or voice notification under the control of the CPU 211.
The power source 29 supplies power to each part of the product identification device 2.
The interface 25 is connected to the interface 35 of the POS terminal 3 to enable data transmission / reception with the POS terminal 3.

The POS terminal 3 includes a microcomputer 31, a display 321, a touch panel 322, a keyboard 33, a customer display 34, an interface 35, an HDD 36, a drawer 37, a printer 38, and a power supply 39. Is done.
The microcomputer 31 executes information processing. The microcomputer 31 is configured by connecting a ROM 312 and a RAM 313 to a CPU 311 that executes various arithmetic processes and controls each unit. A drawer 37, a keyboard 33, a display 321, a touch panel 322, a customer display 34, and an HDD 36 (Hard Disk Drive) are connected to the CPU 311 via an internal bus and input / output circuits. These are controlled by the CPU 311.

The display 321 displays information for the operator according to an instruction from the CPU 311. The touch panel 322 receives an operation input for information displayed on the display 321. The customer display 34 displays information for the customer in accordance with an instruction from the CPU 311.
The keyboard 33 includes a temporary fastening key 331, a fastening key 332, and a numeric keypad 333, and accepts an operator's operation input. The numeric keypad 333 includes numeric keys from 0 to 9 and various operator keys.

  The HDD 36 is a storage unit that stores programs and various files. All or a part of the programs and various files stored in the HDD 36 are copied to the RAM 313 and executed by the CPU 311 when the POS terminal 3 is activated. For example, the feature file 361 and the similar product database 362 are stored in the HDD 36, but a program for processing product sales data may be stored. The feature file 361 is a product file in which associations between information related to product sales registration and images obtained by photographing the product from a plurality of directions are set for each of the handled products displayed and sold in the store. Functions as a product dictionary. The similar product database 362 is a database that learns combination information of similar products and information on the feature direction of these products. In the drawings, the similar product database 362 is abbreviated as “similar product DB”.

The interface 35 is connected to the product identification device 2 and enables data transmission / reception with the product identification device 2.
The printer 38 prints on a receipt or the like. The POS terminal 3 prints transaction details of each transaction on a receipt under the control of the CPU 311.
The power supply 39 supplies power to each part of the POS terminal 3.

FIG. 3 is a functional block diagram showing an outline of the product identification device 2 and the like in the first embodiment. In the following description, FIGS. 1 and 2 will be referred to as appropriate.
The CPU 211 of the merchandise item identification device 2 executes a program stored in the ROM 212, whereby an image acquisition unit 90, an object detection unit 91, a similarity calculation unit 92, a similarity determination unit 93, a guidance unit 94, and a confirmation notification unit 95, the input acquisition unit 96 and the information output unit 97 are embodied. Similarly, the CPU 311 of the POS terminal 3 embodies the sales registration unit 99 by executing a program stored in the HDD 36. The HDD 36 of the POS terminal 3 further stores a feature amount file 361 and a similar product database 362.

  The image acquisition unit 90 outputs an imaging on signal to the camera 27 and causes the camera 27 to start an imaging operation. Further, the image acquisition unit 90 sequentially captures frame images captured by the camera 27 and stored in the RAM 213. The frame image is captured by the image acquisition unit 90 in the order stored in the RAM 213.

The object detection unit 91 detects all or part of the object included in the frame image captured by the image acquisition unit 90 using a pattern matching technique or the like.
Specifically, when the operator points the product to the reading window 52 for sales registration, the image acquisition unit 90 captures an image of the product with the camera 27. The object detection unit 91 binarizes the captured frame image and extracts a contour line. Next, the object detection unit 91 detects (recognizes) the object that is the product by comparing the contour line extracted from the previous frame image with the contour line extracted from the current frame image.

  Another specific method is shown below. When the operator grips the product with his / her hand and points it toward the reading window 52 for sales registration, the image acquisition unit 90 takes an image of the product and the hand with the camera 27. The object detection unit 91 detects the presence or absence of a skin color region from the captured frame image. When the skin color area is detected, that is, when the operator's hand is detected, the object detection unit 91 detects a contour line in the vicinity of the skin color area. As a result, the contour of the product that the operator's hand is supposed to hold is extracted. When the contour indicating the shape of the hand is detected and the contour of another object is further detected in the vicinity of the contour of the hand, the object detection unit 91 detects a product from the contour of the object.

The similarity calculation unit 92 reads the state of the surface such as the hue of the product and the unevenness of the surface from the product image captured by the camera 27 as a feature amount. The similarity calculation unit 92 does not consider the contour or size of the product. Thereby, the similarity calculation part 92 can shorten processing time.
Further, the similarity calculation unit 92 reads the surface state such as the color of the product handled and the surface irregularities from the product image of each product (hereinafter referred to as “handled product”) recorded in the feature file 361 as a feature value. The similarity between the photographed product and the handled product recorded in the feature file 361 is calculated by comparing the feature quantity with the photographed product. Here, the similarity refers to the degree of similarity of all or some of the products when the product image assumed by each product recorded in the feature file 361 is assumed to be 100% similarity. Is shown. For example, the similarity calculation unit 92 may calculate the similarity by changing the weighting between the hue and the unevenness state of the surface.

Recognizing an object included in an image in this way is called generic object recognition. In "Current status and future of general object recognition" written by Keiji Yanai, a survey of general object recognition research is added to methods, data sets and evaluation benchmarks are performed, and the future is prospected.
Keiji Yanai, “Current Status and Future of General Object Recognition”, [online], Journal of Information Processing Society of Japan, November 15, 2007, Vol. 48, No. SIG16, pp. 1-24, [September 8, 2014 Day search], Internet <URL: http: //mm.cs.uec.ac.jp/IPSJ-TCVIM-Yanai.pdf>

A technique for performing general object recognition by dividing an image into regions for each object is described in the following document.
Jamie Shotton et al., “Semantic Texton Forests for Image Categorization and Segmentation”, Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, [Search September 8, 2014], Internet <URL: http: // citeseerx .ist.psu.edu / viewdoc / download? doi = 10.1.1.145.3036 & rep = rep1 & type = pdf>

  It should be noted that the method for calculating the degree of similarity between the captured product image and the product image of the handled product recorded in the feature file 361 is not particularly limited. For example, the similarity between the captured product image and each handled product recorded in the feature file 361 may be calculated as an absolute evaluation or may be calculated as a relative evaluation.

  When calculating the similarity as an absolute evaluation, the captured product image is compared with each handling product recorded in the feature file 361 on a one-to-one basis, and the similarity derived as a result of this comparison is used as it is. do it. Moreover, when calculating a similarity as a relative evaluation, it calculates so that the sum total of the similarity with each handling product may be 1.0 (100%). For example, it is assumed that four products # 1 to # 4 are recorded in the feature amount file 361. The product imaged at this time has, for example, a similarity of 0.65 for product # 1, a similarity of 0.2 for product # 2, and a similarity of 0.1 for product # 3. For the product # 4, the similarity is calculated as 0.05.

  The similarity determination unit 93 compares the degree of similarity between the product image and the product image recorded in the feature amount file 361 for each frame image captured by the image acquisition unit 90. In the present embodiment, a plurality of conditions are provided for the similarity between the product image of the handled product and the photographed image of the product. The similarity determination unit 93 selects the confirmation of the handled product or the navigation (guidance) of the product recognition according to the satisfied conditions. Although the conditions regarding the similarity are not particularly limited, the case where the conditions M and N are used will be described below.

  Here, the condition M is a condition for determining an object on the frame image as one of the handled products recorded in the feature amount file 361. The condition N is a condition for extracting a plurality of candidates for handled products recorded in the feature file 361 from objects on the frame image.

  For example, the similarity determination unit 93 determines that a handled product that satisfies the condition M is a product that has a one-to-one correspondence with an object on the frame image. Further, the similarity determination unit 93 determines that the handled product satisfying the condition N is one of a plurality of candidates for the product captured by the camera 27. And the similarity determination part 93 extracts the candidate of the handling goods with respect to the image | photographed goods by extracting the handling goods which satisfy | fill the conditions N from the several handling goods recorded on the feature-value file 361.

  The conditions M and N are not particularly limited as long as the conditions M and N are set stepwise according to the degree of similarity. For example, the conditions M and N can be set with a plurality of preset threshold values. Here, the case where the conditions M and N are set by the threshold value T and the predetermined value D will be described.

  The similarity determination unit 93 is configured such that the similarity of the product having the highest similarity with each handled product is equal to or greater than the threshold T, and the difference between the similarity of the first product and the similarity of the second product Is equal to or greater than a predetermined value D, it is determined that the condition M is satisfied. The condition M is a condition for uniquely specifying a product.

Furthermore, the similarity determination unit 93 is configured such that the similarity of the product having the highest similarity with each handled product is greater than or equal to the threshold T, and the similarity of the first product and the similarity of the second product If the difference is less than the predetermined value D, it is determined that the condition N is satisfied. Condition N is a condition for a plurality of candidate products. And the handling goods which satisfy | filled the condition N guide | invade rotation of goods to an operator, and it determines with the confirmation of another surface being required.
Each condition M and N can be set as appropriate, and is not limited to the above example.

  The guiding unit 94 causes the display 221 to display information for guiding the handling product that satisfies the condition N to rotate. More specifically, the guide unit 94 reads out the photographic images in each direction of the candidate products that satisfy the condition N from the feature amount file 361, and sets the handling products in the direction in which the similarity with the candidate products is likely to increase. Prompt to rotate. For example, when the stuffed animal is photographed from the top of the head (upward), the guide unit 94 causes the display 221 to display an arrow that prompts the user to turn to show the face that is a feature of the stuffed animal.

For example, a rabbit stuffed animal is recognized as an object using an image with its upper side facing the camera 27, and the similarity with the reference image of the stuffed rabbit in the feature amount file 361 is not so high that the product can be confirmed. Suppose the stuffed toy and the stuffed bear are listed as candidates. The feature amounts of the reference images taken from the respective directions of the stuffed rabbit and the bear doll which are candidate products are compared, and the direction in which the difference is large and the difference in appearance is significant is identified. Assuming that the front direction is specified, the guide unit 94 causes the display 221 to display an arrow that prompts the operator to rotate the product in order to rotate the front direction from the upper direction.
The guiding unit 94 stores such combination information of candidate products and information on the characteristic directions of these candidate products in the similar product database 362. Thereby, it is possible to navigate quickly without performing the calculation process for the same candidate under the same photographing condition.

  Even if an arrow prompting the operator to rotate the product is displayed on the display 221, an image acquisition process by the image acquisition unit 90, an object detection process by the object detection unit 91, and a similarity calculation unit 92 The similarity calculation processing by is continued.

  The confirmation notifying unit 95 notifies the operator or customer that the product captured by the camera 27 is uniquely confirmed as a handled product that satisfies the conditions, by image output or voice output.

  More specifically, the confirmation notifying unit 95 causes the display 221 to display a confirmation screen indicating that the handled product that satisfies the condition is uniquely confirmed as the product captured by the camera 27 (confirmed product). Information about the product is output to the speaker 28. The speaker 28 notifies the operator and the customer of information indicating the confirmed product by outputting the input information.

The input acquisition unit 96 receives various input operations corresponding to the display on the display 221 via the touch panel 222 or the keyboard 23.
The information output unit 97 outputs information indicating the product (for example, product ID, product name, discount information, etc.) to the POS terminal 3 via the interface 25 for the confirmed product confirmed as described above.

  The information output unit 97 may output the sales quantity separately input via the touch panel 222 or the keyboard 23 to the POS terminal 3 together with the product ID and the like. Further, as information output from the information output unit 97 to the POS terminal 3, the product ID read out from the feature file 361 by the information output unit 97 may be directly notified, or a product for which the product ID can be specified. The POS terminal 3 may be notified of the name, the product image, and the file name of the photo image.

  The sales registration unit 99 of the POS terminal 3 performs sales registration of the corresponding product based on the product ID and the sales quantity output from the information output unit 97. More specifically, the sales registration unit 99 refers to the feature amount file 361 and records the notified product ID and the corresponding product classification, product name, unit price, and sales quantity in a sales master file or the like. Sales registration (temporary registration).

FIG. 4 is a front view of the product identification device 2 according to the first embodiment.
The commodity identification device 2 is installed on the counter table 5. The product identification device 2 includes a thin rectangular parallelepiped housing 51, a display / operation unit 22 attached to the upper part of the housing 51, and a customer display installed face down on the left back side of the display / operation unit 22. 24.
A reading window 52 is provided on the front surface of the housing 51.
The reading window 52 is provided with an illumination 271 and a camera 27 (imaging means). The recognition area 8M is an area where the camera 27 captures an object, detects the object, and specifies a product.
The product identification device 2 detects an object by the camera 27, identifies the product name of the object, and causes the POS terminal 3 to perform product registration processing.

FIG. 5 is a diagram showing the configuration of the feature amount file 361.
Each record of the feature amount file 361 includes a feature amount column 361a, an image link column 361b, a direction column 361c, a product ID column 361d, a unit price column 361e, and a product name column 361f. The feature amount file 361 stores a combination of a reference image obtained by shooting each product from a plurality of directions and information on the shooting direction.
The feature amount column 361a is a column for storing the feature amount vector of the reference image of the product. The stored information in the feature amount column 361a is obtained by analyzing an image in the image link column 361b described later.
The image link column 361b is a column for storing the link of the reference image of this product. The reference image indicated by the link is an image obtained by photographing the product from any one of six directions.
The direction column 361c is a vector indicating the shooting direction of the reference image of the product.
The product ID column 361d is a column for storing information for identifying the product.
The unit price column 361e is a column for storing the unit price of this product.
The product name column 361f is a column for storing the name of this product.

FIGS. 6A to 6F are diagrams showing reference images in six directions of the stuffed rabbit.
FIG. 6A shows a reference image in the front direction of the stuffed rabbit. At this time, if the X axis is defined as the right direction when viewed from the front direction, the Y axis is defined as the upward direction when viewed from the front direction, and the Z axis is defined as the back direction when viewed from the front direction, the vector of the shooting direction is (X, Y, Z ) = (0, 0, 1). In general, for a stuffed animal, the reference image in the front direction is the direction most suitable for object recognition.

FIG.6 (b) has shown the reference | standard image of the back direction of the stuffed rabbit. At this time, the vector of the shooting direction is (X, Y, Z) = (0, 0, −1).
FIG. 6C shows an upper reference image of the stuffed rabbit. At this time, the vector of the shooting direction is (X, Y, Z) = (0, −1, 0).
FIG. 6D shows a reference image in the downward direction of the stuffed rabbit. At this time, the vector of the shooting direction is (X, Y, Z) = (0, 1, 0).
FIG. 6E shows a reference image in the right direction of the stuffed rabbit. At this time, the vector of the shooting direction is (X, Y, Z) = (− 1, 0, 0).
FIG. 6F shows a reference image in the left direction of the stuffed rabbit. At this time, the vector of the shooting direction is (X, Y, Z) = (1, 0, 0).
In this way, by recording the reference images of the products in six directions in advance, it is possible to easily recognize the products regardless of the direction in which the products are picked up.

FIGS. 7A to 7F are diagrams showing reference images in six directions of the stuffed bear.
FIG. 7A shows a reference image in the front direction of the stuffed bear. At this time, the vector in the shooting direction is (X, Y, Z) = (0, 0, 1), and the difference from the stuffed rabbit shown in FIG. Therefore, the bear stuffed animal and the rabbit stuffed animal have the smallest similarity between the images in the front direction.

FIG.7 (b) has shown the reference | standard image of the back direction of the stuffed bear. At this time, the vector of the shooting direction is (X, Y, Z) = (0, 0, −1). The difference from the stuffed rabbit shown in FIG. 6B is relatively small.
FIG.7 (c) has shown the reference | standard image of the upward direction of the stuffed bear. At this time, the vector of the shooting direction is (X, Y, Z) = (0, −1, 0). The difference from the stuffed rabbit shown in FIG. 6C is relatively small.
FIG. 7D shows a reference image in the downward direction of the stuffed bear. At this time, the vector of the shooting direction is (X, Y, Z) = (0, 1, 0). The difference from the stuffed rabbit shown in FIG. 6 (d) is relatively small.
FIG. 7E shows a reference image in the right direction of the stuffed bear. At this time, the vector of the shooting direction is (X, Y, Z) = (− 1, 0, 0). The difference from the stuffed rabbit shown in FIG. 6 (e) is relatively small.
FIG. 7F shows a reference image in the left direction of the stuffed bear. At this time, the vector of the shooting direction is (X, Y, Z) = (1, 0, 0). The difference from the stuffed rabbit shown in FIG. 6 (f) is relatively small.

FIG. 8 is a diagram illustrating a configuration of the similar product database 362 according to the first embodiment.
The similar product database 362 includes a first product ID column 362a, a first direction column 362b, a second product ID column 362c, a second direction column 362d, a feature image column 362e, a feature direction column 362f, and a rotation axis vector column 362g. Composed. The similar product database 362 is a learning result of past product identification, and by referring to this, the object recognition calculation process can be skipped and processed at high speed.
The first product ID column 362a is a column that stores one product ID of two similar candidate products, and the second product ID column 362c is a column that stores the other product ID.
The first direction column 362b is a column that stores the shooting direction of the reference image of one of the two similar candidate products. The second direction column 362d is a column that stores the shooting direction of the reference image of the other product. It is.
The feature image column 362e is a column for storing links of images in which the similarity between two candidate products among the reference images of one of the two similar candidate products is minimum, and thus the difference in features is remarkable. is there.
The feature direction column 362f is a column for storing a direction vector in which the similarity between the two candidate products becomes the minimum among the reference images of one of the two similar candidate products, and thus the difference in features becomes remarkable. .
The rotation axis vector column 362g is a column for storing a rotation axis vector when the target product is directed in the minimum similarity direction. By converting this vector into a vector on the captured image, an arrow on the screen that induces rotation can be displayed.

FIG. 9 is a flowchart showing the provisional product registration process according to the first embodiment.
In the temporary registration process of the product according to the first embodiment, for example, an operator (store clerk) takes out the stuffed rabbit which is the product of the first shopping basket 4L (see FIG. 1) and stores it in the camera 27 of the reading window 52. It is a series of processes until the stuffed rabbit is temporarily registered.

In step S <b> 11, the image acquisition unit 90 outputs an imaging on signal to the camera 27 and starts capturing (imaging) a product image by the camera 27. The image acquisition unit 90 captures a frame image (captured image) captured by the camera 27 and stored in the RAM 213.
In step S12, the object detection unit 91 performs object recognition processing on the frame image captured by the image acquisition unit 90, and tries to recognize (detect) all or part of the object that is the product.

  In step S <b> 13, the object detection unit 91 determines whether or not all or part of the object that is the product has been successfully recognized. If the object detection unit 91 succeeds in recognizing the object that is the product (Yes), the process proceeds to step S14. If the object detection unit 91 does not succeed in recognizing the object that is the product (No), the object detection unit 91 proceeds to the process in step S11. Return. Specifically, Steps S11 to S13 are a series of processes in which the operator puts a product on the camera 27 of the reading window 52 and the product identification device 2 succeeds in detecting (recognizing) an object that is the product.

In step S14, the similarity calculation unit 92 reads the feature amount of the product from all or part of the product image. Further, the similarity determination unit 93 determines whether or not there is a handling product whose similarity is equal to or greater than the threshold value T in the feature amount file 361, and searches and calls the product name and the product unit price of the product.
In step S <b> 15, the similarity determination unit 93 determines whether or not the product is uniquely specified based on whether or not there is a handled product in the feature amount file 361 whose similarity is equal to or greater than the threshold T. The similarity determination unit 93 proceeds to the process of step S22 if the handled product whose similarity is equal to or greater than the threshold T is specified, and proceeds to the process of step S16 if there are a plurality of candidate products. If no candidate product exists, the process returns to step S11. Specifically, the similarity determination unit 93 determines that the stuffed rabbit and the stuffed bear exist as candidate products. Further, the similarity determination unit 93 determines that this product is most similar to the stuffed rabbit and then similar to the stuffed bear.

The process from step S16 to step S21 is a series of processes related to the rotational navigation of the product. This process is repeated when there are a plurality of candidate products.
In step S <b> 16, the guiding unit 94 determines whether or not the candidate product rotation direction information exists in the similar product database 362. That is, the guiding unit 94 searches the similar product database 362 for a combination of the first and second candidate products having a high degree of similarity, and determines the presence of the rotation direction information of the candidate product depending on whether or not a corresponding record exists. to decide. If there is rotation direction information of the candidate product in the similar product database 362 (Yes), the guiding unit 94 proceeds to the process of step S19, and if there is no rotation direction information (No), the guide unit 94 proceeds to the process of step S17. .

In step S <b> 17, the guide unit 94 compares reference images in the same direction with respect to two candidate products, and specifies a reference image in which a difference in features is significant. More specifically, reference images in six directions of a stuffed rabbit and a stuffed bear are compared. Here, the degree of similarity between the front images of both products is minimal, that is, the difference in features is significant.
In step S <b> 18, the guiding unit 94 identifies a shooting direction in which the difference in features becomes significant, and records the shooting direction in the similar product database 362 together with the combination information of the two candidate products. Thereby, when similar candidate goods are detected, it is possible to promptly display without calculating the similarity.

In step S <b> 19, the guiding unit 94 calculates a rotation axis vector based on the outer product of the shooting direction vector in which the feature difference becomes significant and the shooting direction vector of the reference image. Furthermore, the guide unit 94 records the calculated rotation axis vector in the similar product database 362.
In step S20, the guide unit 94 converts the rotation axis vector into a display coordinate system. Thereby, the rotation axis in the display coordinate system can be specified.
In step S <b> 21, the guiding unit 94 navigates the rotation of the product that the operator is hesitant by displaying an arrow on the screen. When the process of step S21 ends, the process returns to step S11.

The processing from step S22 to step S23 is processing when a product is uniquely specified.
In step S <b> 22, the confirmation notification unit 95 displays the product name and unit price in a window together with the product image, and outputs information about the confirmed product to the speaker 28. As a result, the confirmation notification unit 95 can notify the operator that the product has been uniquely identified.
In step S23, the information output unit 97 outputs information indicating the product (for example, product ID, product name, discount information, etc.) to the POS terminal 3 via the interface 25 for the confirmed product, and the processing of FIG. Exit. The POS terminal 3 temporarily registers the product and registers it in the product master file together with an instruction for payment.

FIGS. 10A and 10B are screen examples showing an operation example of the first embodiment.
FIG. 10A is a display example of the display 221 when the upward direction of the stuffed rabbit is put on the camera 27.
The display 221 displays an arrow for guiding the rotation of the stuffed toy together with the stuffed rabbit. This arrow guides the operator to rotate the stuffed rabbit in the direction of the arrow. At this time, the similarity determination unit 93 determines the stuffed rabbit and the stuffed bear as candidate products.
Thereafter, the operator rotates the stuffed rabbit in the direction of the arrow. As a result, the stuffed rabbit's face is scolded by the camera 27.

FIG. 10B is a display example of the display 221 when the front direction of the stuffed rabbit is looked at by the camera 27.
On the display 221, a “Usaki / Stuffed Toy / 800 Yen” window is displayed together with a stuffed rabbit. This window informs the operator that the product has been uniquely identified.

(Second Embodiment)
FIG. 11 is a diagram illustrating a configuration of the similar product database 362A according to the second embodiment. The same elements as those in the similar product database 362 of the first embodiment are given the same reference numerals.
Unlike the similar product database 362 of the first embodiment, the similar product database 362A does not include the feature direction column 362f and the rotation axis vector column 362g, but otherwise includes the same column.

FIG. 12 is a flowchart showing a provisional product registration process according to the second embodiment. The same reference numerals are assigned to the same elements as those in the flowchart of the temporary registration process of the product according to the first embodiment shown in FIG.
In the temporary registration process of the product according to the second embodiment, for example, the operator (store clerk) takes out the stuffed bear which is the product of the first shopping basket 4L (see FIG. 1) and stores it in the camera 27 of the reading window 52. It is a series of processes until the bear stuffed animal is temporarily registered.

The processing from step S11 to step S16 is the same as each processing of the first embodiment. Specifically, the similarity determination unit 93 determines that the stuffed rabbit and the stuffed bear exist as candidate products. Further, the similarity determination unit 93 determines that this product is most similar to a bear stuffed animal and then similar to a rabbit stuffed animal.
In step S30, the guiding unit 94 determines whether there is information on the reference image in which the difference in the feature of the candidate product is significant in the similar product database 362A. If this information exists (Yes), the guiding unit 94 proceeds to the process of step S33, and if this information does not exist (No), the guide unit 94 proceeds to the process of step S31.

In step S31, the guide unit 94 compares reference images in the same direction with respect to two candidate products, and specifies a reference image having a distinctive or remarkable feature. Specifically, the reference images in the six directions of the stuffed bear and the stuffed rabbit are compared. Here, the difference between the front images of both products is remarkable, and the similarity is the smallest.
In step S <b> 32, the guide unit 94 records a reference image in which the difference between the features of both products is significant together with the combination information of the two candidate products in the similar product database 362. Thereby, when similar candidate goods are detected, it is possible to promptly display without calculating the similarity.
In step S <b> 33, the guiding unit 94 navigates the rotation of the product on which the operator is hesitant by displaying a reference image in which the difference between the features of both products is significant. When the process of step S20 ends, the process returns to step S11.
By operating in this way, the guide unit 94 can navigate the product that the operator is hesitating so that the product is in the direction of the reference image where the difference in features is significant.

FIG. 13 is a screen example showing an operation example of the second embodiment.
FIG. 13A is a display example of the display 221 when the upward direction of the stuffed bear is looked at the camera 27.
At this time, the bear stuffed animal is displayed on the full screen on the display 221, and a reference image in the front direction of the stuffed animal and a guidance “rotate like this” are displayed. This guidance guides the operator so that the front direction of the teddy bear is directed toward the camera 27.
Thereafter, the operator rotates the stuffed bear in the direction of the arrow. As a result, the face of the stuffed bear is deceived by the camera 27. Thereby, the product identification device 2 can identify that this product is a stuffed bear.

FIG. 13B is a display example of the display 221 when the front direction of the teddy bear is looked at by the camera 27.
At this time, a window of “bear, stuffed toy, 1200 yen” is displayed on the display 221 together with the stuffed bear. This window informs the operator that the product has been uniquely identified.

The present invention is not limited to the above-described embodiment, and can be modified without departing from the spirit of the present invention. For example, there are the following (a) to (d).
(A) The shooting direction of the reference image of the product is not limited to six directions.
(B) Product rotation navigation is not limited to the above embodiment. For example, product rotation may be navigated by voice or polygon rotation animation.
(C) The number of candidate products is not limited to two, and the direction in which the feature (difference) is noticeable may be determined from three or more candidate products.
(D) The comparison of the reference images of the candidate products is not limited to the comparison with the same direction defined at the time of shooting. You may compare by the relative direction from the direction of two reference images with high similarity.

The invention described in the scope of claims attached to the application of this application will be added below. The item numbers of the claims described in the appendix are as set forth in the claims attached to the application of this application.
[Appendix]
<Claim 1>
Photographing means for photographing an image of the object that the operator has deceived;
Object detecting means for detecting the object from an image taken by the photographing means;
Similarity determination means for determining similarity by comparing the object detected by the object detection means with a reference image in a plurality of directions of each product;
If the object is similar to a plurality of products, a reference image in each direction of the plurality of products is compared to identify a significant difference, and a direction related to the specified reference image is directed to the photographing unit. And guiding means for guiding the rotation of the object to the operator;
A product identification device comprising:
<Claim 2>
Display means for displaying an image taken by the photographing means and displaying an arrow in a direction for guiding the rotation of the object if the object recognized from the image taken by the photographing means is similar to a plurality of products;
The product identification device according to claim 1, comprising:
<Claim 3>
The guiding means is identified if the object recognized from the image photographed by the photographing means is similar to a plurality of products and the reference images of the plurality of products are compared with each other to identify significant differences. Storing information on the direction of the reference image in the storage means;
The product identification device according to claim 1, wherein the product identification device is a product identification device.
<Claim 4>
When the image captured by the image capturing unit is displayed and the object recognized from the image captured by the image capturing unit is similar to a plurality of products, an image having a remarkable difference among the reference images of the plurality of products is displayed. Display means for displaying,
The product identification device according to claim 1, comprising:
<Claim 5>
The guiding means is specified if the object recognized from the image taken by the photographing means is similar to a plurality of products and the reference images of the plurality of products are compared to identify a significant difference. Storing information for identifying the reference image in the storage means;
The product identification device according to claim 1, further comprising:
<Claim 6>
A shooting step for shooting an image of an object that the operator has deceived;
An object detection step of detecting the object from the image captured in the imaging step;
A similarity determination step of determining the similarity by comparing the detected object with a reference image in a plurality of directions of each product;
If the object is similar to a plurality of products, the reference images in each direction of the plurality of products are compared to identify the difference, and the direction of the specified reference image is photographed. A guidance step for guiding the rotation of the object to an operator;
A product recognition navigation method for a product identification device, comprising:

1 Product Registration Device 2 Product Identification Device 21 Microcomputer (Control Unit)
211 CPU
212 ROM
213 RAM
22 Display / Operation Unit 221 Display (Display means)
222 Touch Panel 23 Keyboard 24 Customer Display 27 Camera (Photographing Means)
28 Speaker 3 POS terminal 36 HDD
361 Feature file 362 Similar product database 90 Image acquisition unit 91 Object detection unit (object detection means)
92 similarity calculation unit 93 similarity determination unit (similarity determination means)
94 Guiding part (guidance means)
95 Confirmation notification part 96 Input acquisition part 97 Information output part 99 Sales registration part

The present invention relates to an information display device and a guidance display method.

An object of the present invention is to enable navigation in a direction in which an operator leans an object so that an operator can easily understand.

In order to solve the above problems, an information display device according to the present invention includes an acquisition unit that acquires an image of an object, an object detection unit that detects the object from an image acquired by the acquisition unit, and the object detection unit. Display control means for causing an operator to display an arrow in a direction for guiding the rotation of the object together with the image so that the detected object is directed in a predetermined direction.

The guidance display method of the information display device according to the present invention includes an acquisition step of acquiring an image of an object, an object detection step of detecting the object from the image acquired by the acquisition step, and the object detection step. And a display control step of displaying an arrow of a direction for guiding the rotation of the object together with the image so that the object is directed in a predetermined direction.

According to the present invention, it is possible to navigate the direction in which the operator is leaning so that the operator can easily understand .

Claims (6)

Photographing means for photographing an image of the object that the operator has deceived;
Object detecting means for detecting the object from an image taken by the photographing means;
Similarity determination means for determining similarity by comparing the object detected by the object detection means with a reference image in a plurality of directions of each product;
If the object is similar to a plurality of products, a reference image in each direction of the plurality of products is compared to identify a significant difference, and a direction related to the specified reference image is directed to the photographing unit. And guiding means for guiding the rotation of the object to the operator;
A product identification device comprising: Display means for displaying an image taken by the photographing means and displaying an arrow in a direction for guiding the rotation of the object if the object recognized from the image taken by the photographing means is similar to a plurality of products;
The product identification device according to claim 1, comprising: The guiding means is identified if the object recognized from the image photographed by the photographing means is similar to a plurality of products and the reference images of the plurality of products are compared with each other to identify significant differences. Storing information on the direction of the reference image in the storage means;
The product identification device according to claim 1, wherein the product identification device is a product identification device. When the image captured by the image capturing unit is displayed and the object recognized from the image captured by the image capturing unit is similar to a plurality of products, an image having a remarkable difference among the reference images of the plurality of products is displayed. Display means for displaying,
The product identification device according to claim 1, comprising: The guiding means is specified if the object recognized from the image taken by the photographing means is similar to a plurality of products and the reference images of the plurality of products are compared to identify a significant difference. Storing information for identifying the reference image in the storage means;
The product identification device according to claim 1, further comprising: A shooting step for shooting an image of an object that the operator has deceived;
An object detection step of detecting the object from the image captured in the imaging step;
A similarity determination step of determining the similarity by comparing the detected object with a reference image in a plurality of directions of each product;
If the object is similar to a plurality of products, the reference images in each direction of the plurality of products are compared to identify the difference, and the direction of the specified reference image is photographed. A guidance step for guiding the rotation of the object to an operator;
A product recognition navigation method for a product identification device, comprising:

Images