JP6384309B2 - Product identification device, product recognition navigation method and program - Google Patents

Description

The present invention relates to a product identification device , a product recognition navigation method, and a program .

Conventionally, an electronic cash register (product registration device) reads a data code such as a barcode attached to a product by a barcode reading unit and identifies the product. The bar code assigned to the product includes maker code information, item code information, and check digit information. After determining that the barcode is correctly read based on the check digit information, the product registration apparatus identifies the product based on the item code information. By printing the barcode on the product packaging in advance, the burden on the cash register operator can be reduced.
However, it is extremely troublesome to individually attach barcodes to unpackaged products such as fruits and vegetables, which takes time and cost. Conventionally, for fruit and vegetable products, operations such as pressing a button indicating a product, touching a screen, or selecting a corresponding product code from a separately prepared bar code table and reading it with a scanner have been performed.

  In recent years, product registration apparatuses for recognizing and identifying fruits and vegetables and the like, and registering the identified products for sales are becoming popular. This merchandise registration apparatus extracts the feature quantity of the appearance from image information obtained by photographing fruit and vegetable products and compares it with a feature quantity database relating to the appearance of the merchandise, thereby identifying which merchandise the article is. Thereby, even if the product is not packaged in advance, the product registration device can recognize the product without sticking the barcode individually, so the burden on the store side can be reduced.

  Each product has a difference in appearance. When the difference between the target product and the appearance of the standard product is large, it is difficult to uniquely determine the target product due to the similarity during object recognition. In such a case, the product registration device having the object recognition function can display a confirmation screen of provisionally confirmed products that are candidates for the target product, or select a plurality of products that are candidates for the target product on the screen. It was displayed. The operator needs to perform an operation such as touching a product to be packaged from this option.

  For example, in paragraph 0068 of Patent Document 1, “In the product candidate presentation area 83, product candidate illustration images G1, G2, G3 and product names are displayed in descending order of registered products with high similarity. The images G1, G2, and G3 are configured to be selectable in accordance with a selection operation on the touch panel 105. A selection button 84 for selecting a product from the product list is provided below the product candidate presentation area 83. The product selected from the product list is processed as the above-mentioned provisionally confirmed product. "

JP2013-182323A (FIG. 8 and paragraph 0068)

  When purchasing a product in a supermarket, the customer puts the product into a shopping cart or the like, and then passes the shopping cart to an operator (clerk) of the cash register to request payment of the product price. At this time, the operator reads each item in the shopping cart (hereinafter referred to as the “first shopping basket”) brought by the customer in order in the product registration device, thereby selling the product information of the product. Register each product in the first shopping basket by registering the information temporarily and repeating the operation of putting the product into another shopping basket (hereinafter referred to as “second shopping basket”). .

At that time, the product registration apparatus may not be able to recognize the product immediately. For example, if the barcode portion of the product with the barcode cannot be photographed well, the product registration device cannot immediately recognize the product. In this case, the operator causes the merchandise registration apparatus to recognize the merchandise by an operation of bringing the merchandise as close as possible to the camera of the merchandise registration apparatus.
If the target product for object recognition cannot be photographed at an appropriate angle, the product registration device cannot immediately recognize the product. In this case, the operator may operate the product as close as possible to the camera of the product registration device with the same feeling as the barcode reading operation.

  However, the cause when the product registration apparatus cannot recognize the product as an object is often the direction of the product presented to the camera is not appropriate. For example, even if the direction of the product including many feature quantities that can identify the product is the front direction, the operator points the top surface of the product toward the camera, or conversely, The main factor is that the operator points the front of the product toward the camera even though the direction is the top direction. In such a case, the product registration device presents a plurality of registered product candidates.

  In the invention described in Patent Document 1, when there are a plurality of registered merchandise candidates, one registered merchandise candidate selected by the touch panel operation is set as a provisionally confirmed merchandise. However, this selection of candidate products hinders the flow of operator product registration. Therefore, it takes time and labor for the operator, and further causes waiting time for the customer.

Accordingly, an object of the present invention is to provide a product identification device , a product recognition navigation method, and a program for identifying a product without hindering the flow of product registration by an operator.

To achieve the above object, product identification apparatus according to the present invention includes a photographing control means for obtaining an image of the object by imaging the object by the operator holds up the camera by the camera, acquired by the imaging control unit A similarity determination unit that determines whether the object is similar to a plurality of products based on an image of the object, and a case where the similarity determination unit determines that the object is similar to a plurality of products, When the image of the object is an image obtained by capturing any part of the entire image of the object so as to deviate from the angle of view of the camera, a plurality of images determined to be similar to the object a specifying unit that differences between products to identify the salient parts, so that a specific portion by the specifying means fits into the angle of view of the camera, to induce movement of the object relative to the operator Characterized in that it comprises an induction means.

Also, product recognition navigation method according to the present invention, a photographing control step of acquiring an image of the object by imaging the object by the operator holds up the camera by the camera, the image of the object acquired by the photographing control step a similarity determining step of determining whether the object is similar to a plurality of goods based on the a case where the object in the similarity judgment step is determined to be similar to multiple items, the image of the object Difference between a plurality of products determined to be similar to the object when any part of the overall image of the object is captured and acquired so as to deviate from the angle of view of the camera a specifying step is to identify the salient parts, so the that portion identified in a particular step fits into the angle of view of the camera, movement of the object relative to the operator Characterized in that it comprises an induction step of induction,.
Further, the program according to the present invention is based on the image of the object acquired by the imaging control unit, the imaging control unit that acquires an image of the object by imaging the object that the operator holds over the camera with the camera. Similarity determination means for determining whether or not the object is similar to a plurality of products, and when the similarity determination means determines that the object is similar to a plurality of products, the image of the object is the When any part of the entire image of the object is an image that is captured and acquired so as to deviate from the angle of view of the camera, there is a significant difference between a plurality of products determined to be similar to the object A specifying means for specifying the correct part, and a guiding means for guiding the movement of the object to the operator so that the part specified by the specifying means falls within the angle of view of the camera. Characterized in that to function as a.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to provide the goods identification apparatus and goods recognition navigation method which identify goods, without disturbing the flow of an operator's goods registration.

It is a perspective view which shows the external appearance of the goods identification apparatus etc. in this embodiment. It is a lineblock diagram showing the outline of the goods discernment device etc. in this embodiment. It is a functional block diagram which shows the outline of the goods identification apparatus etc. in this embodiment. It is a front view of the goods identification device in this embodiment. It is a figure which shows the structure of a feature-value file. It is a figure which shows the reference | standard image of a persimmon and a radish. It is a figure which shows the structure of a similar region database. It is a flowchart which shows the provisional registration process of the goods in this embodiment. It is a screen which shows the 1st example of operation. It is a screen which shows the 2nd example of operation.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.
FIG. 1 is a perspective view showing the external appearance of a product identification device 2 and the like in the present embodiment.
As shown in FIG. 1, the product registration device 1 includes a product identification device 2 that registers each product, and a POS terminal 3 that performs sales registration and settlement of products related to each transaction. The product identification device 2 is a POS connection scanner that reads and registers information about products.
The merchandise item identification device 2 is installed at the center in the longitudinal direction of the counter table 5 having a horizontally long table shape. The commodity identification device 2 includes a thin rectangular parallelepiped housing 51. A camera 27 (see FIG. 2) is arranged on the front surface of the housing 51 through a reading window 52.

  A display / operation unit 22 is attached to the upper portion of the housing 51. The display / operation unit 22 is provided with a display 221. A touch panel 222 (see FIG. 2) is laminated on the surface of the display 221. A keyboard 23 is arranged on the right side of the display 221. A card reading groove of a card reader (not shown) is provided on the right side of the keyboard 23. On the left back side of the display / operation unit 22, a customer display 24 for providing information to the customer is installed face down.

The POS terminal 3 is placed on the upper surface of the drawer 37 on the checkout table 6. The drawer 37 is controlled by the POS terminal 3 for the opening operation.
On the upper surface of the POS terminal 3, a keyboard 33 for operation by an operator (clerk) is arranged. A display 321 for displaying information is provided on the upper back side of the keyboard 33 as viewed from the operator. A touch panel 322 (see FIG. 2) is stacked on the surface of the display 321. Further on the back side of the display 321, a customer display 34 for displaying information is installed so as to be rotatable in the left-right direction. The customer display 34 shown in FIG. 1 faces the front side of the figure. This customer display 34 displays information toward the customer by rotating it toward the far side of the figure.

A horizontally long table-like counter table 5 is arranged so as to form an L shape with the checkout table 6 on which the POS terminal 3 is placed. A load receiving surface is formed on the upper surface of the counter table 5. The customer undergoes a product registration process while moving from the left side to the right side in FIG. That is, the first shopping basket 4L and the second shopping basket 4R for storing products are placed on the load receiving surface of the counter table 5. Hereinafter, when there is no particular distinction between the first shopping basket 4L and the second shopping basket 4R, the shopping basket 4 may be simply described. These shopping baskets 4 are not limited to the so-called basket shape, and may be trays or the like. The shopping basket 4 is not limited to a so-called basket shape, and may be a box shape or a bag shape.
The first shopping basket 4L is brought in by a customer and stores products related to one transaction. The second shopping basket 4R is placed at a position sandwiching the merchandise item identification device 2 from the first shopping basket 4L. The product in the first shopping basket 4L is taken out by an operator who operates the product identification device 2 and moved to the second shopping basket 4R. In this moving process, the product is picked up by the reading window 52 of the product identification device 2. At this point, the camera 27 (see FIG. 2) arranged in the reading window 52 images the product.

In the product identification device 2, the product included in the image captured by the camera 27 is identified on the display 221 as to which product recorded in a feature value file 361 (see FIG. 2) described later corresponds. The product identification device 2 further notifies the POS terminal 3 of the product ID of the identified product. Based on the product ID notified from the product identification device 2, the POS terminal 3 stores information related to sales registration such as the product classification, product name, and unit price of the product corresponding to the product ID, as a sales master file (not shown), and the like. And record sales.
The present invention may be applied to a self-registration in which a customer himself operates a cash register as an operator, and is not limited.

FIG. 2 is a configuration diagram showing an outline of the product identification device 2 and the like in the present embodiment.
The product registration apparatus 1 includes a product identification device 2 and a POS terminal 3.
The product identification device 2 includes a microcomputer 21, a display / operation unit 22, an interface 25, a camera 27, a speaker 28, and a power source 29.
The microcomputer 21 is a control unit configured by connecting a ROM (Read Only Memory) 212 and a RAM (Random Access Memory) 213 to a CPU (Central Processing Unit) 211 via a bus. The ROM 212 stores a program executed by the CPU 211.
The CPU 211 is connected to the display / operation unit 22, the interface 25, the camera 27, and the speaker 28 via an internal bus and input / output circuits (not shown).

The display / operation unit 22 includes a display 221, a touch panel 222, a customer display 24, and a keyboard 23, and the operation is controlled by the CPU 211.
The display 221 is a display unit that displays information for the operator according to an instruction from the CPU 211. The touch panel 222 receives an operation input for information displayed on the display 221. The customer display 24 displays information for the customer in accordance with an instruction from the CPU 211.
The keyboard 23 is composed of a plurality of operation keys and accepts operator input.
The interface 25 is connected to the interface 35 of the POS terminal 3 to enable data transmission / reception with the POS terminal 3.

  The camera 27 is a color CCD image sensor, a color CMOS image sensor, or the like, and is an imaging unit that performs imaging from the reading window 52 (see FIG. 1) under the control of the CPU 211. The camera 27 captures a moving image of 30 [fps], for example. Frame images (captured images) sequentially captured by the camera 27 at a predetermined frame rate are stored in the RAM 213.

The speaker 28 generates a preset warning sound or the like. The speaker 28 performs warning sound or voice notification under the control of the CPU 211.
The power source 29 supplies power to each part of the product identification device 2.
The interface 25 is connected to the interface 35 of the POS terminal 3 to enable data transmission / reception with the POS terminal 3.

The POS terminal 3 includes a microcomputer 31, a display 321, a touch panel 322, a keyboard 33, a customer display 34, an interface 35, an HDD 36, a drawer 37, a printer 38, and a power supply 39. Is done.
The microcomputer 31 executes information processing. The microcomputer 31 is configured by connecting a ROM 312 and a RAM 313 to a CPU 311 that executes various arithmetic processes and controls each unit. A drawer 37, a keyboard 33, a display 321, a touch panel 322, a customer display 34, and an HDD 36 (Hard Disk Drive) are connected to the CPU 311 via an internal bus and input / output circuits. These are controlled by the CPU 311.

The display 321 displays information for the operator according to an instruction from the CPU 311. The touch panel 322 receives an operation input for information displayed on the display 321. The customer display 34 displays information for the customer in accordance with an instruction from the CPU 311.
The keyboard 33 includes a temporary fastening key 331, a fastening key 332, and a numeric keypad 333, and accepts an operator's operation input. The numeric keypad 333 includes numeric keys from 0 to 9 and various operator keys.

  The HDD 36 is a storage unit that stores programs and various files. All or a part of the programs and various files stored in the HDD 36 are copied to the RAM 313 and executed by the CPU 311 when the POS terminal 3 is activated. For example, the feature file 361 and the similar part database 362 are recorded in the HDD 36, but a product sales data processing program may be recorded. The feature file 361 is a product file in which associations between information related to product sales registration and images obtained by photographing the product from a plurality of directions are set for each of the handled products displayed and sold in the store. Functions as a product dictionary.

The interface 35 is connected to the product identification device 2 and enables data transmission / reception with the product identification device 2.
The printer 38 prints on a receipt or the like. The POS terminal 3 prints transaction details of each transaction on a receipt under the control of the CPU 311.
The power supply 39 supplies power to each part of the POS terminal 3.

FIG. 3 is a functional block diagram showing an outline of the product identification device 2 and the like in the present embodiment. In the following description, FIGS. 1 and 2 will be referred to as appropriate.
The CPU 211 of the merchandise item identification device 2 executes a program stored in the ROM 212, whereby an image acquisition unit 90, an object detection unit 91, a similarity calculation unit 92, a similarity determination unit 93, a guidance unit 94, and a confirmation notification unit 95, the input acquisition unit 96 and the information output unit 97 are embodied. Similarly, the CPU 311 of the POS terminal 3 embodies the sales registration unit 99 by executing a program stored in the HDD 36. A feature file 361 is further stored in the HDD 36 of the POS terminal 3.

  The image acquisition unit 90 outputs an imaging on signal to the camera 27 and causes the camera 27 to start an imaging operation. Further, the image acquisition unit 90 sequentially captures frame images captured by the camera 27 and stored in the RAM 213. The frame image is captured by the image acquisition unit 90 in the order stored in the RAM 213.

The object detection unit 91 detects all or part of the object included in the frame image captured by the image acquisition unit 90 using a pattern matching technique or the like.
Specifically, when the operator points the product to the reading window 52 for sales registration, the image acquisition unit 90 captures an image of the product with the camera 27. The object detection unit 91 binarizes the captured frame image and extracts a contour line. Next, the object detection unit 91 detects (recognizes) the object that is the product by comparing the contour line extracted from the previous frame image with the contour line extracted from the current frame image.

  Another specific method is shown below. When the operator grips the product with his / her hand and points it toward the reading window 52 for sales registration, the image acquisition unit 90 takes an image of the product and the hand with the camera 27. The object detection unit 91 detects the presence or absence of a skin color region from the captured frame image. When a skin color area is detected, that is, when a store clerk's hand is detected, the object detection unit 91 detects a contour line in the vicinity of the skin color area. As a result, the contour of the product that the operator's hand is supposed to hold is extracted. When the contour indicating the shape of the hand is detected and the contour of another object is further detected in the vicinity of the contour of the hand, the object detection unit 91 detects a product from the contour of the object.

The similarity calculation unit 92 reads the state of the surface such as the hue of the product and the unevenness of the surface from the product image captured by the camera 27 as a feature amount. The similarity calculation unit 92 does not consider the contour or size of the product. Thereby, the similarity calculation part 92 can shorten processing time.
Further, the similarity calculation unit 92 reads the surface state such as the color of the product handled and the surface irregularities from the product image of each product (hereinafter referred to as “handled product”) recorded in the feature file 361 as a feature value. The similarity between the photographed product and the handled product recorded in the feature file 361 is calculated by comparing the feature quantity with the photographed product. Here, the similarity refers to the degree of similarity of all or some of the products when the product image assumed by each product recorded in the feature file 361 is assumed to be 100% similarity. Is shown. For example, the similarity calculation unit 92 may calculate the similarity by changing the weighting between the hue and the unevenness state of the surface.

Recognizing an object included in an image in this way is called generic object recognition. In "Current status and future of general object recognition" written by Keiji Yanai, a survey of general object recognition research is added to methods, data sets and evaluation benchmarks are performed, and the future is prospected.
Keiji Yanai, “Current Status and Future of General Object Recognition”, [online], Journal of Information Processing Society of Japan, November 15, 2007, Vol. 48, No. SIG16, pp. 1-24, [September 8, 2014 Day search], Internet <URL: http: //mm.cs.uec.ac.jp/IPSJ-TCVIM-Yanai.pdf>

A technique for performing general object recognition by dividing an image into regions for each object is described in the following document.
Jamie Shotton et al., “Semantic Texton Forests for Image Categorization and Segmentation”, Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, [Search September 8, 2014], Internet <URL: http: // citeseerx .ist.psu.edu / viewdoc / download? doi = 10.1.1.145.3036 & rep = rep1 & type = pdf>

  It should be noted that the method for calculating the degree of similarity between the captured product image and the product image of the handled product recorded in the feature file 361 is not particularly limited. For example, the similarity between the captured product image and each handled product recorded in the feature file 361 may be calculated as an absolute evaluation or may be calculated as a relative evaluation.

  When calculating the similarity as an absolute evaluation, the captured product image is compared with each handling product recorded in the feature file 361 on a one-to-one basis, and the similarity derived as a result of this comparison is used as it is. do it. Moreover, when calculating a similarity as a relative evaluation, it calculates so that the sum total of the similarity with each handling product may be 1.0 (100%). For example, it is assumed that four products # 1 to # 4 are recorded in the feature amount file 361. The product imaged at this time has, for example, a similarity of 0.65 for product # 1, a similarity of 0.2 for product # 2, and a similarity of 0.1 for product # 3. For the product # 4, the similarity is calculated as 0.05.

  The similarity determination unit 93 compares the degree of similarity between the product image and the product image recorded in the feature amount file 361 for each frame image captured by the image acquisition unit 90. In the present embodiment, a plurality of conditions are provided for the similarity between the product image of the handled product and the photographed image of the product. The similarity determination unit 93 selects the confirmation of the handled product or the navigation (guidance) of the product recognition according to the satisfied conditions. Although the conditions regarding the similarity are not particularly limited, the case where the conditions M and N are used will be described below.

  Here, the condition M is a condition for uniquely determining an object on the frame image as a handling product recorded in the feature amount file 361. The condition N is a condition for extracting a plurality of candidates for handled products recorded in the feature file 361 from objects on the frame image.

  For example, the similarity determination unit 93 determines that a handled product that satisfies the condition M is a product that has a one-to-one correspondence with an object on the frame image. Further, the similarity determination unit 93 determines that the handled product satisfying the condition N is one of a plurality of candidates for the product captured by the camera 27. And the similarity determination part 93 extracts the candidate of the handling goods with respect to the image | photographed goods by extracting the handling goods which satisfy | fill the conditions N from the several handling goods recorded on the feature-value file 361.

  The conditions M and N are not particularly limited as long as the conditions M and N are set stepwise according to the degree of similarity. For example, the conditions M and N can be set with a plurality of preset threshold values. Here, the case where the conditions M and N are set by the threshold value T and the predetermined value D will be described.

  The similarity determination unit 93 is configured such that the similarity of the product having the highest similarity with each handled product is equal to or greater than the threshold T, and the difference between the similarity of the first product and the similarity of the second product Is equal to or greater than a predetermined value D, it is determined that the condition M is satisfied. The condition M is a condition for uniquely specifying a product.

  Furthermore, the similarity determination unit 93 is configured such that the similarity of the product having the highest similarity with each handled product is greater than or equal to the threshold T, and the similarity of the first product and the similarity of the second product If the difference is less than the predetermined value D, it is determined that the condition N is satisfied. Condition N is a condition for a plurality of candidate products. And the handling goods which satisfy | filled the condition N guide | invade rotation of goods to an operator, and it determines with the confirmation of another surface being required.

  Each condition M and N can be set as appropriate, and is not limited to the above example.

  The guiding unit 94 causes the display 221 to display information for guiding the handled product that satisfies the condition N to move. More specifically, the guide unit 94 reads out the photographic images in each direction of the candidate products that satisfy the condition N from the feature amount file 361, and sets the handling products in the direction in which the similarity with the candidate products is likely to increase. Prompt to move. For example, when the bamboo leaf portion is photographed, the guiding unit 94 causes the display 221 to display an arrow that prompts the user to move to show the root portion that is a feature of the bamboo shoot.

For example, it is assumed that an object is recognized using an image in which the leaf portion of the cocoon is taken by the camera 27, and cocoon and radish are listed as candidates. Determine which part of the reference image is taken from the horizontal direction of the candidate candy and radish, and identify the root part where the difference in the reference image is large and the difference in appearance is significant . If the direction of the root portion is specified, the guiding unit 94 causes the display 221 to display an arrow that prompts the operator to move the product so that the root portion is held by the camera 27.
The guiding unit 94 stores such combination information of candidate products and information on the feature direction of these candidate products in the similar part database 362. Thereby, it is possible to navigate quickly without performing the calculation process for the same candidate under the same photographing condition.

  Even if an arrow prompting the operator to rotate the product is displayed on the display 221, an image acquisition process by the image acquisition unit 90, an object detection process by the object detection unit 91, and a similarity calculation unit 92 The similarity calculation processing by is continued.

  The confirmation notifying unit 95 notifies the operator or customer that the product captured by the camera 27 is uniquely confirmed as a handled product that satisfies the conditions, by image output or voice output.

  More specifically, the confirmation notifying unit 95 causes the display 221 to display a confirmation screen indicating that the handled product that satisfies the condition is uniquely confirmed as the product captured by the camera 27 (confirmed product). Information about the product is output to the speaker 28. The speaker 28 notifies the operator and the customer of information indicating the confirmed product by outputting the input information.

  The input acquisition unit 96 receives various input operations corresponding to the display on the display 221 via the touch panel 222 or the keyboard 23.

  The information output unit 97 outputs information indicating the product (for example, product ID, product name, discount information, etc.) to the POS terminal 3 via the interface 25 for the confirmed product confirmed as described above.

  The information output unit 97 may output the sales quantity separately input via the touch panel 222 or the keyboard 23 to the POS terminal 3 together with the product ID and the like. Further, as information output from the information output unit 97 to the POS terminal 3, the product ID read out from the feature file 361 by the information output unit 97 may be directly notified, or a product for which the product ID can be specified. The POS terminal 3 may be notified of the name, the product image, and the file name of the photo image.

  The sales registration unit 99 of the POS terminal 3 performs sales registration of the corresponding product based on the product ID and the sales quantity output from the information output unit 97. More specifically, the sales registration unit 99 refers to the feature amount file 361 and records the notified product ID and the corresponding product classification, product name, unit price, and sales quantity in a sales master file or the like. Sales registration (temporary registration).

FIG. 4 is a front view of the product identification device 2 in the present embodiment.
The commodity identification device 2 is installed on the counter table 5. The product identification device 2 includes a thin rectangular parallelepiped housing 51, a display / operation unit 22 attached to the upper part of the housing 51, and a customer display installed face down on the left back side of the display / operation unit 22. 24.
A reading window 52 is provided on the front surface of the housing 51.
The reading window 52 is provided with an illumination 271 and a camera 27 (imaging means). The recognition area 8M is an area where the camera 27 captures an object, detects the object, and specifies a product.
The product identification device 2 detects an object by the camera 27, identifies the product name of the object, and causes the POS terminal 3 to perform product registration processing.

FIG. 5 is a diagram showing the configuration of the feature amount file 361.
Each record of the feature amount file 361 includes a feature amount column 361a, an image link column 361b, a product ID column 361c, a unit price column 361d, and a product name column 361e. The feature amount file 361 stores a combination of a reference image obtained by shooting each product from a plurality of directions and information on the shooting direction.
The feature amount column 361a is a column for storing the feature amount vector of the reference image of the product. The stored information in the feature amount column 361a is obtained by analyzing an image in the image link column 361b described later.
The image link column 361b is a column for storing the link of the reference image of this product. The reference image indicated by the link is an image obtained by photographing the product from any one of six directions.
The product ID column 361d is a column for storing information for identifying the product.
The unit price column 361e is a column for storing the unit price of this product.
The product name column 361f is a column for storing the name of this product.

FIGS. 6A and 6B are diagrams showing reference images of buds and radishes.
FIG. 6A is a diagram illustrating a reference image of wrinkles. Here, an image obtained by photographing the ridge from the side is used as a reference image.
FIG. 6B is a diagram illustrating a reference image of radish. Similarly, an image obtained by photographing a radish from the side is used as a reference image.
These buds and radishes are Brassicaceae, and the leaves are similar. For this reason, it is difficult to specify only the leaf portion. Both can be identified by the appearance of these root portions. That is, the merchandise item identification device 2 can easily identify the cocoon and the radish by guiding the cocoon or radish root part to the camera 27.

FIG. 7 is a diagram illustrating a configuration of the similar part database 362 according to the first embodiment.
The similar part database 362 includes a first product ID field 362a, a first similar part field 362b, a second product ID field 362c, a second similar part field 362d, and a characteristic part field 362e. The similar part database 362 is a learning result of past product identification. By referring to the similar part database 362, the object recognition calculation process can be skipped and processed at high speed.
The first product ID column 362a is a column that stores one product ID of two similar candidate products, and the second product ID column 362c is a column that stores the other product ID.
The first similar part field 362b is a field for storing information on similar parts of the reference image of one product among two similar candidate products, and the second similar part field 362d is a reference image of the other product. This is a column for storing information on similar parts.
The feature part column 362e is a column for storing information on parts where the similarity between the two candidate products becomes the smallest among the reference images of one of the two similar candidate products, and thus the difference in features becomes remarkable. is there. By converting this information into position information on the captured image, an arrow on the screen for guiding the movement can be displayed.

FIG. 8 is a flowchart showing a provisional product registration process according to this embodiment.
In the provisional product registration process according to the present embodiment, for example, an operator (clerk) takes out a bag which is a product of the first shopping basket 4L (see FIG. 1) and puts it on the camera 27 of the reading window 52. It is a series of processes up to temporary registration of candy.

In step S <b> 11, the image acquisition unit 90 outputs an imaging on signal to the camera 27 and starts capturing (imaging) a product image by the camera 27. The image acquisition unit 90 captures a frame image (captured image) captured by the camera 27 and stored in the RAM 213.
In step S12, the object detection unit 91 performs object recognition processing on the frame image captured by the image acquisition unit 90, and tries to recognize (detect) all or part of the object that is the product.

  In step S <b> 13, the object detection unit 91 determines whether or not all or part of the object that is the product has been successfully recognized. If the object detection unit 91 succeeds in recognizing the object that is the product (Yes), the process proceeds to step S14. If the object detection unit 91 does not succeed in recognizing the object that is the product (No), the object detection unit 91 proceeds to the process in step S11. Return. Specifically, Steps S11 to S13 are a series of processes in which the operator puts a product on the camera 27 of the reading window 52 and the product identification device 2 succeeds in detecting (recognizing) an object that is the product.

In step S14, the similarity calculation unit 92 reads the feature amount of the product from all or part of the product image. Further, the similarity determination unit 93 determines whether or not there is a handling product whose similarity is equal to or greater than the threshold value T in the feature amount file 361, and searches and calls the product name and the product unit price of the product.
In step S <b> 15, the similarity determination unit 93 determines whether or not the product is uniquely specified based on whether or not there is a handled product in the feature amount file 361 whose similarity is equal to or greater than the threshold T. The similarity determination unit 93 proceeds to the process of step S22 if the handled product whose similarity is equal to or greater than the threshold T is specified, and proceeds to the process of step S16 if there are a plurality of candidate products. If no candidate product exists, the process returns to step S11. Specifically, the similarity determination unit 93 determines that strawberry and radish exist as candidate products. Further, the similarity determination unit 93 determines that this product is most similar to the bamboo leaf portion and then to the radish leaf portion.

The processing from step S16 to step S22 is a series of processing relating to the parallel movement navigation of the product. This process is repeated when the recognized object matches a plurality of candidate product portions.
In step S <b> 16, the guiding unit 94 determines whether or not the recognized object matches the reference image portions of a plurality of candidate products. If the guide unit 94 matches the reference image portion (Yes), the process proceeds to step S17. If the guide unit 94 matches the entire reference image (Yes), the guide unit 94 returns to step S11.

In step S <b> 17, the guiding unit 94 determines whether information on similar parts of the candidate product exists in the similar part database 362. That is, the guide unit 94 searches the similar part database 362 for a combination of the first and second candidate products having high similarity and the parts that recognized them, and determines whether there is a corresponding record. The presence of similar part information of the product is determined. If there is information on the similar part of the candidate product in the similar part database 362 (Yes), the guiding unit 94 proceeds to the process of step S20, and if there is no information on the similar part of the candidate product (No), The process proceeds to S18.
In step S <b> 18, the guiding unit 94 identifies feature parts that are significantly different from each other for the two candidate products. More specifically, with respect to the reference images of buds and radishes, each of the whole root portions is a characteristic part where the difference is significant, and the similarity is minimum.
In step S <b> 19, the guide unit 94 records a site where the candidate product is similar and a feature site where the difference is significant in the similar site database 362. This makes it possible to omit similar calculation processing from the next time on.
In step S20, the guide unit 94 converts the detected part into a display coordinate system.
In step S <b> 21, the guiding unit 94 calculates a movement vector so that the detected part is held by the camera 27. Here, the direction of the movement vector is a direction opposite to the direction of the detection portion with respect to the center of the screen.

  In step S <b> 22, the guide unit 94 navigates the parallel movement of the product that the operator is hesitant by displaying an arrow on the screen. As a result, the guide unit 94 can guide the product identification device 2 to identify the product without hindering the flow of product registration by the operator. When the process of step S22 ends, the process returns to step S11.

The processing from step S23 to step S24 is processing when a product is uniquely specified.
In step S <b> 23, the confirmation notification unit 95 displays the product name and unit price together with the product image in a window, and outputs information about the confirmed product to the speaker 28. As a result, the confirmation notification unit 95 can notify the operator that the product has been uniquely identified.
In step S24, the information output unit 97 outputs information indicating the product (for example, product ID, product name, discount information, etc.) to the POS terminal 3 via the interface 25 for the confirmed product, and the processing of FIG. Exit. The POS terminal 3 temporarily registers the product and registers it in the product master file together with an instruction for payment.

FIGS. 9A and 9B are screens showing a first operation example.
FIG. 9A shows a screen when the bamboo leaf portion is placed on the camera 27.
At this time, an arrow 71 is displayed on the screen for urging the parallel movement so that the camera root is held by the camera root. The operator translates the heel according to the arrow 71 and places the root portion on the camera 27. As a result, the screen transitions to the screen of FIG.

FIG. 9B shows a screen when the entire camera including the heel root portion is obscured by the camera 27.
At this time, the entire bag is displayed on the screen, and the unit price of the bag is displayed in the window 72. Thus, the operator can know that the product has been confirmed.

10A and 10B are screens showing a second operation example.
FIG. 10A shows a screen when the tip of the radish root is put on the camera 27.
At this time, an arrow 71 is displayed on the screen for urging the translation so that the entire portion of the radish root is held by the camera 27. The operator translates the radish according to the arrow 71 and places the entire root portion on the camera 27. As a result, the screen transitions to the screen of FIG.
FIG. 10B shows a screen when the entire camera including the root portion of the radish is put on the camera 27.
At this time, the entire radish is displayed on the screen, and the unit price of the radish is displayed in the window 72. Thus, the operator can know that the product has been confirmed.

The present invention is not limited to the above-described embodiment, and can be modified without departing from the spirit of the present invention. For example, there are the following (a) and (b).
(A) Product navigation is not limited to the above embodiment. For example, the movement of a product may be navigated by voice or polygon movement animation.
(B) The number of candidate products is not limited to two, and the direction in which the degree of similarity is minimized may be determined from three or more candidate products.

The invention described in the scope of claims attached to the application of this application will be added below. The item numbers of the claims described in the appendix are as set forth in the claims attached to the application of this application.
[Appendix]
<Claim 1>
Photographing means for photographing an image of the object that the operator has deceived;
Similarity determination means for determining whether the object photographed by the photographing means is similar to a plurality of products;
If it is determined by the similarity determination means that the part of the object photographed by the photographing means is similar to a plurality of products, a portion where the difference is remarkable is identified from the reference images of the plurality of products, and the identified Guiding means for guiding the movement of the object to the operator so as to photograph a portion with the photographing means;
A product identification device comprising:
<Claim 2>
Display means for displaying an image taken by the photographing means and displaying an arrow in a direction for guiding the movement of the object if the portion of the object photographed by the photographing means is similar to a plurality of products;
The product identification device according to claim 1, comprising:
<Claim 3>
The guiding means is specified if the part of the object recognized from the image taken by the photographing means is similar to a plurality of products, and a portion where the difference is remarkable among the reference images of the plurality of products is specified. Information on similar parts of the reference image and information on parts where the difference is significant are stored in the storage means.
The product identification device according to claim 1, wherein the product identification device is a product identification device.
<Claim 4>
A shooting step for shooting an image of an object that the operator has deceived;
A similarity determination step for determining whether or not the object photographed by the photographing means is similar to a plurality of products;
If it is determined by the similarity determination means that the part of the object photographed by the photographing means is similar to a plurality of products, a portion where the difference is remarkable is identified from the reference images of the plurality of products, and the identified A guiding step for guiding the movement of the object to the operator so as to photograph a part in the photographing step;
A product recognition navigation method for a product identification device, comprising:

1 Product Registration Device 2 Product Identification Device 21 Microcomputer (Control Unit)
211 CPU
212 ROM
213 RAM
22 Display / Operation Unit 221 Display (Display means)
222 Touch Panel 23 Keyboard 24 Customer Display 25 Interface 27 Camera (Photographing Means)
28 Speaker 29 Power supply 3 POS terminal 36 HDD
361 Feature file 362 Similar part database 90 Image acquisition unit 91 Object detection unit (object detection means)
92 similarity calculation unit 93 similarity determination unit (similarity determination means)
94 Guiding part (guidance means)
95 Confirmation notification part 96 Input acquisition part 97 Information output part 99 Sales registration part

Claims (5)

An imaging control means for acquiring an image of the object by imaging an object dressed by the operator with the camera ;
Similarity determination means for determining whether the object is similar to a plurality of products based on the image of the object acquired by the imaging control means ;
When the similarity determination unit determines that the object is similar to a plurality of products, the image of the object is such that any part of the entire image of the object deviates from the angle of view of the camera. A specifying means for specifying a portion in which a difference is noticeable among a plurality of products determined to be similar to the object in the case of an image captured and acquired ;
Guiding means for guiding the movement of the object to the operator so that a portion specified by the specifying means is within an angle of view of the camera ;
A product identification device comprising: The guiding unit displays an arrow in a direction for guiding the movement of the object on a display screen on which an image of the object acquired by the photographing control unit is displayed .
The merchandise item identification device according to claim 1. The guidance means accumulates information of the identified part in a storage means when a part that is noticeably different between a plurality of products is identified by the identification means.
The product identification device according to claim 1, wherein the product identification device is a product identification device. An imaging control step for acquiring an image of the object by imaging an object dressed by the operator with the camera ;
A similarity determination step of determining whether or not the object is similar to a plurality of products based on the image of the object acquired in the shooting control step ;
In the case where it is determined in the similarity determination step that the object is similar to a plurality of products, the image of the object is such that any part of the entire image of the object deviates from the angle of view of the camera. A specific step of identifying a portion where the difference is noticeable among a plurality of products determined to be similar to the object when the image was acquired by being photographed ;
A guiding step for guiding the movement of the object to the operator so that the portion specified in the specifying step is within an angle of view of the camera ;
A product recognition navigation method for a product identification device, comprising: Computer
An imaging control means for acquiring an image of the object by imaging an object dressed by the operator with the camera,
Similarity determination means for determining whether the object is similar to a plurality of products based on an image of the object acquired by the imaging control means;
When the similarity determination means determines that the object is similar to a plurality of products, the image of the object is such that any part of the overall image of the object deviates from the angle of view of the camera. A specifying means for specifying a portion where a difference is noticeable among a plurality of products determined to be similar to the object when the image is acquired by being photographed;
Guidance means for guiding the movement of the object to the operator so that the part identified by the identification means falls within the angle of view of the camera;
A program characterized by functioning as

Images