JP6295944B2 - Product identification device and product recognition navigation method - Google Patents

Description

  The present invention relates to a product identification device and a product recognition navigation method for identifying a product to be registered for sales.

  2. Description of the Related Art Conventionally, an electronic cash register (product registration device) includes a product identification device that identifies each product, and a POS terminal that performs sales registration and settlement of products related to each transaction. The merchandise item identification device reads a data code such as a bar code attached to the merchandise item by the bar code reading unit to identify the merchandise item. The bar code assigned to the product includes maker code information, item code information, and check digit information. The product identification device identifies the product based on the item code information after determining that the barcode is correctly read based on the check digit information. By printing the barcode on the product packaging in advance, the burden on the cash register operator can be reduced.

  However, it is extremely troublesome to individually attach barcodes to unpackaged products, such as fruits and vegetables, which takes time and cost. For this reason, conventionally, for fruits and vegetables, an operation has been performed in which a product code is selected from a button indicating a product, a touch on a screen, or a bar code table prepared separately and read by a scanner.

  2. Description of the Related Art In recent years, product registration apparatuses that have a product identification device that recognizes and identifies fruits and vegetables and the like, and that register sales of products identified by the product identification device have begun to spread (for example, see Patent Document 1). The product identification device of this product registration device extracts the feature amount of the appearance from the image information obtained by photographing the fruits and vegetables, and compares it with the feature amount database related to the appearance of the product to determine which product the product is. Identify Since this product identification device can identify a product without attaching a barcode individually even if the product is not packaged in advance, the burden on the store side can be reduced.

JP2013-182323A (FIG. 8 and paragraph 0068)

  However, as described below, the conventional product identification device of the product registration device has a problem that it is desired to improve the accuracy of narrowing down candidates for the product to be traded.

  For example, when a customer purchases a product in a supermarket, after placing the product in a shopping cart, the customer takes the shopping cart to a cash register, passes the shopping cart to the cash register operator, and settles the price for the product. Ask.

  At this time, the operator reads each product in the shopping cart brought by the customer (hereinafter referred to as “first shopping cart”) in order into the product identification device of the product registration device, thereby the product. Each item in the first shopping basket by repeatedly registering the product information in the sales information and placing the product in another shopping basket (hereinafter referred to as a “second shopping basket”). Register the product sales.

  At that time, the product identification device may not be able to recognize the product immediately. For example, if the barcode portion of the barcode-added product cannot be photographed well, the product identification device cannot immediately recognize the product. In this case, the operator causes the merchandise item identification apparatus to recognize the merchandise object by an operation of bringing the merchandise item as close as possible to the camera of the merchandise item identification device.

  At this time, the product identification device determines the similarity between the captured image of the object acquired by the camera and the reference image representing the appearance of each product recorded in advance in the storage unit. Then, when the product identification device can uniquely narrow down the product candidates, the product registration device temporarily registers the product candidates as the products to be traded. On the other hand, the product identification device displays a confirmation screen for the target product on the display unit or displays a selection screen for the target product on the display unit when the product candidates cannot be uniquely narrowed down. It was.

  Here, the “target product confirmation screen” refers to, for example, one product having a relatively high degree of similarity to an object shown in a captured image as a target product candidate, and allows the operator to check whether the candidate is correct. If it is correct, the candidate is temporarily registered as a target product, and if not correct, it means a screen configured to search for another candidate. In addition, the “target product selection screen” refers to, for example, a plurality of products having a relatively high degree of similarity to an object shown in a captured image as target product candidates, and allows the operator to select a target product from the plurality of candidates. Means the screen that is configured.

  However, when a screen such as a confirmation screen or a selection screen for the target product is displayed, the operator needs to perform some information input operation. Therefore, the operator needs time and labor for input operation. Therefore, it is preferable for the product identification device to suppress the display of screens such as confirmation screens and selection screens for target products as much as possible, and to narrow down the candidates for target products for transactions so that display of these screens is suppressed It was desired to improve.

  Therefore, an object of the present invention is to provide a product identification device and a product recognition navigation method that improve the accuracy of narrowing down product candidates.

Claim 1 includes a reference image representing the appearance of the products handled in each Products, and guidance information to guide the shooting of the specific portion to accurately identify the products handled within the appearance of the products handled, a The management means managed by the storage means , and the product picked up by the operator is photographed, a captured image of the product is acquired by the photographing means , and the captured product is recognized and recognized for each handled product stored and managed in the storage means . acquires the narrowing means to narrow the candidates of pulling Shipping up by comparison of the reference image, the guidance information stored in said storage means in response to the transaction Shipping narrowed by the narrowing section, the induction Based on the information, which direction of the product appearance in the appearance of the product to be traded is given to the photographing means, and which specific part of the product appearance in the direction is indicated to the photographing means, to induce the induction image It characterized in that it and a guidance display means for displaying.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to provide the goods identification apparatus and goods recognition navigation method which improve the narrowing-down precision of the candidate of the goods for transaction.

It is a perspective view which shows the external appearance of the goods registration apparatus containing the goods identification apparatus which concerns on embodiment. It is a block diagram which shows the outline of the goods registration apparatus containing the goods identification apparatus which concerns on embodiment. It is a functional block diagram which shows the outline of the goods registration apparatus containing the goods identification device which concerns on embodiment. It is a front view of the goods identification device concerning an embodiment. It is a figure which shows the structure of the feature-value file used by embodiment. It is a figure which shows an example of the reference | standard image of the stuffed rabbit, the image for extraction, and an auxiliary image. It is a figure which shows an example of the reference | standard image of the stuffed bear, the image for extraction, and an auxiliary image. It is a flowchart which shows operation | movement of the goods registration apparatus containing the goods identification apparatus which concerns on embodiment. It is explanatory drawing which shows an example of a display screen.

  Hereinafter, an embodiment of the present invention (hereinafter referred to as “the present embodiment”) will be described in detail with reference to the drawings. Each figure is only schematically shown so that the present invention can be fully understood. Therefore, the present invention is not limited to the illustrated example. Moreover, in each figure, the same code | symbol is attached | subjected about the common component and the same component, and those overlapping description is abbreviate | omitted.

[Embodiment]
The merchandise item identification device according to the present embodiment stores auxiliary information in advance in the storage means that indicates the characteristic part of the handled merchandise that is displayed and sold in the store. Specifically, the product identification device uses a region having a relatively large feature amount of the handled product (hereinafter referred to as a “high feature region”) as a feature portion of the handled product, and an image including the high feature region (feature portion) as an object. The auxiliary image is used as an auxiliary image at the time of recognition, and the auxiliary image is stored in advance in the storage unit as auxiliary information.

  The product identification device performs object recognition of an object (product) based on a reference image, and when candidates for a target product for transaction are narrowed down under a predetermined condition (for example, similarity to an object) When the candidate for the target product is narrowed down to one), the high feature region (feature portion) of the object is imaged to the operator based on the auxiliary information corresponding to the narrowed candidate. The guidance is reminiscent of. As a result, the product identification device acquires a captured image including the high feature area, and uniquely narrows down target product candidates based on the similarity between the acquired captured image and the auxiliary image. As a result, the product identification device can improve the accuracy of narrowing down target product candidates.

  In the present embodiment, description will be made assuming that the auxiliary information is auxiliary image information (hereinafter simply referred to as “auxiliary image”) indicating a characteristic part of the product. However, the auxiliary information may include an auxiliary image and message information indicating a characteristic part of the handled product.

<External configuration of product registration device>
Hereinafter, with reference to FIG. 1, an external configuration of the product registration device 1 including the product identification device 2 according to the embodiment will be described. FIG. 1 is a perspective view illustrating an appearance of a product registration apparatus 1 including a product identification device 2 according to the embodiment.

As shown in FIG. 1, the product registration device 1 includes a product identification device 2 that identifies each product, and a POS terminal 3 that performs sales registration and settlement of products related to each transaction.
The merchandise item identification device 2 is a POS connection scanner that reads and registers information about merchandise items. The merchandise item identification device 2 is installed at the center in the longitudinal direction of the counter table 5 having a horizontally long table shape. The commodity identification device 2 includes a thin rectangular parallelepiped housing 51. A camera 27 (see FIG. 2) is arranged on the front surface of the housing 51 through a reading window 52.

  A display / operation unit 22 is attached to the upper portion of the housing 51. The display / operation unit 22 is provided with a display 221. A touch panel 222 (see FIG. 2) is laminated on the surface of the display 221. A keyboard 23 is arranged on the right side of the display 221. A card reading groove of a card reader (not shown) is provided on the right side of the keyboard 23. On the left back side of the display / operation unit 22, a customer display 24 for providing information to the customer is installed face down.

The POS terminal 3 is placed on the upper surface of the drawer 37 on the checkout table 6. The drawer 37 is a cash storage unit, and is controlled by the POS terminal 3 for an opening operation.
On the upper surface of the POS terminal 3, a keyboard 33 for operation by an operator (clerk) is arranged. A display 321 for displaying information is provided on the upper back side of the keyboard 33 as viewed from the operator. A touch panel 322 (see FIG. 2) is stacked on the surface of the display 321. Further on the back side of the display 321, a customer display 34 for displaying information is installed so as to be rotatable in the left-right direction. The customer display 34 shown in FIG. 1 faces the front side of the figure. This customer display 34 displays information toward the customer by rotating it toward the far side of the figure.

  A horizontally long table-like counter table 5 is arranged so as to form an L shape with the checkout table 6 on which the POS terminal 3 is placed. A load receiving surface is formed on the upper surface of the counter table 5. The customer undergoes a product registration process while moving from the left side to the right side in FIG. That is, the first shopping basket 4L and the second shopping basket 4R for storing products are placed on the load receiving surface of the counter table 5. Hereinafter, when there is no particular distinction between the first shopping basket 4L and the second shopping basket 4R, the shopping basket 4 may be simply described. These shopping baskets 4 are not limited to the so-called basket shape, and may be trays or the like. The shopping basket 4 is not limited to a so-called basket shape, and may be a box shape or a bag shape.

  The first shopping basket 4L is brought in by a customer and stores products related to one transaction. The second shopping basket 4R is placed at a position sandwiching the merchandise item identification device 2 from the first shopping basket 4L. The product in the first shopping basket 4L is taken out by an operator who operates the product identification device 2 and moved to the second shopping basket 4R. In this movement process, the product is held on the reading window 52 of the product identification device 2. At this time, the camera 27 (see FIG. 2) disposed in the reading window 52 captures the product and acquires a captured image of the product.

  The product identification device 2 identifies which product (handling product) recorded in a feature value file 361 (see FIG. 2) described later corresponds to an object (product) included in a captured image acquired by the camera 27. And displayed on the display 221. The merchandise item identification device 2 further notifies the POS terminal 3 of the merchandise ID of the identified merchandise item. Based on the product ID notified from the product identification device 2, the POS terminal 3 stores product information related to sales registration such as product classification, product name, and unit price of the product corresponding to the product ID in a sales master file (not shown). Record sales etc. and register sales.

<Internal configuration of product registration device>
Hereinafter, the internal configuration of the commodity registration apparatus 1 will be described with reference to FIG. FIG. 2 is a configuration diagram illustrating an outline of the commodity registration apparatus 1.
As shown in FIG. 2, the product registration device 1 includes a product identification device 2 and a POS terminal 3.
The product identification device 2 includes a microcomputer 21, a display / operation unit 22, an interface 25, a camera 27, a speaker 28, and a power source 29.
The microcomputer 21 is configured by connecting a ROM (Read Only Memory) 212 and a RAM (Random Access Memory) 213 to a CPU (Central Processing Unit) 211. The ROM 212 stores a program executed by the CPU 211.
The CPU 211 is connected to the display / operation unit 22, the interface 25, the camera 27, and the speaker 28 via an internal bus and input / output circuits (not shown).

The display / operation unit 22 includes a display 221, a touch panel 222, a customer display 24, and a keyboard 23, and the operation is controlled by the CPU 211.
The display 221 is a display unit that displays information for the operator according to an instruction from the CPU 211. The touch panel 222 receives an operation input for information displayed on the display 221. The customer display 24 displays information for the customer in accordance with an instruction from the CPU 211.
The keyboard 23 is composed of a plurality of operation keys and accepts operator input.
The interface 25 is connected to the interface 35 of the POS terminal 3 to enable data transmission / reception with the POS terminal 3.

  The camera 27 is a photographing unit including a color CCD image sensor, a color CMOS image sensor, and the like. The camera 27 shoots an object (product) that is trapped in the reading window 52 (see FIG. 1) under the control of the CPU 211. For example, the camera 27 captures a moving image of 30 [fps]. Frame images (captured images) sequentially captured by the camera 27 at a predetermined frame rate are stored in the RAM 213.

The speaker 28 is sound output means for generating a preset warning sound or the like. The speaker 28 performs warning sound or voice notification under the control of the CPU 211.
The power source 29 supplies power to each part of the product identification device 2.
The interface 25 is connected to the interface 35 of the POS terminal 3 to enable data transmission / reception with the POS terminal 3.

The POS terminal 3 includes a microcomputer 31, a display 321, a touch panel 322, a keyboard 33, a customer display 34, an interface 35, an HDD 36, a drawer 37, a printer 38, and a power supply 39. Is done.
The microcomputer 31 executes information processing. The microcomputer 31 is configured by connecting a ROM 312 and a RAM 313 to a CPU 311 that executes various arithmetic processes and controls each unit. A drawer 37, a keyboard 33, a display 321, a touch panel 322, a customer display 34, and an HDD 36 (Hard Disk Drive) are connected to the CPU 311 via an internal bus and input / output circuits. These are controlled by the CPU 311.

The display 321 displays information for the operator according to an instruction from the CPU 311. The touch panel 322 receives an operation input for information displayed on the display 321. The customer display 34 displays information for the customer in accordance with an instruction from the CPU 311.
The keyboard 33 includes a temporary fastening key 331, a fastening key 332, and a numeric keypad 333, and accepts an operator's operation input. The numeric keypad 333 includes numeric keys from 0 to 9 and various operator keys.

  The HDD 36 is a storage unit that stores programs and various files. All or a part of the programs and various files stored in the HDD 36 are copied to the RAM 313 and executed by the CPU 311 when the POS terminal 3 is activated. The HDD 36 stores, for example, a product sales data processing program, a feature amount file 361, and the like.

  The feature file 361 is a product file in which an association between information related to sales registration of a handling product and an image representing the appearance of the handling product is set for each handling product displayed and sold in a store. It functions as a dictionary. The “image representing the appearance of the handled product” is used as a reference for calculating the degree of similarity between the object and the handled product when performing object recognition of the object shown in the frame image (captured image) captured by the camera 27. It is done. Hereinafter, an image representing the appearance of the handled product is referred to as a “reference image”.

The interface 35 is connected to the product identification device 2 and enables data transmission / reception with the product identification device 2.
The printer 38 prints on a receipt or the like. The POS terminal 3 prints transaction details of each transaction on a receipt under the control of the CPU 311.
The power supply 39 supplies power to each part of the POS terminal 3.

FIG. 3 is a functional block diagram showing an outline of the product registration apparatus. In the following description, FIGS. 1 and 2 will be referred to as appropriate.
As illustrated in FIG. 3, the CPU 211 (see FIG. 2) of the product identification device 2 executes a program stored in the ROM 212 (see FIG. 2), thereby causing the image acquisition unit 90, the object detection unit 91, and the similarity to be similar. The calculation unit 92, the narrowing-down unit 93, the guidance unit 94, the confirmation notification unit 95, the input acquisition unit 96, and the information output unit 97 are embodied. Similarly, the CPU 311 (see FIG. 2) of the POS terminal 3 implements the sales registration unit 99 by executing a program stored in the HDD 36 (see FIG. 2). The HDD 36 (see FIG. 2) of the POS terminal 3 further stores a feature amount file 361.

  The image acquisition unit 90 outputs an imaging on signal to the camera 27 and causes the camera 27 to start an imaging operation. Further, the image acquisition unit 90 sequentially captures frame images (captured images) captured by the camera 27 and stored in the RAM 213. The frame image is captured by the image acquisition unit 90 in the order stored in the RAM 213.

The object detection unit 91 detects all or part of the object included in the frame image captured by the image acquisition unit 90 using a pattern matching technique or the like.
Specifically, when the operator points the product to the reading window 52 for sales registration, the image acquisition unit 90 captures the product with the camera 27 and acquires a frame image. The object detection unit 91 binarizes the captured frame image and extracts a contour line. Next, the object detection unit 91 detects (recognizes) the object that is the product by comparing the contour line extracted from the previous frame image with the contour line extracted from the current frame image.

  Another specific method is shown below. When the operator grips the product with his / her hand and points it toward the reading window 52 for sales registration, the image acquisition unit 90 takes an image of the product and the hand with the camera 27. The object detection unit 91 detects the presence or absence of a skin color region from the captured frame image. When the skin color area is detected, that is, when the operator's hand is detected, the object detection unit 91 detects a contour line in the vicinity of the skin color area. As a result, the contour of the product that the operator's hand is supposed to hold is extracted. When the contour indicating the shape of the hand is detected and the contour of another object is further detected in the vicinity of the contour of the hand, the object detection unit 91 detects a product from the contour of the object.

The similarity calculation unit 92 reads the surface state such as the hue of the object (product) and the surface unevenness from the frame image (captured image) of the object (product) captured by the camera 27 as a feature amount. At this time, the similarity calculation unit 92 does not consider the contour or size of the object (product). Thereby, the similarity calculation part 92 can shorten processing time.
Further, the similarity calculation unit 92 reads the surface state such as the hue of the handled product and the unevenness of the surface from the reference image of each handled product recorded in the feature file 361 as a feature amount, and obtains a frame image (imaging image). The feature amount of the object (product) included in the image) and the feature amount of each handled product included in the reference image are respectively compared, and the recorded object (product) and the feature amount file 361 are recorded. The degree of similarity with the handled product is calculated. Here, the “similarity” means that all or a part of an image of an object (product) is obtained when the reference image assumed by each handling product recorded in the feature amount file 361 is 100% similarity. It shows how similar to the reference image of the product being handled. For example, the similarity calculation unit 92 may calculate the similarity by changing the weighting between the hue and the unevenness state of the surface.

In this way, recognizing an object included in a frame image (captured image) is called general object recognition. In "Current status and future of general object recognition" written by Keiji Yanai, a survey of general object recognition research is added to methods, data sets and evaluation benchmarks are performed, and the future is prospected.
Keiji Yanai, “Present and Future of General Object Recognition”, [online], Transactions of Information Processing Society of Japan, November 15, 2007, Vol. 48, no. SIG16, pp. 1-24, [September 8, 2014 search], Internet <URL: http: // mm. cs. uec. ac. jp / IPSJ-TCVIM-Yanai. pdf>

A technique for performing general object recognition by dividing a frame image (captured image) into regions for each object is described in the following documents.
Jamie Shotten et al., “Semantic Textile Forests for Image Categorization and Segmentation”, Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, [Search September 8, 2014], Internet <URL: http: // siteseeerx. ist. psu. edu / viewdoc / down1oad? doi = 10.1.1.1145.3036 & rep = rep1 & type = pdf>

  Note that the method for calculating the similarity between the frame image (captured image) of the photographed object (product) and the reference image of the handled product recorded in the feature amount file 361 is not particularly limited. For example, the similarity between the captured image of the product and the reference image of each handled product recorded in the feature file 361 may be calculated as an absolute evaluation or may be calculated as a relative evaluation.

  When calculating the similarity as an absolute evaluation, the frame image (captured image) of the photographed object (product) and the reference image of each handled product recorded in the feature amount file 361 are compared on a one-to-one basis. The similarity derived as a result of this comparison may be used as it is. Moreover, when calculating similarity as relative evaluation, it calculates so that the sum total of the similarity with respect to each handling product may be 1.0 (100%). For example, it is assumed that four handling products # 1 to # 4 are recorded in the feature amount file 361. The object (product) photographed at this time has, for example, a similarity to the handling product # 1 of 0.65, a similarity to the handling product # 2, 0.2, and a similarity to the handling product # 3 of 0.1. The degree of similarity with respect to product # 4 is calculated such as 0.05.

  For each frame image (captured image) captured by the image acquisition unit 90, the narrowing-down unit 93 includes a frame image (captured image) of the object (product) and a reference image of each handled product recorded in the feature amount file 361. Are compared to search for target products to be traded, and narrow down target product candidates corresponding to the object (product) on the captured image.

  In the present embodiment, a plurality of conditions are provided for the similarity between the reference image of the handled product and the frame image (captured image) of the object (product). The narrowing-down unit 93 selects the confirmation of the handled product or the navigation (guidance) of the product recognition according to the satisfied conditions. Although the conditions regarding the similarity are not particularly limited, a case will be described below where conditions M and N that define threshold values SM and SN (where SM> SN) are used, respectively.

  Here, the condition M is a condition that defines a threshold value SM for uniquely narrowing down (determining) an object on the frame image (captured image) as one of the handled products recorded in the feature file 361. is there. The condition N is a threshold value for extracting (narrowing down) some target product candidates corresponding to the object on the frame image (captured image) from all the handled products recorded in the feature file 361. This is a condition that defines SN. The threshold values SM and SN can be appropriately set to arbitrary values according to operation.

  For example, the narrowing-down unit 93 uniquely identifies a handling product that satisfies the condition M (that is, a handling product whose similarity to the object exceeds the threshold value SM defined by the condition M) with respect to the object on the frame image (captured image). It is determined that the product is narrowed down to (confirmed). Then, the narrowing-down unit 93 extracts one handled product corresponding to the photographed object (product) by extracting the handled product satisfying the condition M from each handled product recorded in the feature file 361. Can be narrowed down as candidates.

  Further, the narrowing-down unit 93 does not satisfy the condition M but satisfies the condition N (ie, the similarity to the object is equal to or lower than the threshold value SM defined by the condition M, but the threshold value SN defined by the condition N). Is determined to be one of several candidates for the handled product corresponding to the object (product) photographed by the camera 27, not the confirmed product determined as the target product for the transaction. Then, the narrowing-down unit 93 extracts a handled product satisfying the condition N from each handled product recorded in the feature amount file 361, thereby selecting several candidates for the handled product corresponding to the photographed object (product). Can be narrowed down as candidates for the target product. At this time, for example, the narrowing-down unit 93 can narrow down one handled product having the highest similarity to the object as a candidate for the target product although it is equal to or less than the threshold value SM defined by the condition M.

  When the narrowing-down unit 93 narrows down candidates for the target product under the predetermined conditions M and N, the guiding unit 94 operates based on auxiliary information (such as auxiliary images and message information) corresponding to the narrowed-down candidates. The camera 27 is guided to make the high feature area (feature part) of the object appear.

  In the present embodiment, the auxiliary information is configured as an auxiliary image, and the guide unit 94 generates and generates a guide image GP (see FIG. 9B) based on the auxiliary image corresponding to the narrowed candidates. It is assumed that the induced image GP is displayed on the display 221.

  More specifically, the guidance unit 94 refers to the guidance time reference field 361g (see FIG. 5) of the feature value file 361 corresponding to the handling product having the highest degree of similarity with the object shown in the captured image. An image for extraction and an auxiliary image of handling products are acquired, and the acquired image for extraction and the auxiliary image are combined with a frame image (captured image) to create a guide image GP (see FIG. 9B). The created guide image GP is displayed on the display 221.

  Even if the guidance image GP is displayed on the display 221, the image acquisition process by the image acquisition unit 90, the object detection process by the object detection unit 91, and the similarity calculation process by the similarity calculation unit 92 are the same. Will continue.

  However, the guidance unit 94 displays the message information on the display 221 as text when, for example, message information such as “Take the tail of the stuffed animal to the camera” is included in the auxiliary information, Guidance may be performed. In such a case, the guiding unit 94 may perform guidance by outputting a sound corresponding to the message information from the speaker 28 (see FIG. 2).

  The confirmation notifying unit 95 confirms that the object (product) photographed by the camera 27 is a handled product that satisfies the conditions and is uniquely confirmed as a target product of the transaction by image output or voice output. Notify customers.

  More specifically, the confirmation notifying unit 95 uniquely determines a transaction target product as a confirmed product when a target product for the transaction that satisfies the condition is extracted as an object (product) captured by the camera 27. A confirmation screen indicating this is displayed on the display 221 and information related to the confirmed product is output to the speaker 28. The speaker 28 notifies the operator and the customer of information indicating the confirmed product by outputting the input information by voice.

The input acquisition unit 96 receives various input operations corresponding to the display on the display 221 via the touch panel 222 or the keyboard 23.
The information output unit 97 outputs information indicating the product (for example, product ID, product name, discount information, etc.) to the POS terminal 3 via the interface 25 for the confirmed product confirmed as described above.

  The information output unit 97 may output the sales quantity separately input via the touch panel 222 or the keyboard 23 to the POS terminal 3 together with the product ID and the like. Further, as information output from the information output unit 97 to the POS terminal 3, the product ID read out from the feature file 361 by the information output unit 97 may be directly notified, or a product for which the product ID can be specified. The POS terminal 3 may be notified of the name, or the file name of the reference image and the photo image.

  The sales registration unit 99 of the POS terminal 3 performs sales registration of the target product for the transaction based on the product ID and the number of sales output from the information output unit 97. Specifically, the sales registration unit 99 refers to the feature value file 361 and records the notified product ID, the corresponding product classification, product name, unit price, and sales quantity in a sales master file or the like. Then, sales registration (provisional registration) of the target product of the transaction is performed.

<Configuration around the display of the product identification device>
Hereinafter, the configuration around the display 221 of the product identification device 2 will be described with reference to FIG. FIG. 4 is a front view of the product identification device 2 according to the embodiment.

As shown in FIG. 4, the merchandise item identification device 2 is installed on the counter stand 5. The product identification device 2 includes a thin rectangular parallelepiped housing 51, a display / operation unit 22 attached to the upper part of the housing 51, and a customer display installed face down on the left back side of the display / operation unit 22. 24.
A reading window 52 is provided on the front surface of the housing 51.
In the reading window 52, an illumination 271 and a camera 27 (imaging means) are arranged. The recognition area 8M is an area in which the camera 27 images an object, detects the object, and identifies a product.
The product identification device 2 detects an object by the camera 27, identifies the product name of the object, and causes the POS terminal 3 to perform product registration processing.

<Outline of main information>
Hereinafter, an outline of main information used in the embodiment will be described with reference to FIGS. FIG. 5 is a diagram illustrating a configuration of the feature amount file 361 used in the embodiment. FIG. 6 is a diagram illustrating an example of a stuffed rabbit reference image, an extraction image, and an auxiliary image. FIG. 7 is a diagram illustrating an example of the reference image of the stuffed bear, the image for extraction, and the auxiliary image. Here, the product that the customer is trying to purchase (the product to be traded) is “stuffed rabbit”, and the product identification device 2 recognizes the object “stuffed rabbit” and, as a result, “stuffed rabbit” A description will be given assuming that “bear toy” is extracted as a candidate for a product to be traded. In the present embodiment, it is assumed that the shape of the tail part of the “rabbit plush” is different from the shape of the tail part of the “bear plush”, and the tail part is a high feature region.

  As shown in FIG. 5, each record of the feature value file 361 includes a feature value field 361a, an image link field 361b, a type field 361c, a product ID field 361d, a unit price field 361e, a product name field 361f, and a guidance time reference field 361g. Includes each column. The feature amount file 361 includes, for each product, a reference image of each product used for object recognition, an image for extraction for extracting a high feature region, an image such as an auxiliary image including the high feature region, A combination with the feature amount of the image is stored.

  The feature amount column 361a is a column for storing the feature amount of the image corresponding to this product. The stored information in the feature amount column 361a is obtained by analyzing an image in the image link column 361b described later.

  The image link column 361b is a column for storing link information of an image corresponding to this product. For example, in the example illustrated in FIG. 5, a reference image (N0001F.jpg), an extraction image (N0001B.jpg), and an auxiliary image (N0001SP.jpg) are handled for a handling product whose product ID is “N0001”. Stores link information to be associated. For the English letter attached to the end of the code representing each image, “F” means the front image of the handled product, “B” means the back image of the handled product, and “SP”. Means an enlarged image of the high feature area of the product.

  Here, the case where the front image of the handling product is a reference image will be described. However, depending on the products handled, an image in another direction may be the reference image. In addition, here, a description will be given assuming that the high feature area of the handled product is included in the back image of the handled product. However, there is a case where the high feature area of the handling product is included in an image in another direction (for example, a front image). If the front image includes a high feature area of the product to be handled, the front image becomes an extraction image, and link information of the front image is registered in the image link field 361b of the extraction image. .

  In the present embodiment, “auxiliary image” linked to the product ID of each handled product by link information is used as “auxiliary information”. Note that the feature amount file 361 may include, for example, message information such as “Put the tail of the stuffed animal on the camera” as “auxiliary information” corresponding to the product ID of each handled product.

  The type column 361c is a column for storing type information of the image linked in the image link column 361b (hereinafter referred to as “link image”). For example, in the example illustrated in FIG. 5, the type field 361 c includes “reference” indicating that the link image is a reference image, “extraction” indicating that the link image is an extraction image, and the link image is an auxiliary image. The three types of “auxiliary” type information indicating this are stored.

The product ID column 361d is a column for storing information for identifying the product.
The unit price column 361e is a column for storing the unit price of this product.
The product name column 361f is a column for storing the name of this product.

  The guidance reference column 361g stores reference information for designating an image (hereinafter referred to as “reference image”) that is referred to when the guidance image GP (see FIG. 9B) is created by the guidance unit 94. ing. For example, in the example illustrated in FIG. 5, an extraction image (N0001B.jpg) is designated for the reference image (N0001F.jpg) in the guidance reference column 361g of the handling product whose product ID is “N0001”. In addition, reference information for specifying an auxiliary image (N0001SP.jpg) for the extraction image (N0001B.jpg) is stored. These pieces of reference information are information indicating which part of the handled product the auxiliary image is.

  FIG. 6A shows a reference image (N0001F.jpg) in the front direction of “stuffed rabbit”, and FIG. 6B shows an image for extracting “stuffed rabbit” (N0001B.jpg). FIG. 6C shows an auxiliary image (N0001SP.jpg) of “stuffed rabbit”. The auxiliary image (N0001SP.jpg) shown in FIG. 6C is an image obtained by capturing the high feature region SP1 included in the extraction image (N0001B.jpg) shown in FIG.

  On the other hand, FIG. 7A shows a reference image (N0002F.jpg) in the front direction of “the stuffed bear”, and FIG. 7B shows an image for extracting “the stuffed bear” (N0002B). Further, FIG. 7C shows an auxiliary image (N0002SP.jpg) of “the stuffed bear”. The auxiliary image (N0002SP.jpg) shown in FIG. 7C is an image obtained by photographing the high feature area SP2 included in the extraction image (N0002B.jpg) shown in FIG.

  In general, for a stuffed animal, the reference image in the front direction is the direction most suitable for object recognition. The product identification device 2 uses the link information in the image link field 361b of the feature amount file 361 when performing object recognition of the object (here, “the stuffed rabbit” which the customer wants to purchase) reflected in the captured image. A product search is performed based on the designated reference image to narrow down the target products.

  In this narrowing down, for example, when the similarity to an object is equal to or less than a threshold value, but the candidate for the target product is narrowed down to one, the product identification device 2 performs the target based on the auxiliary image of the candidate narrowed down to one. Attempt to narrow down (determine) product candidates uniquely. Further, in this narrowing down, for example, when narrowing down to candidates of some handling products whose similarity to the object exceeds the threshold, the product identification device 2 performs the target based on the auxiliary image of the candidate having the highest similarity to the object. Attempt to narrow down (confirm) a product uniquely. Thereby, the merchandise item identification device 2 improves the accuracy of narrowing down target merchandise candidates.

<Operation of product registration device>
In the present embodiment, the merchandise item identification device 2 of the merchandise registration device 1 performs object recognition of an object (product) based on a reference image, and when target product candidates are narrowed down under a predetermined condition ( For example, when the similarity to an object is equal to or less than a threshold value but the product candidates are narrowed down to one), an attempt is made to uniquely narrow down the target product candidates by object recognition based on the auxiliary image. At that time, the merchandise item identification device 2 guides the operator so that the high feature area of the object is placed on the camera 27 based on the auxiliary information.

  Hereinafter, the operation of the product registration apparatus 1 will be described with reference to FIGS. FIG. 8 is a flowchart showing the operation of the merchandise registration apparatus 1. FIG. 9 is an explanatory diagram illustrating an example of a display screen displayed on the display 221 (see FIG. 1) of the product identification device 2.

  The merchandise registration apparatus 1 starts operating when an operator instructs the start of merchandise registration processing. In the product temporary registration process of the embodiment, for example, an operator (clerk) takes out a stuffed rabbit which is an article (product) contained in the first shopping basket 4L (see FIG. 1), and the camera of the reading window 52 27, a series of processes from temporarily registering the stuffed rabbit as a commodity to be traded and putting it in the second shopping basket 4R.

  As shown in FIG. 8, in step S <b> 11, the image acquisition unit 90 outputs an imaging ON signal to the camera 27 and starts capturing (imaging) a frame image (captured image) of the product by the camera 27. The image acquisition unit 90 captures a frame image (captured image) captured by the camera 27 and stored in the RAM 213.

  The merchandise item identification device 2 of the merchandise registration apparatus 1 displays the frame image (captured image) captured by the image acquisition unit 90 on the display 221 (see FIG. 1). Here, the case where the product identification device 2 displays the captured image CP (see FIG. 9A) on the display 221 will be described.

  In step S12, the object detection unit 91 performs object recognition processing on the frame image captured by the image acquisition unit 90, and tries to recognize (detect) all or part of the object that is the product. Thereby, the object detection part 91 performs an object detection process.

In step S <b> 13, the object detection unit 91 determines whether or not all or part of the object that is the product has been successfully recognized. When the recognition of the object that is the product is successful (Yes), the process proceeds to step S14. On the other hand, when the recognition of the object that is the product is not successful (No), the process returns to step S11.
Specifically, Steps S11 to S13 are a series of processes in which the operator puts a product on the camera 27 of the reading window 52 and the product identification device 2 succeeds in detecting (recognizing) an object that is the product.

  In step S14, the similarity calculation unit 92 reads the feature amount of the product from all or part of the product image. Further, the narrowing-down unit 93 determines whether or not there is a handling product in the feature amount file 361 whose similarity to the object exceeds the threshold value SM. Further, the narrowing-down unit 93 determines whether or not there is a handling product exceeding the threshold SN in the feature amount file 361 even though the similarity to the object is equal to or less than the threshold SM. Then, when there is one or a plurality of corresponding handled products, the narrowing-down unit 93 sets each handled product as a candidate for the target product, and searches for product information such as a candidate product ID, a product name, and a product unit price. Then call. Then, the similarity calculation unit 92 performs a narrowing process and attempts to narrow down candidates for the target product.

  In step S15, the narrowing-down unit 93 determines whether candidates have been narrowed down. If it is determined in step S15 that candidates have been narrowed down (Yes), the process proceeds to step S16. If it is determined that candidates have not been narrowed down (No), the process returns to step S11. .

  Here, “when it is determined that the candidates have been narrowed down (Yes)” means that a number of handled products whose similarity to the object exceeds the threshold SM can be extracted, and the similarity to the object is This means both the case where a number of products handled that are less than or equal to the threshold value SM but exceed the threshold value SN can be extracted.

  Further, “when it is determined that the candidates could not be narrowed down (No)” means a case where only handled products whose similarity to the object is equal to or less than the threshold SN can be extracted.

  When it is determined in step S15 that the candidates could not be narrowed down (No), in step S16, the guiding unit 94 is reflected in the captured image CP among the candidates for the target product extracted by the narrowing-down unit 93. Reference is made to the guidance reference column 361g (see FIG. 5) of the feature file 361 corresponding to the handling product having the highest degree of similarity with the object (here, “rabbit plush with product ID“ N0001 ”).

  Next, in step S <b> 17, the guiding unit 94 generates an image for extracting the handling product (N0001B.jpg (see FIG. 6B)) and an auxiliary image (N0001SP.jpg (see FIG. 6C)). get.

  In step S18, the guide unit 94 uses the acquired image for extraction (N0001B.jpg) and the auxiliary image (N0001SP.jpg) to obtain a frame image (captured image) in order to improve the target product candidate narrowing accuracy. ) A guide image GP (see FIG. 9B) is created by combining with CP, and the created guide image GP is displayed on the display 221 as product navigation. Thereby, the guidance part 94 performs a guidance process with respect to an operator.

  At this time, the guidance unit 94 displays, for example, message information such as “Please put the tail of the stuffed animal on the camera” on the display 221 as the navigation of the product, or the voice corresponding to the message information is displayed on the speaker 28 ( The guidance process may be performed by outputting from (see FIG. 2).

  The operator performs an operation of changing the direction of the object or moving the object so that the high-characteristic region of the object is brought to the camera 27 according to the navigation of the product (for example, display of the guidance image GP). Thereby, the merchandise registration apparatus 1 acquires a frame image (captured image) in which the high feature region of the object is captured. Then, the narrowing-down unit 93 tries to uniquely narrow down candidates for the target product by comparing the object detected by the object detection unit 91 with the auxiliary image of the handling product having the highest degree of similarity and determining the similarity. .

  Next, in step S <b> 19, the narrowing-down unit 93 has uniquely narrowed down the target product candidates by determining whether the similarity to the object exceeds a predetermined threshold in the determination of the similarity based on the auxiliary image. Judge whether or not.

  If the candidate for the target product cannot be uniquely narrowed down in step S19 (No), the process returns to step S11. On the other hand, when candidates for the target product are uniquely narrowed down in step S19 (Yes), the process proceeds to step S21. In this case, in step S21, the confirmation notifying unit 95 sets the candidates of the target products that are uniquely narrowed down as the confirmed products, displays the product name and unit price together with the image of the confirmed products, and displays information on the confirmed products as a speaker. To 28. As a result, the confirmation notification unit 95 can notify the operator that the candidates for the target product have been uniquely narrowed down (that is, the confirmed product has been confirmed).

  In step S22, the information output unit 97 outputs information indicating the product (for example, product ID, product name, discount information, etc.) to the POS terminal 3 via the interface 25 for the confirmed product, and the processing of FIG. Exit. The POS terminal 3 temporarily registers the product and registers it in the product master file together with an instruction for payment.

  In such a configuration, the product identification device 2 performs object recognition of an object (product) based on the reference image, and when target product candidates are narrowed down under a predetermined condition (for example, similarity to an object) When the target product candidate is narrowed down to one or when the similarity to the object is narrowed down to a small number of target product candidates exceeding the threshold), the height of the object is determined based on the auxiliary information. Guidance is given to the operator so that the characteristic region is placed on the camera 27. Thereby, the product identification device 2 can acquire a captured image including a high feature area, and can attempt to uniquely narrow down target product candidates based on the similarity between the acquired captured image and the auxiliary image. . As a result, the product identification device 2 can improve the accuracy (specific accuracy) for narrowing down candidates for the target product.

  It should be noted that the contents of the feature amount file 361, the extracted image, the auxiliary image, and the like can be changed as appropriate according to the product handled and according to the operation. For example, there may be a plurality of auxiliary images instead of only one. In addition, the auxiliary image may be an image of one or more parts of each part such as the upper surface, the lower surface, the right surface, the left surface, and the back surface of the handled product.

  In addition, the product identification device 2 sequentially monitors the captured images when the operator performs a manipulation operation on an object (product) in front of the camera 27 regardless of the determination result of the degree of similarity. When a candidate is estimated to some extent, a guidance screen indicating how to perform a fraud operation on an object (product) may be displayed on the display 221.

  As described above, according to the product identification device 2 according to the present embodiment, it is possible to improve the accuracy of narrowing down target product candidates.

  The present invention is not limited to the above-described embodiment, and various changes and modifications can be made without departing from the gist of the present invention.

  For example, the above-described embodiment has been described in detail in order to easily understand the gist of the present invention. Therefore, the present invention is not necessarily limited to the one having all the configurations described. In the present invention, a part of the configuration of the embodiment can be replaced with another configuration. In the present invention, a part of the configuration can be deleted from the configuration of the embodiment.

  Further, for example, when the product identification device 2 acquires a front image of an object as a captured image and the high feature region (feature portion) of the object is on the front side of the object, the high feature region May be displayed on the display 221 such that the object is moved horizontally or brought close to the camera 27.

  Further, for example, the present invention is not limited to a cash register operated by a store clerk as an operator, but is applied to a cash register called a “self-register”, which is operated by a customer as an operator. Also good.

The invention described in the scope of claims attached to the application of this application will be added below. The item numbers of the claims described in the appendix are as set forth in the claims attached to the application of this application.
[Appendix]
<Claim 1>
A storage means in which a reference image representing the appearance of the handled product and auxiliary information indicating a characteristic part of the handled product are stored for each handled product;
An imaging means for capturing an image of the object by capturing an object that the operator has deceived;
Object detection means for detecting the object from a captured image acquired by the imaging means;
Narrowing means for narrowing down the candidates for merchandise to be traded based on the similarity between the object detected by the object detecting means and the reference image of each handled product;
When the candidate for the target product is narrowed down under a predetermined condition by the narrowing-down means, the imaging means is caused to cause the operator to hesitate the characteristic portion of the object based on the auxiliary information corresponding to the narrowed-down candidate. A product identification apparatus comprising: a guidance unit that performs guidance.
<Claim 2>
The product identification device according to claim 1,
The predetermined condition defines a threshold of similarity to the object,
The narrowing-down means narrows down one handling product having the highest similarity to the object as a candidate for the target product, although the similarity to the object is not more than the threshold.
<Claim 3>
In the merchandise item identification device according to claim 1 or 2,
The auxiliary information includes an auxiliary image indicating the characteristic portion,
The product identification apparatus according to claim 1, wherein the guidance unit performs the guidance by generating a guidance image in which the captured image and the auxiliary image are combined and displaying the guidance image on a display unit.
<Claim 4>
In the merchandise item identification device according to any one of claims 1 to 3,
The auxiliary information includes message information indicating the characteristic part,
The product identification device according to claim 1, wherein the guidance means performs the guidance by displaying the message information on a display means as text.
<Claim 5>
In the merchandise item identification device according to any one of claims 1 to 4,
The auxiliary information includes message information indicating the characteristic part,
The product identification device according to claim 1, wherein the guidance unit performs the guidance by outputting a voice corresponding to the message information from a voice output unit.
<Claim 6>
In the merchandise item identification device according to any one of claims 1 to 5,
The auxiliary information includes an auxiliary image indicating the characteristic portion,
The narrowing-down means compares the object detected by the object detection means with the auxiliary image of the handling product having the highest similarity after the guidance by the guidance means, and judges the similarity of the target product. A product identification device characterized by uniquely narrowing down candidates.
<Claim 7>
For each product handled, a storage means storing a reference image representing the appearance of the product handled and auxiliary information indicating a characteristic part of the product handled, and capturing an image taken by the operator to obtain a captured image of the object An object detection step of detecting the object from the captured image acquired by the imaging means;
The control means, a narrowing step of narrowing down candidates for products for transaction based on the similarity between the object detected in the object detection step and the reference image of each handled product;
When the candidate for the target product is narrowed down under a predetermined condition in the narrowing down step, the control means determines the object characteristic portion to the operator based on auxiliary information corresponding to the narrowed down candidate. A merchandise recognition navigation method, comprising: a guidance step of performing guidance that causes the photographing means to appear.

1 Product Registration Device 2 Product Identification Device 21 Microcomputer (Control Unit)
211 CPU
212 ROM
213 RAM
22 Display / Operation Unit 221 Display (Display means)
222 Touch panel 27 Camera (Photographing means)
3 POS terminal 36 HDD (storage means)
361 Feature file 90 Image acquisition unit 91 Object detection unit (object detection means)
92 similarity calculation unit 93 narrowing unit (narrowing means)
94 Guide part (guide means)
95 Confirmation notification section 96 Input acquisition section 97 Information output section 99 Sales registration section CP Captured image GP Guide image N0001F. jpg, N0002F. jpg Reference image N0001B. jpg, N0002B. jpg extraction image N0001SP. jpg, N0002SP. jpg Auxiliary image SP1, SP2 High feature area

Claims (5)

A reference image representing the appearance of the products handled in each Products, manages the guidance information to guide the photographing of a particular site to accurately identify the products handled within the appearance of the products handled, with storage means Management means;
Transactions by the operator by photographing a commodity that holds up to acquire a captured image of the product by the photographing means, compared with the reference image of each Products stored managed in the storage unit by recognizing the captured image and narrowing means to narrow down the candidate of the subject merchandise,
The guidance information stored in the storage means corresponding to the transaction target products narrowed down by the narrowing means is acquired, and based on the guidance information, the product appearance in which direction among the appearances of the transaction target products is obtained. Guidance display means for displaying a guidance image that guides whether to take the imaging means, and further, which specific part of the appearance of the product in the direction to be tricked to the imaging means ;
A product identification device comprising: The guidance display means, when the similarity of the products when the candidates are narrowed down by the narrowing means is equal to or less than a predetermined threshold, and the transaction target products are narrowed down to one, the transaction target products Retrieve the corresponding stored guidance information,
The merchandise item identification device according to claim 1. The guidance display means obtains guidance information stored corresponding to the transaction target product when the narrowing means narrows down to a small number of transaction target products whose product similarity exceeds a predetermined threshold.
The merchandise item identification device according to claim 1. The guidance display means ends the guidance by displaying the guidance image when the candidates for the transaction target products when the products are narrowed down by the narrowing means can not be narrowed down uniquely. In this case, guidance is performed by displaying the guidance image.
The product identification device according to any one of claims 1 to 3. A reference image representing the appearance of the products handled in each Products, manages the guidance information to guide the photographing of a particular site to accurately identify the products handled within the appearance of the products handled, with storage means Administrative steps;
Transactions by the operator by photographing a commodity that holds up to acquire a captured image of the product by the photographing means, compared with the reference image of each Products stored managed in the storage unit by recognizing the captured image A refinement step to narrow down candidate products,
The guidance information stored in the storage unit corresponding to the transaction target products narrowed down in the narrowing step is acquired, and based on the guidance information, the product appearance in which direction among the appearances of the transaction target products is obtained. A guidance display step for displaying a guidance image for guiding whether to take the imaging means, or which particular part of the appearance of the product in the direction to be tricked to the imaging means ;
A product recognition navigation method comprising:

Images