JP2016110480A - Commodity registration device and commodity registration method - Google Patents

Abstract

Sales related information can be registered without complicated operations. A product registration device (1) includes storage means for storing sales related information of a product in association with an index object used for registering sales related information indicating information related to the sale of the product, and photographing. When the object is recognized using the photographed image photographed by the means, and the product and the index object are recognized, the sales related information corresponding to the index object stored in the storage means, together with the information of the recognized product Control means for registration. [Selection] Figure 3

Description

  The present invention relates to a merchandise registration apparatus and merchandise registration method using object (object) recognition.

  Conventionally, it is related to general object recognition for extracting the feature amount of an object from image data obtained by photographing the article and comparing it with data (feature amount) for collation prepared in advance to recognize the type of the article. Technology is known. And the shop system which carries out sales registration of the recognized goods using the technique which concerns on this general object recognition for recognition of goods, such as bread and fruit and vegetables, is proposed (refer patent documents 1 and 2).

  On the other hand, information on sales using a product registration device (electronic cash register) (hereinafter referred to as “sales information”) at merchandise stores such as supermarkets, bakery stores, fruit and vegetable stores, and convenience stores. Registration (recording) is performed. The sales information includes, for example, identification information and prices of sales products, sales date and time, and the like. In addition to this sales information, information that will be used as an index for future plans such as purchases and sales of special sale dates, special sale time (time sale) discounts, sales volume, customer demographics, etc. (hereinafter referred to as “sales-related information”) Registration (recording) is also performed together with the sales information. In a merchandise store, for example, a customer segment information input key is provided on a keyboard of a POS (Point Of Sale system) register terminal, and a customer segment information input key (for example, “m・ By specifying (pressing down) the “Adult” key, “Woman / Child” key, etc., information on the customer base can be obtained, or the customer base information can be obtained by analyzing the image information of the customers who visited the store. (See Patent Document 3). Many merchandise stores categorize customers by gender, age, etc., examine products with high sales by customer segment, and use these data for operations such as new product development and product ordering.

JP 2011-170745 A JP 2013-182457 A JP 2010-055248 A

  However, the technology for acquiring sales related information such as customer demographic information by analyzing image information obtained by photographing the customer described above has been pointed out as a problem at the time of acquiring image information from the viewpoint of customer personal information (privacy). The current situation is that it is difficult to adopt. On the other hand, when recognizing an object (object) and recognizing a product and registering the recognized product, if sales related information such as a customer base is entered, the object recognition process is performed and then the sales of the customer base etc. are performed. In order to input related information, it is necessary to input a key of a register, and the operation becomes complicated. Further, it is recognized that the customer is performing an operation other than product registration, and there is a risk of causing doubts about what operation is being performed.

  An object of the present invention is to enable sales related information to be registered without performing complicated operations.

  The product registration apparatus of the present invention is photographed by a storage unit storing the sales related information in association with an index object used for registering sales related information indicating information related to the sale of the product, and a photographing unit. When the object is recognized using the photographed image and the product and the index object are recognized, the sales related information corresponding to the index object stored in the storage means is the information on the recognized product. And a control means for registering together.

  The product registration method of the present invention includes a step of storing the sales related information in a storage unit in advance in association with an index object used for registering sales related information indicating information related to the sale of the product, and a photographing unit. A step of recognizing an object using a photographed image taken by the method, and when the product and the indicator are recognized, sales related information corresponding to the indicator stored in the storage means is recognized. And registering with the product information.

  According to the present invention, sales related information can be registered without performing complicated operations.

It is the schematic of the external appearance of the goods registration apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows the hardware constitutions of the goods registration apparatus which concerns on the 1st Embodiment of this invention. It is a functional block diagram of the goods registration device concerning a 1st embodiment of the present invention. It is a figure for demonstrating the determination process of a goods and an index thing which the goods registration apparatus which concerns on the 1st Embodiment of this invention performs. It is a figure which shows the data structural example of the merchandise master information which concerns on the 1st Embodiment of this invention. It is a figure which shows the data structural example of the index thing master information which concerns on the 1st Embodiment of this invention. It is a figure for demonstrating the predetermined area | region and division in which the index object which concerns on the 1st Embodiment of this invention is located. It is a figure which shows the data structural example of the index object position corresponding | compatible information which concerns on the 1st Embodiment of this invention. It is a figure which shows confirming customer segment information by the predetermined area | region and division in which the index thing which concerns on the 1st Embodiment of this invention is located. It is a figure which shows confirming discount information by the predetermined area | region and division in which the index object which concerns on the 1st Embodiment of this invention is located. It is a figure which shows confirming goods etc. by the predetermined area | region where the index object which concerns on the 1st Embodiment of this invention is located. It is a figure which shows the data structural example of the merchandise sales registration information which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the flow of the goods and sales related information registration process of the goods registration apparatus which concerns on the 1st Embodiment of this invention. It is a functional block diagram of the goods registration apparatus which concerns on the 2nd Embodiment of this invention. It is a figure which shows the data structural example of the gesture master information which concerns on the 2nd Embodiment of this invention. It is a figure which shows making a number recognize by the gesture of the index object (hand) which concerns on the 2nd Embodiment of this invention. It is a figure which shows making a discount object goods recognize by the gesture of the index thing (hand) concerning a 2nd embodiment of the present invention. It is a flowchart which shows the flow of the goods and sales related information registration process of the goods registration apparatus which concerns on the 2nd Embodiment of this invention.

  Next, the product registration apparatus 1 in a mode for carrying out the present invention (hereinafter referred to as “the present embodiment”) will be described.

<First Embodiment>
First, the product registration apparatus 1 according to the first embodiment of the present invention will be described.
FIG. 1 is a schematic view of the appearance of a product registration apparatus 1 according to the first embodiment of the present invention. As shown in FIG. 1, the commodity registration apparatus 1 includes a customer display 11, a touch display 12, a cash drawer 13, a printer 14, a photographing apparatus 15, a photographing stand 16, and a backlight source 17, and a counter stand of a merchandise store. 2 is installed. It should be noted that when accounting for a product, an operator (clerk) who handles the product registration apparatus 1 stands on the front side of the counter table 2 and a customer who purchases the product stands on the back side of the counter table 2.
The product registration device 1 functions as a POS register terminal, receives information such as the type (identification information) and the number of products, and inputs / outputs information for displaying the total price, sales management, sales performance management, and the like. .

The customer display 11 is a liquid crystal display device, for example, and faces the back of the drawing on the customer side. The customer display 11 displays information (product name, amount, etc.) related to product accounting to the customer.
The touch display 12 is formed by laminating a touch panel 12B (see FIG. 2) on the surface of a display 12A (see FIG. 2) which is a liquid crystal display device, for example, and faces the front side of the drawing on the operator side. The touch display 12 displays a photographed image and various types of information (product name, amount, etc.) to the operator and accepts touch operation input by the operator.

The cash drawer 13 is a drawer that stores banknotes, money, cash vouchers, etc., which are handled at the time of product accounting.
The printer 14 prints the contents of the accounting (product name, amount, etc.) on the receipt when the product is accounted for.

  The photographing device 15 photographs the tray 3 placed on the photographing stand 16, the product on the tray 3, and the index object. An illumination device (not shown) is attached to the photographing device 15 so as to irradiate the photographing region 151 of the photographing device 15. Here, the index object is a tool for sandwiching a product that is preferably not handled by the operator's hand or directly by hand (for example, “tongue” in a bakery), and is used for object recognition. This is used as an index when inputting sales related information. This indicator will be described in detail later.

The photographing stand 16 is a stand on which the customer who purchases the product places the tray 3 on which the product is placed at the time of payment.
The imaging area 151 is an area that can be imaged by the imaging device 15 on the imaging platform 16.

The backlight light source 17 is accommodated in the photographing stand 16 and is backlit from the bottom of the tray 3 so that the photographed image of the commodity becomes clearer when photographing the commodity on the tray 3 by the photographing device 15. Irradiate upward. The backlight light source 17 can be realized by, for example, an LED (Light Emitting Diode), but is not limited thereto.
The tray 3 is translucent so as to transmit light. When the product placed on the tray 3 is photographed by the photographing device 15, irradiation light is emitted from the illumination device, and from the bottom of the tray 3, the backlight is emitted upward by the backlight light source 17. Is done. Thereby, the shadow formed around the product can be eliminated as much as possible.

  In the product registration apparatus 1 according to the first embodiment of the present invention, a product on the tray 3 is recognized by processing an image photographed by the photographing device 15, and a predetermined area of the tray 3, for example, an edge The sales related information is identified and registered by recognizing the overlap between the part (see FIG. 4 to be described later) and an index object such as a hand or tongue, and specifying the position of the overlap. Here, as described above, the sales related information is, for example, customer demographic information of customers, discount information on products, and the like, and is information registered in connection with sales of products.

Next, the configuration of the commodity registration apparatus 1 will be described with reference to FIG.
FIG. 2 is a diagram showing a hardware configuration of the commodity registration apparatus 1 according to the first embodiment of the present invention.
The product registration apparatus 1 includes a CPU (Central Processing Unit) 20, a RAM (Random Access Memory) 21, a ROM (Read Only Memory) 22, and a storage unit 23 in addition to the configuration shown in FIG. Each component is connected to be communicable with each other via an internal bus and each input / output circuit (not shown).

The CPU 20 is a central control device and controls the product registration device 1 as a whole.
The RAM 21 is temporary storage means used by the CPU 20. The image data related to the program executed by the CPU 20 is temporarily stored.
The ROM 22 is a non-volatile storage unit and stores a program executed by the CPU 20.

The customer display 11 is controlled by the CPU 20 and displays a photographed image such as a product and information (product name, amount, etc.) relating to the accounting of the product to the customer.
The display 12 </ b> A is controlled by the CPU 20 and displays a photographed image such as a product and information (product name, price, etc.) relating to product accounting to the operator.
Touch panel 12B accepts an operator's touch operation input for information displayed on display 12A.

  The storage unit 23 includes, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), and the like, and stores various programs and various files. Various programs and various files stored in the storage unit 23 are all or partially copied to the RAM 21 and executed by the CPU 20 when the product registration device 1 is activated. The storage unit 23 stores product master information 100, index object master information 200, index object position correspondence information 300, product sales registration information 400, and the like, which will be described later with reference to FIG.

  The imaging device 15 is configured by a color CCD (Charge Coupled Device) image sensor, a color CMOS (Complementary Metal Oxide Semiconductor) image sensor, and the like, and is an imaging unit that performs imaging under the control of the CPU 20. The imaging device 15 captures a moving image of 30 fps (frame per second), for example. Frame images (captured images) sequentially captured by the image capturing device 15 at a predetermined frame rate are stored in the RAM 21.

  The backlight light source 17 is a backlight from the bottom of the tray 3 so that the photographed image becomes clearer when the photographing device 15 photographs the product or the index object on the tray 3 under the control of the CPU 20. Irradiate upward. Thereby, the shadow in the imaging | photography area | region 151 formed with the illuminating device etc. which were attached adjacent to the imaging device 15 can be made thin, and the precision of image processing can be improved. The timing of backlight irradiation by the backlight light source 17 may be matched with the timing of shooting by the imaging device 15 or may be always, for example.

The cash drawer 13 stores banknotes, money, cash vouchers, etc., which are handled at the time of product accounting, and is opened by an instruction from the CPU 20.
The printer 14 is a thermal transfer printer, for example, and prints a receipt. Specifically, the printer 14 prints the contents of the accounting on the receipt in accordance with an instruction from the CPU 20 at the time of accounting for the product.

Next, the function of the commodity registration apparatus 1 will be described with reference to FIG. FIG. 3 is a functional block diagram of the commodity registration apparatus 1 according to the first embodiment of the present invention.
The CPU 20 of the product registration device 1 embodies each function of the processing unit 10 by executing a program stored in the ROM 22. The processing unit 10 determines whether the object photographed by the photographing unit is a product or an index object based on the photographed image. Then, the processing unit 10 recognizes the product, recognizes the index object, and acquires the position information of the index object, thereby specifying and registering sales related information such as customer demographic information and discount information. The processing unit 10 includes an image storage unit 110, an object recognition processing unit 120, a sales related information determination unit 130, a product / sales related information presentation unit 140, an input acquisition unit 150, a sales information registration unit 160, and an information output unit 170. .

  The image storage unit 110 sequentially captures frame images (color digital images) captured by the image capturing device 15 and stored in the RAM 21. The capturing of the frame image by the image storage unit 110 is performed in the order stored in the RAM 21.

The object recognition processing unit 120 determines whether or not a predetermined region (for example, the edge 30 of the tray 3 (see FIG. 4)) and an index object such as a hand or tongue overlap the frame image captured by the image storage unit 110. Is recognized to determine whether the item is a product or an indicator. Then, the object recognition processing unit 120 detects an object by separating an image of a product or an index object from the background using a pattern matching technique or the like. Specifically, this background separation technique is executed, for example, by binarizing a frame image, extracting and determining each contour line, and partitioning and separating the images of those regions. Note that an image in which an image of a product or an index object is separated from the background may be hereinafter referred to as an object recognition image or a separated image.
The object recognition processing unit 120 recognizes the commodity on the tray 3, recognizes an index object located in a predetermined area (overlaps with the predetermined area), and detects the position of the index object. This will be specifically described below.

The object recognition processing unit 120 includes an object region determination unit 121, a product detection unit 122, and an index object detection unit 125.
The object area determination unit 121 determines whether or not the frame image captured by the image storage unit 110 is an object that overlaps a predetermined area by recognizing the object. Then, the object area determination unit 121 passes the object recognition image (separated image) to the product detection unit 122, assuming that the object is a product for an object determined not to overlap the predetermined region. On the other hand, the object area determination unit 121 passes the object recognition image (separated image) to the index object detection unit 125 regarding the object determined to overlap the predetermined area, assuming that the object is the index object.

  FIG. 4 is a diagram for explaining a process for determining a product and an index object performed by the product registration apparatus 1 according to the first embodiment of the present invention. FIG. 4A shows a case where the product is an object a (curry bread), an object b (melon bread), and an index object (object c) is an operator's “hand”. FIG. 4B shows a case where the product is an object a (curry bread), an object b (melon bread), and an index object (object d) is a “tongue” that sandwiches bread or the like.

  In the example shown in FIG. 4A, the object region determination unit 121 recognizes three objects (object a, object b, object c) and tray 3 (edge 30). Then, the object area determination unit 121 determines that the object a and the object b do not overlap with a predetermined area (here, the edge 30 of the tray 3), that is, is a product, and the object recognition image ( The separated image is delivered to the product detection unit 122. On the other hand, the object area determination unit 121 determines that the object c overlaps a predetermined area (here, the edge 30 of the tray 3), that is, is an index object, and detects the object recognition information as an index object detection. Delivered to part 125.

  In the example shown in FIG. 4B, the object region determination unit 121 recognizes three objects (object a, object b, object d) and tray 3 (edge 30). Then, the object region determination unit 121 determines that the object a and the object b do not overlap with a predetermined region (here, the edge 30 of the tray 3), that is, the product, and the object recognition image (Separated image) is delivered to the product detection unit 122. On the other hand, the object area determination unit 121 determines that the object d overlaps a predetermined area (here, the edge 30 of the tray 3), that is, is an index object, and the object recognition information is detected as an index object. Delivered to part 125. In the following, the case where the index object is the “hand” of the operator will be mainly described.

  Returning to FIG. 3, the product detection unit 122 governs the entire object (object) recognition for the object that the object region determination unit 121 determines not to overlap the predetermined region, specifically, the product (object) recognition. Unit 123 and product similarity determination unit 124.

  The product similarity calculation unit 123 calculates the surface state such as the shape, size, hue, and surface roughness of the acquired object recognition image (separated image) as a feature amount. The product similarity calculation unit 123 further calculates the feature amount of the separated image and the feature amount of each product stored in advance as a part of the product master information 100 (see FIG. 5 described later) in the storage unit 23. Compare and calculate the similarity. Here, the similarity refers to how similar the object recognition image (separated image) is when the feature amount assumed by the product stored in the product master information 100 is 100% similarity. It is shown. When calculating the degree of similarity, when there are a plurality of types of feature quantities, comprehensive evaluation is performed based on the plurality of feature quantities. At that time, weighting may be performed for each feature quantity.

  FIG. 5 is a diagram showing a data configuration example of the product master information 100 according to the first embodiment of the present invention. As shown in FIG. 5, the product master information 100 stores a product name, a unit price of the product, and a feature amount in association with the product ID. Here, the feature amount may be a representative image of the product, or may be data indicating a feature such as a shape, a size, a hue, and the like. Stored in a suitable information format.

Recognizing an object included in an image in this way is called generic object recognition. In the following "Current Status and Future of General Object Recognition" written by Keiji Yanai, a survey of general object recognition research is added to methods, data sets and evaluation benchmarks are performed, and the future is prospected.
Keiji Yanai, “Current Status and Future of General Object Recognition”, [online], Journal of Information Processing Society of Japan, November 15, 2007, Vol. 48, No. SIG16, pp. 1-24, [November 10, 2014 Day search], Internet <URL: http: //mm.cs.uec.ac.jp/IPSJ-TCVIM-Yanai.pdf>

A technique for performing general object recognition by dividing an image into regions for each object (object) is described in the following document.
Jamie Shotton et al., “Semantic Texton Forests for Image Categorization and Segmentation”, Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, [searched November 10, 2014], Internet <URL: http: // citeseerx .ist.psu.edu / viewdoc / download? doi = 10.1.1.145.3036 & rep = rep1 & type = pdf>

  It should be noted that the method of calculating the similarity between the photographed product image and the product image (representative image) of the registered product registered in the product master information 100 is not particularly limited. For example, the similarity between the photographed product image and each registered product registered in the product master information 100 may be calculated as an absolute evaluation or may be calculated as a relative evaluation.

  When the similarity is calculated as an absolute evaluation, the feature amount of the separated image is compared with the feature amount of each product stored in the product master information 100 on a one-to-one basis, and the similarity calculated as a result of this comparison (0 to 100%) may be adopted as it is.

  Further, when calculating the similarity as a relative evaluation, the total of the similarities with each product is calculated to be 1.0 (100%). For example, it is assumed that feature quantities of products A, B, C, and D are stored as the product master information 100. At this time, for the separated image, for example, the similarity is 0.65 for the product A, the similarity is 0.2 for the product B, the similarity is 0.1 for the product C, and the product D For example, the similarity is calculated as 0.05.

Returning to FIG. 3, the product similarity determination unit 124 determines, for each of the separated images of the object, for example, the corresponding product or the like based on the similarity calculated by the product similarity calculation unit 123 by the following three options.
(1) A product corresponding to the separated image is uniquely determined.
(2) There are one or more product candidates corresponding to the separated image.
(3) It is determined that there is no product corresponding to the separated image.

  As a condition for this determination, for example, the storage unit 23 stores conditions X and Y in advance. Hereinafter, a case where the similarity calculation method is absolute evaluation will be described as an example.

  Condition X is a condition for determining (1) above if satisfied. The condition X is, for example, “the similarity of the product with the highest similarity is 90% or more” and “the similarity of the product with the highest similarity and the similarity of the product with the second highest similarity. The difference is 20% or more. Specifically, for example, for an object in a certain separated image, it is assumed that the product having the highest similarity is the product A having a similarity of 95%, and the next is the product B having a similarity of 60%. Since the condition X is satisfied at this time, the product A corresponding to the separated image is uniquely determined.

If the condition X is not satisfied, the condition Y is used.
Condition Y is a condition for determining (2) above if satisfied. The condition Y is, for example, “there is at least one product having a similarity of 60% or more”. Specifically, for example, for an object in a certain separated image, the product with the highest similarity is the product A with the similarity 80%, the second is the product B with the similarity 75%, and the third is the similarity 65%. And the fourth is a product D having a similarity of 55%. Since the condition Y is satisfied at this time, there are products A, B, and C that are targets with a similarity of 60% or more as candidates corresponding to the separated images.

When both the conditions X and Y are not satisfied, it is determined as (3) above. The above-described conditions X and Y are examples, and are not limited to these.
Also, when the similarity calculation method is relative evaluation, the conditions may be set in the same manner.

  The index object detection unit 125 recognizes an object that the object area determination unit 121 determines to overlap a predetermined area (here, the edge 30 of the tray 3), specifically, object (object) recognition for the index object. The index object recognition unit 126 and the index object position determination unit 127 are provided.

  The index object recognizing unit 126 calculates the surface state such as the shape, size, hue, and surface roughness of the object recognition image (separated image) acquired from the object region determination unit 121 as a feature amount. The index object recognition unit 126 compares the calculated feature value of the separated image with the feature value of each index object stored in advance as index object master information 200 (see FIG. 6 described later) in the storage unit 23. The similarity is calculated. Then, for example, if the similarity of the index object having the highest similarity is 70% or more, the index object recognition unit 126 selects the index object corresponding to the separated image having the highest similarity ( For example, “hand” or “tongue”).

  FIG. 6 is a diagram illustrating a data configuration example of the index object master information 200 according to the first embodiment of the present invention. As shown in FIG. 6, the index object master information 200 stores the index object name and the feature amount of the index object in association with the index object ID. Here, examples of “hand” and “tongue” are shown as index objects. The feature quantity stores data indicating the shape, size, hue, and the like of the index object and a representative image. In FIG. 6, as shown in FIG. 4A, as a feature amount of a typical hand shape (left hand / right hand) in the separated image in a state where the operator's hand is overlapped with the edge 30 of the tray 3. In addition, an example in which feature quantities of a plurality of hand shapes shown in the representative image α are registered is shown. In FIG. 6, as shown in FIG. 4B, the representative image β is represented as a characteristic amount of a typical tong shape in the separated image in a state where the tongs overlap the edge 30 of the tray 3. The example which registered the feature-value of the shape of several tongs to show is shown. In this way, a plurality of typical feature quantities typical of hand shapes and tong shapes are registered in the index object master information 200 in advance.

  The index object recognition unit 126 determines the index object corresponding to the separated image by comparing the acquired object recognition image (separated image) with the feature amount of each index object stored in the index object master information 200. . Then, the determined index information (for example, “hand” or “tongue”) is handed over to the sales related information determination unit 130.

  Returning to FIG. 3, the object position determination unit 127 determines that the object recognition image (separated image) acquired from the object region determination unit 121 overlaps a predetermined region (for example, “hand” or “tongue”). The position at the edge 30 of the tray 3 is specified.

For example, as shown in FIG. 7, the index object position determination unit 127 is an image (frame image) obtained by photographing the tray 3 placed on the photographing table 16 from the upper side, as shown in FIG. The area on the left side of the edge 30 of the tray 3 is defined as a first area (reference numeral 301), and the area on the right side of the edge 30 is defined as a second area (reference numeral 302).
Then, the index object position determination unit 127 determines in which region of the predetermined area (here, the edge 30) the index object of the acquired object recognition image (separated image) is located. For example, the index object position determination unit 127 determines that the index object is positioned so as to overlap the section [1] of the first region. The index object position determination unit 127 delivers the determined index object position information (for example, “first region / section [1]”) to the sales related information determination unit 130.

  Returning to FIG. 3, the sales related information determination unit 130 stores the information in the storage unit 23 based on the index object information acquired from the index object recognition unit 126 and the index object position information acquired from the index object position determination unit 127. The sales related information is determined by referring to the index object position correspondence information 300.

FIG. 8 is a diagram illustrating a data configuration example of the index object position correspondence information 300 according to the first embodiment of the present invention. As shown in FIG. 8, in the index object position correspondence information 300, sales related information is stored in association with the index object and the position information of the index object.
For example, the information of the index object acquired from the index object recognition unit 126 is “hand”, and the position information of the index object acquired from the index object position determination unit 127 is the classification “1” of the region “1” (first region). In this case, “female / child” is associated as sales related information (customer demographic information here). Similarly, the index object information acquired from the index object recognizing unit 126 is “hand”, and the position information of the index object acquired from the index object position determining unit 127 is the region “2” (second region) classification “2”. ”Is associated with“ male / adult ”as sales related information.

  The sales related information determination unit 130 determines the sales related information (customer demographic information) by referring to the index object position correspondence information 300 (FIG. 8). For example, in the example illustrated in FIG. 9, the sales related information determination unit 130 indicates that the index object is “hand” and the position information of the index object is the classification [1] of the first area. With reference to the correspondence information 300, it is determined that the sales related information (customer demographic information) is “female / child”. Then, the sales related information determination unit 130 outputs the determined sales related information to the product / sales related information presentation unit 140.

  Note that the sales related information determination unit 130 may set discount information (in this case, a discount amount) as the sales related information. For example, in the third area shown in FIG. 10A, the index object position correspondence information 300 (see FIG. 8) indicates that the index object is “hand” and the position information of the index object is the section [1] of the third area. In this case, “10 yen discount” is stored, “20 yen discount” in the case of category [2], and “50 yen discount” in the case of category [3]. Then, as in the example shown in FIG. 10B, the sales related information determination unit 130 classifies the index object information as “hand” and the index object position information as the area “3” (third area). In the case of “1”, it is determined that the sales related information (in this case, the discount amount) is “10 yen discount”. In this way, the sales related information determination unit 130 can specify the sales related information (discount amount) based on the information of the index object and the position information thereof. In addition, a discount rate (1 discount, 2 discount, 5 discount) or the like may be set as the sales related information.

In addition, the sales related information confirmation unit 130 performs a process (accounting content confirmation process) for confirming registration of a product or the like based on the fact that the index object (hand, tong, etc.) is located in a predetermined area. You can also.
For example, the fourth area and the fifth area of the edge 30 of the tray 3 shown in FIG. 11A are registered in the index object position correspondence information 300 (see FIG. 8) in association with the registration confirmation process of the product or the like. Keep it. Then, as in the example illustrated in FIG. 11B, the sales related information determination unit 130 has the index object information “hand” and the index object position information is the area “4” (fourth area). When the index object position determination unit 127 determines, the registration confirmation process for merchandise and the like (accounting content confirmation process) is executed. In this way, the operator can confirm the identification information, the number, etc. of the product without operating the confirm button on the touch display 12 (see FIG. 1).

  Returning to FIG. 3, the product / sales related information presenting unit 140 determines the product similarity determined by the product similarity determination unit 124 and the plurality of candidates of (2) above. Are displayed on the display 12A and the customer display 11, and confirmed sales-related information and the like are displayed on the display 12A.

The input acquisition unit 150 receives various input operations corresponding to the display on the display 12A via the touch panel 12B. For example, when it is determined as (1) or (2) and one or more images of product candidates are displayed on the display 12A, a touch input operation for product selection using the touch panel 12B by the operator is accepted.
The confirmation of the product, the sales related information, etc. may be performed by accepting an input operation by the operator via the input acquisition unit 150. As described above, the sales related information determination unit 130 This may be performed by recognizing that the index object is located in a predetermined area (in the example of FIG. 11, the fourth area and the fifth area) on the tray 3 that executes the registration confirmation process.

The sales information registration unit 160 registers the product and the sales related information determined by the input acquisition unit 150 or the sales related information determination unit 130 in the product sales registration information 400 in the storage unit 23.
FIG. 12 is a diagram showing a data configuration example of the merchandise sales registration information 400 according to the first embodiment of the present invention. As shown in FIG. 12, in the product sales registration information 400, a sales ID, a sales time, a product ID, a product name, a unit price (list price), a discount amount, a number, a sales amount, customer demographic information, and the like are registered.
Here, the sales information registration unit 160 obtains a product ID, a product name, and a unit price (list price) by referring to the product master information 100 for a product for which sales are confirmed in association with the sales ID and the sales time. sign up. Then, the sales information registration unit 160 calculates and registers the sales amount based on the determined number of sales for each product. In addition, the sales information registration unit 160 registers sales related information such as customer demographic information and a discount amount, which is determined by the sales related information determination unit 130.

  The information output unit 170 displays the product ID of the product determined as described above and the corresponding product name, unit price, sales quantity, etc. on the customer display 11 and the display 12A, and outputs them to the printer 14.

<Process flow>
Next, the process flow of the product registration apparatus 1 will be described.
FIG. 13 is a flowchart showing the flow of the product / sales related information registration process of the product registration apparatus 1 according to the first embodiment of the present invention.
The merchandise registration apparatus 1 continuously captures an image of the imaging region 151 of the imaging table 16 using the imaging device 15. Alternatively, when the customer places the tray 3 on the photographing stand 16 and the operator touches the photographing button on the touch display 12, the processing unit 10 of the commodity registration apparatus 1 takes in the image from the photographing apparatus 15 and performs a series of processing. Start.

First, the processing unit 10 starts capturing an image from the imaging device 15 (step S10). The image storage unit 110 stores a frame image (captured image) captured by the imaging device 15 in the RAM 21.
Next, the object recognition processing unit 120 performs object recognition processing on the frame image stored in the image storage unit 110.
Specifically, the object region determination unit 121 of the object recognition processing unit 120 determines whether or not an object (a product or an index object) on the captured image overlaps a predetermined region (for example, the edge 30 of the tray 3). In addition to recognizing the object (product or index object) from the background, an object recognition image (separated image) is generated (step S11). Then, the object region determination unit 121 determines whether or not the recognized object overlaps a predetermined region (step S12). If the object overlaps (step S12 → Yes), the object region determination unit 121 determines that the object is an index object. The object recognition image is output to the index object detection unit 125, and the process proceeds to step S13. On the other hand, when the recognized object does not overlap the predetermined area (step S12 → No), the object region determination unit 121 determines that the object is a product, and uses the object recognition image as the product detection unit 122. And proceed to step S19.

Next, the index object detection unit 125 repeats the processes of steps S13 to S18 next for each object (object recognition image) acquired from the object region determination unit 121.
First, the index object recognition unit 126 of the index object detection unit 125 calculates feature quantities such as shape, size, hue, and the like for the acquired object recognition image (separated image), and the index object master information 200 in the storage unit 23. The similarity is calculated by comparing with (see FIG. 6) (step S14). Then, the index object recognition unit 126 determines whether or not this object is a registered index object (step S15). When it is determined that the object is an index object (step S15 → Yes), the index object recognition unit 126 outputs the determined index object information to the sales related information determination unit 130, and proceeds to the next step S16. On the other hand, when this object is not determined to be an index object (step S15 → No), the process proceeds to step S18.

  Subsequently, the index object position determination unit 127 of the index object detection unit 125 uses the frame image (captured image) acquired by the object region determination unit 121 to acquire the object recognition image (separated image) acquired from the object region determination unit 121. , The position of the object (for example, “hand” or “tongue”) that overlaps a predetermined region (for example, the edge 30 of the tray 3) on the edge 30 of the tray 3 is determined (step S16). For example, in the example illustrated in FIG. 9, the index object position determination unit 127 determines that the index object (“hand”) is in the division [1] of the first region. Then, the index object position determination unit 127 outputs the determined position information to the sales related information determination unit 130.

Next, the sales related information determination unit 130 indicates the index object in the storage unit 23 based on the index object information acquired from the index object recognition unit 126 and the index object position information acquired from the index object position determination unit 127. By referring to the position correspondence information 300 (FIG. 8), the sales related information is determined (step S17).
For example, in the example illustrated in FIG. 9, the sales related information determination unit 130 refers to the index object position correspondence information 300 (FIG. 8), so that the index object is “hand” and the index object position information is Since it is the classification [1] of one area, it is determined that the sales related information (here, customer demographic information) is “female / child”.

  Then, the processing unit 10 (the index object detection unit 125) determines whether the processing has been repeated for all the object recognition images (separated images) acquired from the object region determination unit 121 (step S18). If the index object detection unit 125 has not repeated the process for all the acquired object recognition images (separated images), the index object detection unit 125 returns to the process of step S13 and repeats the process for all the acquired object recognition images (separated images). If so, the process proceeds to step S23.

On the other hand, in step S12, when the object region determination unit 121 determines that the recognized object does not overlap with the predetermined region (step S12 → No), the product detection unit 122 acquires from the object region determination unit 121. The processing of the next steps S19 to S22 is repeated for each object (object recognition image).
First, the product similarity calculation unit 123 of the product detection unit 122 calculates feature quantities such as shape, size, and hue for the acquired object recognition image (separated image), and the product master information 100 ( The similarity is calculated by comparing with (see FIG. 5) (step S20).

  Subsequently, the product similarity determination unit 124 of the product detection unit 122 detects a product corresponding to the object recognition image (separated image) based on the similarity calculated by the product similarity calculation unit 123 (step S21). Here, when a plurality of corresponding products are detected, an image of the candidate product is displayed on the display 12A or the customer display 11, and the input acquisition unit 150 touches the product selection using the touch panel 12B by the operator. A product is selected by accepting an input operation.

  And the goods detection part 122 judges whether the process was repeated about all the object recognition images (separated image) acquired from the object area | region determination part 121 (step S22). If the product detection unit 122 has not repeated the process for all the acquired object recognition images (separated images), the product detection unit 122 returns to the process of step S19 and repeated the process for all the acquired object recognition images (separated images). If so, the process proceeds to step S23.

  In step S <b> 23, the product / sales related information presentation unit 140 determines the sales related information (for example, customer demographic information) determined by the sales related information determination unit 130 and the product information detected by the product detection unit 122 (product identification information or Unit price, number, total amount, etc.) are displayed on the display 12A. At this time, the product / sales related information presentation unit 140, for example, when the sales related information is customer segment information, information on products other than customer segment information (product identification information, unit price, number, total amount, etc.) Is displayed on the customer display 11. That is, the customer demographic information is not displayed on the customer display 11.

Subsequently, the processing unit 10 determines whether or not the displayed information is confirmed (accounting content is confirmed) (step S24). Specifically, the processing unit 10 detects that the input acquisition unit 150 is touched by the confirmation button when the touch display 12 (touch panel 12B) is operated by the operator, or the index object detection unit 125 As shown in FIG. 11 (b), by detecting that an index object such as “hand” is located in an area (fourth area) indicating registration confirmation, the product information displayed on the display 12A and Confirm sales related information.
If the processing unit 10 determines that the displayed information is not confirmed (step S24 → No), the process returns to step S10 to continue the processing. On the other hand, if it is determined that the displayed information has been confirmed (step S24 → Yes), the process proceeds to the next step S25.

  In step S25, the sales information registration unit 160 registers the confirmed product information and sales related information in the product sales registration information 400 (see FIG. 12) in the storage unit 23, and ends the process.

  By doing so, the product registration device 1 determines whether the product is an index object when registering the product by object recognition, and without performing an operation on the touch display 12 or the like, By specifying the predetermined position information, the sales related information can be determined, and the product information and the sales related information can be registered. Therefore, it is possible to eliminate a complicated operation using the touch display 12 for inputting sales-related information, a register key, or the like.

<Second Embodiment>
Next, a product registration apparatus 1A according to the second embodiment of the present invention will be described.
The schematic diagram and hardware configuration of the product registration device 1A according to the second embodiment are the same as those of the product registration device 1 according to the first embodiment (see FIGS. 1 and 2), and thus description thereof is omitted. .

  In addition to the function of the commodity registration apparatus 1 according to the first embodiment, the commodity registration apparatus 1A according to the second embodiment recognizes the shape (gesture) of the hand when the index object is “hand”. Thus, more detailed sales related information can be registered without performing an operation on the touch display 12 or the like. For example, the merchandise registration apparatus 1A registers the number of a plurality of the same merchandise (object b: melon bread) as shown in FIG. 16B to be described later, or a plurality of the merchandise registration apparatus 1A as shown in FIG. The discount information is registered by specifying one of the products.

FIG. 14 is a functional block diagram of a product registration apparatus 1A according to the second embodiment of the present invention. About the structure provided with the same function as the goods registration apparatus 1 which concerns on the 1st Embodiment of this invention shown in FIG. 3, the same name and code | symbol are attached | subjected and description is abbreviate | omitted.
14 differs from the product registration device 1 (FIG. 3) in that the processing unit 10 in FIG. 3 is the processing unit 10A in FIG. 14, and the index of the object recognition processing unit 120. The index object position determination unit 127 in the object detection unit 125 is the index object position determination unit 127A, the gesture recognition unit 128 is added to the object recognition processing unit 120, and the storage unit 23 The gesture master information 500 is added.

The index object position determination unit 127 </ b> A determines the position of the object (“hand”, “tongue”, etc.) on the edge 30 of the tray 3 that overlaps a predetermined area, which the index object position determination unit 127 according to the first embodiment has. In addition to the functions, it has the following functions.
When the index object recognizing unit 126 determines that the index object is “hand”, the index object position determination unit 127A determines that the index object (“hand”) is based on the frame image captured by the image storage unit 110. It is determined whether or not the product image overlaps. When the index object position determination unit 127A determines that the index object (“hand”) overlaps the product image, the object recognition image (separated image) of the index object (“hand”) is gesture-recognized. Delivered to part 128.

  The gesture recognizing unit 128 calculates the surface state such as the shape, size, hue, and surface roughness of the object recognition image (separated image) acquired from the object region determining unit 121 as a feature amount. The gesture recognition unit 128 compares the calculated feature amount of the separated image with the feature amount of the hand shape (gesture) stored in advance as the gesture master information 500 in the storage unit 23, and calculates the similarity. Then, for example, if the similarity of the hand shape (gesture) having the highest similarity is 80% or more, the gesture recognition unit 128 determines that the index object corresponding to the separated image has the shape of the hand (gesture). Recognize that

FIG. 15 is a diagram illustrating a data configuration example of the gesture master information 500 according to the second embodiment of the present invention. As shown in FIG. 15, the gesture master information 500 includes a feature amount (data or representative image indicating features such as the shape, size, and hue of the hand) of the gesture (hand shape) in association with the gesture ID. Corresponding information (sales related information) regarding the gesture (hand shape) is stored.
For example, for the gesture ID “G001”, the gesture (hand shape) of pointing the index finger with the palm side of the left hand facing the imaging device 15 is “1” (one piece) as correspondence information (sales related information). It is shown that. The gesture ID “G006” indicates that the gesture (hand shape) of pointing the index finger with the palm of the right hand toward the photographing device 15 is “6” (six) as correspondence information (sales related information). Show.
The gesture ID “G020” has a gesture (hand shape) in which the back side of the left hand is pointed toward the photographing device 15 and the index finger is placed on the screen as correspondence information (sales related information). This indicates that the product is “a product targeted for discount”. Similarly, the gesture ID “G021” indicates that the gesture (hand shape) with the index finger pointing the pointing side of the right hand toward the photographing device 15 is the corresponding information (sales related information) and the gesture on the screen. This indicates that the overlapping product is a “discounted product”. Note that the same discount amount (discount rate) may be set in advance for the discount target products with the gesture IDs “G020” and “G021”. For example, the discount target product with the gesture ID “G020” may be “ The discount target product with the “10 yen discount” and the gesture ID “G021” may be set as “20 yen discount”.

  As shown in FIG. 16A, the gesture recognizing unit 128, for example, when the product detection unit 122 cannot accurately recognize the number of products (object b: melon bread) overlapped with each other as shown in FIG. As shown in FIG. 4, the operator recognizes the number of products by gesture. That is, in FIG. 16B, the operator recognizes an image by overlapping the shape (gesture) of the hand with the gesture ID “G005” in the gesture master information 500 (see FIG. 15) with the product (object b: melon bread). As a result, it is recognized that there are “5” merchandise (object b: melon bread).

  In addition, as shown in FIG. 17A, the gesture recognition unit 128, when two different products (object a: curry bread, object b: melon bread) are detected by the product detection unit 122, When it is desired to discount only an object (for example, object b: melon bread), the product registration device 1A recognizes the image shown in FIG. That is, the operator recognizes the image by overlapping the shape (gesture) of the hand with the gesture ID “G021” in the gesture master information 500 (see FIG. 15) with the product to be discounted, in this case, the object b (melon pan). Accordingly, the product (object b: melon bread) is the discount target product, and the discount amount (for example, 20 yen discount) can be determined.

<Process flow>
Next, the process flow of the merchandise registration apparatus 1A will be described.
FIG. 18 is a flowchart showing a flow of merchandise / sales related information registration processing of the merchandise registration apparatus 1A according to the second embodiment of the present invention. Of the processes shown in FIG. 18, the same processes as those in the product registration apparatus 1 according to the first embodiment of the present invention shown in FIG.

  First, the processing unit 10A of the commodity registration apparatus 1A performs the same processing as steps S10 to S12 in FIG. 13 and determines whether or not the recognized object overlaps a predetermined area (step S12). When the object area determination unit 121 determines that the recognized object does not overlap the predetermined area (step S12 → No), the object area determination unit 121 determines that the object is a product, and the object for the product in steps S19 → S22 Perform recognition processing.

  On the other hand, when the object region determination unit 121 determines that the recognized object overlaps the predetermined region (step S12 → Yes), the index object detection unit 125 and the gesture recognition unit 128 perform the following processing on the object (object recognition Repeat for each image) (steps S13 to S18).

  First, the index object recognition unit 126 of the index object detection unit 125 calculates feature quantities such as shape, size, hue, and the like for the acquired object recognition image (separated image), and the index object master information 200 in the storage unit 23. The similarity is calculated by comparing with (see FIG. 6) (step S14). Then, the index object recognition unit 126 determines whether or not this object is a registered index object (step S15). If it is determined that the object is an index object (step S15 → Yes), the process proceeds to step S100. On the other hand, when this object is not determined to be an index object (step S15 → No), the process proceeds to step S18.

  In step S100, the index object position determination unit 127A recognizes that the index object recognition unit 126 recognizes that the index object is “hand”, and whether or not the index object (“hand”) overlaps the product image. Determine whether. Here, when the condition of step S100 is not satisfied (step S100 → No), the process proceeds to step S16, and the index object position determination unit 127A determines a predetermined area (for example, the edge 30 of the tray 3) for the object. Determine the position at. Then, the sales related information determination unit 130 refers to the index object position correspondence information 300 (FIG. 8) to determine the sales related information (step S17).

  On the other hand, when the condition of step S100 is satisfied (step S100 → Yes), the gesture recognition unit 128 is stored in advance as the feature amount of the index object and the gesture master information 500 (FIG. 15) in the storage unit 23. The feature amount of the hand shape (gesture) is compared, and the hand shape (gesture) of the index object is recognized (step S101). Then, the gesture recognition unit 128 acquires correspondence information (sales related information) regarding the recognized hand shape (gesture) (step S102).

Then, in step S18, the processing unit 10A determines whether the processing has been repeated for all of the object recognition images (separated images) acquired from the object region determination unit 121. If the processing unit 10A has not repeated the process for all acquired object recognition images (separated images), the processing unit 10A returns to the process of step S13, and if the process has been repeated for all acquired object recognition images (separated images). If so, the process proceeds to step S23.
The processing after step S23 is the same as the processing shown in FIG. 13, and product sales registration information 400 (see FIG. 12) is displayed when product information, sales related information, etc. are displayed on the display 12A and confirmed. ) And finish the process.

  In this way, the product registration device 1A recognizes the shape (gesture) of the “hand” that is the index object when registering the product by object recognition, thereby displaying more detailed sales related information on the touch display. It is possible to register without performing an operation at 12 or the like. Therefore, it is possible to eliminate a complicated operation using the touch display 12 for inputting sales-related information, a register key, or the like.

(Modification)
The present invention is not limited to the above embodiment, and can be modified without departing from the spirit of the present invention.
(A) The product registration apparatus 1 of the present invention is a food other than bread, for example, vegetables represented by cabbage and carrots, fruits represented by apples and pears, and cooked products represented by chikuwa and tempura. At the time of recognition, customer demographic information of the customer who purchased the product, discount information on the product, etc. may be registered as sales related information together with product information.
(B) In the present embodiment, the edge 30 (see FIG. 4) of the tray 3 has been described as an example of the predetermined area. However, the present invention is not limited to this, and the product registration apparatus 1 previously stores the inside of the imaging area 151 of the imaging apparatus 15. By registering a specific area, the same effect can be obtained. For example, even when the tray 3 is not provided, a colored area is provided on the surface of the imaging table 16 in the imaging area 151 so that it can be recognized as a predetermined area, and the object overlaps the area. The merchandise registration apparatus 1 may determine that it is an index object.
(C) In the present embodiment, the hand and tongue have been described as examples of the index object. However, according to the product to be handled, chopsticks, forks, spoons, and the like may be used. ) In advance.

<Others>
The invention described in the scope of claims attached to the application of this application will be added below. The item numbers of the claims described in the appendix are as set forth in the claims attached to the application of this application.
[Appendix]
<Claim 1>
Storage means in which the sales related information is stored in association with an index object used for registering sales related information indicating information related to sales of products;
When the object is recognized using the photographed image photographed by the photographing means and the product and the index object are recognized, the sales related information corresponding to the index object stored in the storage means is recognized. Control means for registering together with the product information;
A product registration apparatus comprising:
<Claim 2>
The control means recognizes an object positioned over a predetermined area among objects recognized using the captured image as the index object, and selects an object positioned without overlapping the predetermined area as the commodity. The product registration device according to claim 1, wherein the product registration device is recognized.
<Claim 3>
The sales related information is customer demographic information indicating a customer demographic that purchases the product,
The storage means stores a customer group having a different attribute in association with each position where the indicator in the predetermined area overlaps,
The product registration apparatus according to claim 2, wherein the control unit determines the customer base based on a position where the index object overlaps the predetermined area and registers the customer base as the sales related information.
<Claim 4>
The sales related information is discount information on the price of the product,
The storage means stores the discount information in association with each position where the index object overlaps in the predetermined area,
The product registration apparatus according to claim 2, wherein the control unit acquires the discount information and registers the discount information as the sales related information based on a position where the index object overlaps the predetermined area.
<Claim 5>
The storage means stores a plurality of shapes of the hand as the index object in association with the sales related information,
The control means recognizes the shape of the hand when the object recognized as the index object is a hand and the object recognized as the product and the object recognized as the index object overlap in the captured image. 3. The product registration apparatus according to claim 2, wherein the corresponding sales related information is acquired and registered.
<Claim 6>
6. The product registration apparatus according to claim 5, wherein the storage unit stores each of a plurality of shapes of the hand as the index object in association with the number of the products as the sales related information.
<Claim 7>
6. The product registration apparatus according to claim 5, wherein the storage unit stores each of a plurality of shapes of the hand that is the index object as the sales related information in association with discount information of the product.
<Claim 8>
Storing the sales related information in a storage unit in advance in association with an index object used for registering sales related information indicating information related to the sale of the product;
Recognizing an object using a photographed image photographed by a photographing means;
When the product and the index product are recognized, registering sales related information corresponding to the index product stored in the storage unit together with the recognized product information;
A product registration method characterized by comprising:

DESCRIPTION OF SYMBOLS 1,1A Merchandise registration apparatus 10,10A Processing part 11 Display for customer 12 Touch display 12A Display 12B Touch panel 13 Cash drawer 14 Printer 15 Imaging device 16 Imaging stand 17 Backlight light source 20 CPU
21 RAM
22 ROM
23 storage unit 100 product master information 110 video storage unit 120 object recognition processing unit 121 object region determination unit 122 product detection unit 123 product similarity calculation unit 124 product similarity determination unit 125 index object detection unit 126 index object recognition unit 127, 127A Index object position determination unit 128 Gesture recognition unit 130 Sales related information determination unit 140 Product / sales related information presentation unit 150 Input acquisition unit 160 Sales information registration unit 170 Information output unit 200 Index object master information 300 Index object position correspondence information 400 Product sales Registration information 500 Gesture master information

Claims (8)

Storage means in which the sales related information is stored in association with an index object used for registering sales related information indicating information related to sales of products;
When the object is recognized using the photographed image photographed by the photographing means and the product and the index object are recognized, the sales related information corresponding to the index object stored in the storage means is recognized. Control means for registering together with the product information;
A product registration apparatus comprising:   The control means recognizes an object positioned over a predetermined area among objects recognized using the captured image as the index object, and selects an object positioned without overlapping the predetermined area as the commodity. The product registration device according to claim 1, wherein the product registration device is recognized. The sales related information is customer demographic information indicating a customer demographic that purchases the product,
The storage means stores a customer group having a different attribute in association with each position where the indicator in the predetermined area overlaps,
The product registration apparatus according to claim 2, wherein the control unit determines the customer base based on a position where the index object overlaps the predetermined area and registers the customer base as the sales related information. The sales related information is discount information on the price of the product,
The storage means stores the discount information in association with each position where the index object overlaps in the predetermined area,
The product registration apparatus according to claim 2, wherein the control unit acquires the discount information and registers the discount information as the sales related information based on a position where the index object overlaps the predetermined area. The storage means stores a plurality of shapes of the hand as the index object in association with the sales related information,
The control means recognizes the shape of the hand when the object recognized as the index object is a hand and the object recognized as the product and the object recognized as the index object overlap in the captured image. 3. The product registration apparatus according to claim 2, wherein the corresponding sales related information is acquired and registered.   6. The product registration apparatus according to claim 5, wherein the storage unit stores each of a plurality of shapes of the hand as the index object in association with the number of the products as the sales related information.   6. The product registration apparatus according to claim 5, wherein the storage unit stores each of a plurality of shapes of the hand that is the index object as the sales related information in association with discount information of the product. Storing the sales related information in a storage unit in advance in association with an index object used for registering sales related information indicating information related to the sale of the product;
Recognizing an object using a photographed image photographed by a photographing means;
When the product and the index product are recognized, registering sales related information corresponding to the index product stored in the storage unit together with the recognized product information;
A product registration method characterized by comprising:

Images