JP2016024596A - Information processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for supporting operator's determination on an object product.SOLUTION: An information processor (10) comprises: a determination unit (11) for determining at least one candidate product image indicating a candidate of a product, by using a comparison result between the reference image of the product and a product image of the product photographed by a photographing part and supplementary information for supporting the determination of the product in the product image; and a display processing unit (12) for displaying at least one determined candidate product image on a display.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a technology that supports the determination of a product.

  In Patent Document 1 below, a scanner recognizes the shape of a product from image data of the product obtained by imaging the product to be settled, and a candidate for the product to be settled based on the recognized product shape. There has been proposed a method of selecting the images and displaying them on the screen. According to this method, it is possible to shorten the key input time for data related to products for which barcodes cannot be attached.

JP 2010-198137 A

  However, in the proposed method described above, the operator may not be able to accurately determine the target product. For example, when a plurality of registered products similar to each other are displayed as candidates, the operator may not be able to select the correct product from the candidates. Specifically, when “Jonagold”, “Fuji”, and “Yubayashi”, which are similar in shape, size, color, etc., are displayed as apple varieties, the operator must select the same varieties as the target product. There is a possibility that it cannot be determined accurately.

  Such a problem does not occur only in the operator of the POS register device, but also occurs in various business workers who handle the product, and may also occur in the purchaser (customer) of the product. There may be a scene for the purchaser who wants to know exactly what the product is.

  The present invention provides a technology for supporting product judgment by a user.

  Each aspect of the present invention employs the following configurations in order to solve the above-described problems.

  The first aspect relates to an information processing apparatus. The information processing apparatus according to the first aspect uses a comparison result between the reference image of the product and the product image captured by the imaging unit, and auxiliary information for assisting in identifying the product in the product image, A specifying unit that specifies at least one candidate product image indicating the product candidate; and a display processing unit that displays the specified at least one candidate product image on the display unit.

  The second aspect relates to a product image presentation method executed by at least one computer. The product image presentation method according to the second aspect uses a comparison result between a reference image of a product and a product image captured by an imaging unit, and auxiliary information for assisting in identifying a product in the product image. , Specifying at least one candidate product image indicating the product candidate, and causing the display unit to display the specified at least one candidate product image.

  Another aspect of the present invention is a program that causes at least one computer to execute the method of the second aspect, and is a computer-readable recording medium that records such a program. This recording medium includes a non-transitory tangible medium.

  According to each aspect described above, it is possible to provide a technology that supports a user's product determination.

It is a figure which shows notionally the hardware structural example of the information processing apparatus in 1st embodiment. It is a figure which shows notionally the process structural example of the information processing apparatus in 1st embodiment. It is a figure which shows the example of the memory | storage part with which the reference | standard image and auxiliary information were matched with the brand name. It is a flowchart which shows the operation example of the information processing apparatus in 1st embodiment. It is a figure which shows notionally the process structural example of the information processing apparatus in 2nd embodiment. It is a flowchart which shows the operation example of the information processing apparatus in 2nd embodiment. It is a figure which shows notionally the process structural example of the information processing apparatus in a modification. It is a figure which shows the structural example of a POS register | resistor apparatus. It is a figure which shows the structural example of a POS system.

  Embodiments of the present invention will be described below. In addition, each embodiment given below is an illustration, respectively, and this invention is not limited to the structure of each following embodiment.

[First embodiment]
[Hardware configuration]
FIG. 1 is a diagram conceptually illustrating a hardware configuration example of the information processing apparatus according to the first embodiment. As illustrated in FIG. 1, the information processing apparatus 10 includes a CPU (Central Processing Unit) 1, a memory 2, an input / output interface (I / F) 3, a communication unit 4, and the like. The CPU 1 is connected to other units via a communication line such as a bus. The memory 2 is a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, or the like. The communication unit 4 exchanges signals with other devices and devices. A portable recording medium or the like can be connected to the communication unit 4. The input / output I / F 3 is connected to the display unit 5, the input unit 6, the imaging unit 7, the sound collecting unit 8, and the like.

  The display unit 5 displays a screen corresponding to drawing data processed by a CPU 1 or a GPU (Graphics Processing Unit) (not shown) such as an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube) display. Is a unit. The input unit 6 is a unit that receives an input of a user operation, and is realized as a hardware button unit, a touch sensor, or the like, for example. The display unit 5 and the input unit 6 can be integrated and realized as a touch panel. The imaging unit 7 is a camera formed from a lens, an imaging element, and the like, and captures still images and moving images. The sound collection unit 8 is a microphone unit.

  The hardware configuration of the information processing apparatus 10 is not limited to the configuration shown in FIG. The information processing apparatus 10 may further have a configuration other than the configuration shown in FIG. The information processing apparatus 10 may not include the display unit 5, the input unit 6, the imaging unit 7, and the sound collection unit 8. In this case, the information processing apparatus 10 displays on the display unit 5 included in another apparatus, acquires an image captured by the imaging unit 7, and obtains the sound obtained by the sound collection unit 8 through communication via the communication unit 4. Data acquisition can be used.

[Processing configuration]
FIG. 2 is a diagram conceptually illustrating a processing configuration example of the information processing apparatus 10 in the first embodiment. As illustrated in FIG. 2, the information processing apparatus 10 according to the first embodiment includes a specifying unit 11, a display processing unit 12, and the like. Each processing unit is realized by the CPU 1 executing a program stored in the memory 2. The program may be installed from a portable recording medium such as a CD (Compact Disc) or a memory card or another computer via the input / output I / F 3 and stored in the memory 2.

  The specifying unit 11 uses the comparison result between the reference image of the product and the product image captured by the imaging unit 7 and auxiliary information for assisting the specification of the product in the product image, to determine the product candidate. At least one candidate product image to be shown is identified. The specifying unit 11 may use the reference image as the candidate product image, or may use another image provided in addition to the reference image as the candidate product image.

  The “reference image of a product” is a representative image of a product that has been identified in advance. For example, there are a Jonagold reference image, a Fuji reference image, and the like. The “product image captured by the imaging unit 7” is a processing target image in which a product that is a support target by the information processing apparatus 10 is captured. The reference image is stored in a storage unit (memory 2) inside the information processing apparatus 10 or a storage unit (not shown).

  The comparison result between the reference image and the product image can have various forms. As an example, information regarding the presence of a reference image that matches or approximates a product image can be a comparison result. Also, the similarity indicating the degree of coincidence between the reference image and the product image can be a comparison result. The similarity can be calculated, for example, by comparing feature amounts between images. The image feature amount is obtained by quantifying image features such as color, shape, luminance value, texture (spatial change pattern), and the like. The comparison of feature amounts between images includes calculation of similarity between feature amounts. As the similarity, various index values such as the Euclidean distance, the distance between vectors, the degree of matching between histograms, and the correlation coefficient are used. The comparison between the reference image and the product image may be executed by the information processing apparatus 10 or may be executed by another apparatus. When the comparison is performed by another apparatus, the information processing apparatus 10 acquires the comparison result from the other apparatus. The comparison result can be stored in an internal storage unit (memory 2) of the information processing apparatus 10 or an external storage unit (not shown). Various methods for comparing such images are known, and any method may be used.

  “Auxiliary information” is information that helps identify the product in the product image, and is all information except the product image itself, the product identification code, and an image from which the product identification code can be extracted. The product identification code is code information provided to completely identify a product such as a JAN (Japan Article Number) code, and the image from which the product identification code can be extracted is a product such as a barcode. It is an image representing a code symbol in which an identification code is encoded or a product identification code itself. The reason why the product image itself is removed from the “auxiliary information” is to avoid duplication, because the product image is already used as a comparison result between the product image and the reference image. In addition, the product identification code and the image from which the product identification code can be extracted are excluded from the “auxiliary information” because the product identification code can identify the product without using the comparison result between images. It is. On the other hand, since the product name may be duplicated, it may correspond to “auxiliary information” unlike the product identification code.

  It is desirable that “auxiliary information” is acquired within a predetermined time span including the time when the product image is acquired. Since “auxiliary information” needs to assist the identification of the product in the product image, the information acquired at a timing unrelated to the acquisition of the product image cannot assist the specification of the product. Or, it is likely that the auxiliary ability is low. For example, when the product image is acquired at the cash register of the store, it is desirable that the “auxiliary information” is acquired within the settlement time including the product in the product image. The predetermined time width is set to a time length according to the usage scene of the present embodiment. For example, when the present embodiment is used at a cash register, the predetermined time width is preferably set between the start of liquidation including the target product and the completion thereof. The predetermined time width may be fixedly determined or may be dynamically determined according to event detection.

  As the “auxiliary information”, for example, information about an accessory of the target product, information about a person related to the target product, information about another product related to the target product, and the like are used. For example, a product information label (a seal on which a product name or a producer is printed), a product bundling member, a decorative member and an accessory (goods, etc.) and a target product attached to the target product are attached to the target product. There may be containers and packaging members to be accommodated. A specific example of “auxiliary information” will be further described in a detailed embodiment described later. The “auxiliary information” is stored in an internal storage unit (memory 2) of the information processing apparatus 10 or an external storage unit (not shown). The reference image and the auxiliary information may be stored in association with each other in advance.

  FIG. 3 shows an example of a storage unit in which the reference image and the auxiliary information are associated with the product name. In the example of FIG. 3, a product name, a unit price, a reference image, and auxiliary information are associated with each product ID. However, in the example of FIG. 3, the image itself is represented as the reference image, but the ID of the reference image may be associated with auxiliary information or the like. In the example of FIG. 3, the feature quantity of the image representing the decorative member, the product bundling member, and the related person is stored as auxiliary information. However, as auxiliary information, producer name data, IDs of other products having a relationship with the target product, and the like may be stored.

  The display processing unit 12 causes the display unit 5 to display at least one candidate product image specified by the specifying unit 11.

[Operation example (product image presentation method)]
Hereinafter, the product image presentation method in the first embodiment will be described with reference to FIG.
FIG. 4 is a flowchart illustrating an operation example of the information processing apparatus 10 in the first embodiment. The product image presentation method in the first embodiment is executed by at least one computer such as the information processing apparatus 10 and includes a plurality of steps shown in (S31) and (S32) as shown in FIG. Each process is realized by executing a program by the at least one computer (CPU 1). Moreover, since each process is the same as the processing content of each above-mentioned process part which the information processing apparatus 10 has, the detail of each process is abbreviate | omitted suitably.

The information processing apparatus 10 uses the comparison result between the reference image of the product and the product image captured by the imaging unit 7 and the auxiliary information for assisting the specification of the product in the product image, and the product candidate At least one candidate product image indicating is specified (S31).
The information processing apparatus 10 displays at least one candidate product image specified in (S31) on the display unit 5 (S32).

  It supplements the above-mentioned processing. For example, when the imaging unit 7 captures a product image, the information processing apparatus 10 compares the product image with a reference image stored in the storage unit. Based on the comparison result, the information processing apparatus 10 extracts, for example, a reference image having a feature amount with a similarity greater than or equal to a predetermined value as a candidate product image. There may be a plurality of extracted candidate product images.

  For example, when the auxiliary information is a product information label (a sticker on which a product name or a producer is printed) attached to the target product, the information processing device 10 extracts the product label from the product image. . The information processing apparatus 10 narrows down the candidate product images using the extracted product labels. For example, the information processing apparatus 10 is associated with auxiliary information (feature amount) having a correlation greater than or equal to a predetermined value with the feature amount of the extracted product label by referring to the association information as illustrated in FIG. Identify candidate product images. Thereby, for example, when the extracted product label is a seal A representing sweet summer, the candidate product image is narrowed down to sweet summer as shown in FIG.

[Operation and Effect in First Embodiment]
As described above, in the first embodiment, at least one of the comparison result between the reference image of the product and the product image captured by the imaging unit 7 and auxiliary information that assists in specifying the product in the product image are used. A candidate product image is specified, and the specified candidate product image is displayed on the display unit 5. Thereby, the user can confirm a candidate product image as a candidate for the product on the display unit 5 by photographing the desired product.

  Furthermore, in the first embodiment, when specifying the candidate product image, not only the comparison result between images but also auxiliary information for assisting the specification of the product is used. Therefore, according to the first embodiment, it is possible to present more carefully selected candidate product images by taking into account not only the appearance information of the product itself such as shape, size, and hue, but also other auxiliary information. it can. Thereby, identification of the target product by the user can be facilitated by the candidate product image displayed on the display unit 5.

[Second Embodiment]
Hereinafter, further details of the above-described first embodiment will be described as a second embodiment. Hereinafter, the information processing apparatus 10 and the product image presentation method in the second embodiment will be described focusing on the content different from the above-described first embodiment, and the same content as the first embodiment will be omitted as appropriate.

[Processing configuration]
FIG. 5 is a diagram conceptually illustrating a processing configuration example of the information processing apparatus 10 in the second embodiment. As illustrated in FIG. 5, the information processing apparatus 10 according to the second embodiment further includes an information acquisition unit 13, a comparison processing unit 14, and an information storage unit 18 in addition to the configuration of the first embodiment. The information acquisition unit 13, the comparison processing unit 14, and the information storage unit 18 are realized in the same manner as other processing units.

  The information storage unit 18 stores, for a plurality of types of products, first association information that associates a product with a reference image and second association information that associates a product with auxiliary information. For example, the information storage unit 18 stores the association information between the product ID and the reference image or the association information between the product ID and the feature amount of the reference image as the first association information. In the second association information, a plurality of auxiliary information may be associated with one product, or a plurality of products may be associated with one auxiliary information. For example, the information storage unit 18 stores the association information between the product ID and the auxiliary information as the second association information. The auxiliary information associated with the second association information may include an image itself, an image feature amount, character string data, value data, and the like. The first and second association information may be stored in one table as illustrated in FIG.

  The comparison processing unit 14 obtains a comparison result between the product reference image and the product image captured by the imaging unit 7. When the product image captured by the imaging unit 7 is acquired, the comparison processing unit 14 refers to the first association information stored in the information storage unit 18 and compares the product image with each reference image. Any known method can be used for this image comparison method. For example, the image comparison method includes calculating the similarity between the feature amount of the reference image and the feature amount of the product image. The comparison processing unit 14 can obtain a comparison result by such comparison processing between images. For example, the comparison processing unit 14 can acquire the similarity for each reference image as the comparison result.

  The specifying unit 11 specifies at least one candidate product image indicating a product of a reference image whose comparison result obtained by the comparison processing unit 14 satisfies a predetermined condition, and uses the auxiliary information to specify the specified at least one candidate At least one candidate product image is narrowed down from the product images. Conversely, the specifying unit 11 specifies a plurality of candidate product images using the auxiliary information, and uses at least one of the specified candidate product images using the comparison result obtained by the comparison processing unit 14. It is also possible to narrow down one candidate product image.

  For example, when each similarity between the product image and each reference image is obtained by the comparison processing unit 14, the specifying unit 11 selects a predetermined image from all the reference images stored in the information storage unit 18. A reference image showing a similarity exceeding the threshold is specified as a candidate product image. When a display image for each product is stored separately from the reference image, the specifying unit 11 selects a display image indicating the same product as the reference image indicating the degree of similarity exceeding a predetermined threshold as a candidate product image. You may specify as. In this case, the reference image and the display image are associated with each other by information that can identify the product or an image ID. The display image is held by the information processing apparatus 10 or another apparatus.

  The specifying unit 11 further narrows down the candidate product images based on the auxiliary information acquired by the information acquisition unit 13 described later and the second association information stored in the information storage unit 18. For example, the identification unit 11 identifies auxiliary information that matches or approximates auxiliary information acquired by the information acquisition unit 13 described later from the second association information stored in the information storage unit 18. The ID of the product linked to the auxiliary information is extracted. Here, when the auxiliary information is a feature amount of the image, the specifying unit 11 adds the auxiliary information acquired by the information acquiring unit 13 to the auxiliary information indicating a similarity equal to or higher than a predetermined value in the second association information. You may specify from. In addition, when the auxiliary information is character string data, the specifying unit 11 converts the character string acquired as auxiliary information by the information acquisition unit 13 to a character string that matches a predetermined number or more of characters with the second association information. You may specify from among. The specifying unit 11 selects a candidate product image corresponding to the product ID extracted using the auxiliary information from the candidate product images specified by the comparison between the product image and the reference image. Further, the specifying unit 11 can narrow down the candidate product images as follows. The specifying unit 11 acquires a plurality of product IDs corresponding to a plurality of candidate product images specified by comparison between images, and extracts auxiliary information associated with each product ID from the second association information. The identification unit 11 identifies auxiliary information that matches or approximates the auxiliary information acquired by the information acquisition unit 13 among the extracted auxiliary information, and sets the product ID associated with the specified auxiliary information. A corresponding candidate product image is identified.

  The information acquisition unit 13 can acquire auxiliary information from the product image captured by the imaging unit 7. For example, the information acquisition unit 13 includes a product information label, a product bundling member, a decorative member, and an accessory attached to the target product, and a container and a package in which the product in the product image is stored. Auxiliary information is extracted from at least one of the members. In this case, the information acquisition unit 13 acquires auxiliary information from the product image by an acquisition method corresponding to a plurality of candidate product images specified by the specifying unit 11 using the comparison result obtained by the comparison processing unit 14. You may do it.

  The auxiliary information acquisition method may be different for each product ID or product category. For example, when the product category is leafy vegetables, the color, shape, character, or the like of the binding tape (product binding member) can be used as auxiliary information. When the product is a mandarin orange, the color, shape, character, etc. of the seal (product information label) can be used as auxiliary information. On the other hand, according to the candidate product image specified using the comparison result between images, it is possible to identify whether the product is a leafy vegetable or a mandarin orange. You can choose.

  Therefore, the information acquisition unit 13 holds an auxiliary information acquisition method in advance for each product or each product category. Auxiliary information acquisition methods include which part of the product image is subject to recognition processing, whether character recognition (OCR (Optical Character Recognition)) should be performed, whether shape recognition should be performed, and how There may be multiple acquisition techniques with different recognition parameters such as whether to recognize a particular shape or color. The information acquisition unit 13 may select an appropriate auxiliary information acquisition method from the information held as described above, based on the product ID corresponding to the candidate product image specified by the comparison between the images. it can.

[Operation example (product image presentation method)]
Hereinafter, the product image presentation method in the second embodiment will be described with reference to FIG.
FIG. 6 is a flowchart illustrating an operation example of the information processing apparatus 10 according to the second embodiment. The product image presentation method according to the second embodiment is executed by at least one computer such as the information processing apparatus 10 and includes a plurality of steps (S51) to (S56) as shown in FIG. Each process is realized by executing a program by the at least one computer (CPU 1). Moreover, since each process is the same as the processing content of each above-mentioned process part which the information processing apparatus 10 has, the detail of each process is abbreviate | omitted suitably.

  The information processing apparatus 10 acquires the product image captured by the imaging unit 7 (S51). The information processing apparatus 10 may acquire a product image from the imaging unit 7 connected to the input / output I / F 3, or communicate a product image captured by the imaging unit 7 of another device using a portable or recording medium. You may get through.

  The information processing apparatus 10 acquires a comparison result between the product image acquired in (S51) and the reference image of the product (S52). As described above, the information processing apparatus 10 compares the product image acquired in (S51) with each of the plurality of reference images included in the first association information stored in the information storage unit 18, and sets each reference image. Get image comparison results. Comparison between images includes calculation of similarity between images as described above.

  The information processing apparatus 10 specifies a reference image in which the comparison result acquired in (S52) satisfies a predetermined condition, and specifies at least one candidate product image indicating a product corresponding to the specified reference image (S53). . As described above, the reference image itself may be a candidate product image, or a display image indicating a product corresponding to the reference image may be a candidate product image. For example, a threshold value of similarity is set as the predetermined condition.

  The information processing apparatus 10 acquires auxiliary information for assisting in specifying the product of the product image from the product images acquired in (S51) (S54). At this time, the information processing apparatus 10 can use information (for example, product ID) related to the candidate product image specified in (53). For example, the information processing apparatus 10 acquires the auxiliary information from the product image acquired in (S51) by an acquisition method corresponding to at least one candidate product image specified in (S53) (S54). As described above, the information processing apparatus 10 acquires the acquisition method specified by the product ID corresponding to the candidate product image specified in (S53) from the auxiliary information acquisition method stored in advance for each product or each product category. And the auxiliary information is acquired using the selected acquisition method.

  The information processing apparatus 10 narrows down at least one candidate product image using the auxiliary information acquired in (S54) from at least one candidate product image specified in (S53) (S55). This narrowing process is as described above. That is, the information processing apparatus 10 specifies auxiliary information that matches or approximates the auxiliary information acquired in (S54) from the second association information stored in the information storage unit 18, and specifies the specified auxiliary information. The product ID associated with the information is specified. The information processing apparatus 10 selects a candidate product image corresponding to the product ID specified by the auxiliary information from the candidate product images specified in (S53).

  The information processing apparatus 10 causes the display unit 5 to display at least one candidate product image narrowed down (specified) in (S55) (S56).

  The execution order of each process in the product image presentation method in the second embodiment is not limited to the order shown in FIG. For example, in (S54), when the information regarding the candidate product image specified in (S53) is not used, (S54) is immediately after (S51) or (S52) or in parallel with (S52). May be executed. Further, (S53) may be executed immediately after (S55). In this case, (S53) can specify a candidate product image using the comparison result from the candidate product images specified in (S55).

[Operation and Effect in Second Embodiment]
As described above, in the second embodiment, a plurality of candidate product images each indicating the product of the reference image whose comparison result between the product image and the reference image satisfies the predetermined condition are specified, and the plurality of specified candidate product images are specified. Auxiliary information is acquired from the product image using a corresponding acquisition method.

  Therefore, according to the second embodiment, auxiliary information is reliably acquired by an acquisition method (an image recognition method or the like) corresponding to a product or a product category indicated by a candidate product image identified using a comparison result between images. be able to. Furthermore, according to the second embodiment, since information obtained from an object that can be reflected in a product image together with the target product is used as auxiliary information, auxiliary information with high accuracy can be obtained. In addition, in the second embodiment, since the candidate product images specified by the comparison result between images are further narrowed down by such highly accurate auxiliary information, the accuracy of the candidate product images displayed on the display unit 5 Can be improved.

[Other examples of auxiliary information]
The information acquisition unit 13 can use the following auxiliary information instead of or in addition to the auxiliary information obtained from the product image as described in the second embodiment. The multiple types of auxiliary information described above and below can be used in combination or independently. Hereinafter, the information processing apparatus 10 and the product image presentation method described above will be supplementarily described for each type of auxiliary information.

  The information acquisition unit 13 may use information input via the input unit 6 as auxiliary information, or may use information sent from another device as auxiliary information. The auxiliary information exemplified below is also associated with at least one product and stored in the information storage unit 18 as the second association information.

[Example using human attribute information]
The attribute information of the person related to the product in the product image can also be used as auxiliary information. The person who is related to the product may be a person who wants to purchase the product (planned), a person who is interested in the product, a person who wants to identify the product, or the like. The attribute information may include gender, age (including age group, age, etc.), job type (including students), and the like. According to such person attribute information, for example, a male product and a female product can be distinguished.

  As described above, the information processing apparatus 10 identifies candidate product images corresponding to the product image acquired in (S51) by executing (S51) to (S53). Subsequently, the information processing apparatus 10 (information acquisition unit 13) acquires the attribute information of the person related to the product in the product image acquired in (S51) as the auxiliary information. For example, the information acquisition unit 13 can acquire the attribute information of the person from the person image captured by the imaging unit 7 together with the product image. In this case, for example, one imaging unit 7 is installed at a position where a product image can be captured, and the other imaging unit 7 is installed at a position where a person existing in front of the cash register can be captured. In this case, the person in the person image captured by the other imaging unit 7 corresponds to the person who wants to purchase the product in the product image. Further, a product image and a person image may be captured by one imaging unit 7 mounted on a handy terminal used by an operator. In this scene, when the operator says "Please tell me the official name of this product", after shooting the handed product at the handy terminal, Good. In this case, the person in the captured person image corresponds to a person who is interested in the product, a person who wants to identify the product, or the like.

  The information acquisition unit 13 performs well-known image recognition on the human image captured by the imaging unit 7, thereby acquiring attribute information such as gender, age group, and occupation of the person shown in the image. . For example, the information acquisition unit 13 can hold in advance an average face image for each human attribute (for example, for each gender or age group) or a feature amount thereof. The information acquisition unit 13 can acquire, as the attribute of the person image, an attribute corresponding to a face image that shows a degree of similarity equal to or greater than a predetermined threshold with the captured person image using the stored information. The person attribute information acquired as auxiliary information may be one piece of information or a plurality of pieces of information.

  The information processing apparatus 10 (identification unit 11) identifies attribute information that matches or approximates the attribute information of the person acquired by the information acquisition unit 13 from the second association information stored in the information storage unit 18. The ID of the product linked to the attribute information is specified. The specifying unit 11 selects a candidate product image corresponding to the product ID specified using the attribute information (auxiliary information) acquired in (S54) from the candidate product images specified in (S53) ( S55).

  The information processing apparatus 10 (display processing unit 12) displays the candidate product image identified in (S55) on the display unit 5 (S56).

[Example using speaker attribute information]
The attribute information of the person as described above can also be acquired from an audio signal. That is, speaker attribute information can also be used as auxiliary information. According to the analysis on the audio signal, an audio feature amount, message content information, etc. can be extracted. According to the voice feature amount and the speech content information, it is possible to acquire the speaker's attribute information such as the gender and age of the person. For example, age, sex, etc. can be discriminated by ending expression. In this case, the second association information stored in the information storage unit 18 includes the speaker's attribute information as auxiliary information.

  As described above, the information processing apparatus 10 identifies candidate product images corresponding to the product image acquired in (S51) by executing (S51) to (S53). Subsequently, the information processing apparatus 10 (information acquisition unit 13) acquires speaker attribute information obtained by analyzing the audio signal obtained from the sound collection unit 8 as the auxiliary information (S54). The information acquisition unit 13 can acquire an audio feature amount by performing frequency analysis on the audio signal. Moreover, the information acquisition part 13 can also acquire speech content information by performing voice recognition with respect to a voice signal. The information acquisition unit 13 holds in advance a correspondence relationship between the voice feature amount or statement content and the human attribute information, and uses the correspondence relationship to correspond to the voice feature amount or statement content obtained from the voice signal. Acquire human attribute information (speaker attribute information). The sound collection unit 8 is installed at a position where sound of a person related to the product in the product image can be collected. For example, the sound collection unit 8 is installed at a cash register.

  The information processing apparatus 10 (identification unit 11) identifies attribute information that matches or approximates the speaker attribute information acquired by the information acquisition unit 13 from the second association information stored in the information storage unit 18. The ID of the product linked to the attribute information is specified. The specifying unit 11 selects a candidate product image corresponding to the product ID specified using the attribute information (auxiliary information) acquired in (S54) from the candidate product images specified in (S53) ( S55).

  The information processing apparatus 10 (display processing unit 12) displays the candidate product image identified in (S55) on the display unit 5 (S56).

[Examples using information about other products]
Information regarding other products (other types of products and products of the same type) handled at the same time as the target product in the product image can assist in the identification of the target product. For example, if a product in a product image obtained after two “Fuji” apples are handled is an apple, there is a high possibility that it is “Fuji” (hereinafter referred to as a first assumption). In addition, if the product in the product image obtained after the treatment of curry roux, onion and carrot is potato-like, the product is likely to be a varieties of potatoes that serve as ingredients for curry. (Hereafter referred to as the second assumption). The information processing apparatus 10 and the product image presentation method described above can correspond to the first assumption, the second assumption, or both.

  In this case, as illustrated in FIG. 7, the information processing apparatus 10 further includes a determination unit 16 that determines the identification of the product. Confirmation of product identification is equivalent to confirming the product ID of the product. FIG. 7 is a diagram conceptually illustrating a processing configuration example of the information processing apparatus 10 in the modification. In this modification, the information acquisition unit 13 uses the product already confirmed by the determination unit 16 as auxiliary information for assisting in identifying the product in the product image that has not yet been determined by the determination unit 16. Get information about.

  First, a case corresponding to the second assumption will be described. In this case, the second association information stored in the information storage unit 18 includes a plurality of ingredients that can form an ingredient group of a certain dish with the target product as auxiliary information associated with at least one target product. ID of other products is included.

  As described above, the information processing apparatus 10 identifies candidate product images corresponding to the product image acquired in (S51) by executing (S51) to (S53). At this time, the information processing apparatus 10 holds the product ID already determined by the determination unit 16. The information processing apparatus 10 (information acquisition unit 13) acquires a plurality of already confirmed product IDs that are held as the auxiliary information (S54).

  The information processing apparatus 10 (identifying unit 11) determines whether or not the plurality of product IDs acquired in (S54) match a predetermined number or more of product ID groups included in the auxiliary information stored in the second association information. Determine. The predetermined number used as the threshold value is set to a number of 2 or more as a number by which a certain dish type can be selected. Based on the second association information, the information processing apparatus 10 specifies the ID of the product linked to the auxiliary information including the product ID group that matches a predetermined number or more, and the candidate product image specified in (S53). A candidate product image corresponding to the specified product ID is selected from the inside (S55). As a result, the information processing apparatus 10 can further narrow down the candidate product images identified using the product images based on whether or not they can be the same cooking ingredients as other products whose identification has already been confirmed. .

  Next, a case corresponding to the first assumption will be described. In this case, the information processing apparatus 10 (specification unit 11) uses a product ID for which a predetermined number or more is already determined among the plurality of product IDs acquired in (S54). The predetermined number used here is also set to 2 or more. As described above, when two or more products having the same ID have already been identified, there is a high possibility that three or more products are purchased. At this time, the information processing apparatus 10 may acquire the number already determined for each product ID as the auxiliary information. The information processing apparatus 10 (specifying unit 11) specifies a candidate product image corresponding to the ID of a product whose identification is determined by a predetermined number or more from the candidate product images specified in (S53) (S55). As a result, the information processing apparatus 10 can display an appropriate product image for the same product as the product for which product identification has already been confirmed (S56).

  In the case of supporting both the first assumption and the second assumption, the information processing apparatus 10 first executes the process corresponding to the first assumption. The information processing apparatus 10 responds to the second assumption when the candidate product image identified in (S53) does not have a candidate product image corresponding to the ID of a product for which identification is confirmed by a predetermined number or more. What is necessary is just to perform the said process to perform.

[Supplement]
The information processing apparatus 10 (display processing unit 12) can also display information related to each candidate product image on the display unit 5 together with the candidate product images. Such related information only needs to be associated with each candidate product image and held by the information processing apparatus 10 or another apparatus. Related information may include product information such as product name, producer name, production location, popularity, efficacy, recipe information, recommended information, and the like. Furthermore, the information processing apparatus 10 can specify a product that is missing from the recipe by using information on another product identified together with the product, and can also display the product information. In addition, when the attribute information of the user who plans to purchase the product can be obtained, the information processing apparatus 10 uses the attribute information and thinks that the user likes the recipe including the product in the material. It is also possible to estimate a recipe to be displayed and display information on the estimated recipe.

[Modification]
In the second embodiment, the information processing apparatus 10 does not have the information storage unit 18, and another apparatus other than the information processing apparatus 10 may include the information storage unit 18. In this case, the comparison processing unit 14 may compare the reference image with the product image by referring to the information storage unit 18 or may request another device. When the comparison process is not performed by itself, the comparison processing unit 14 sends a comparison request to the other device together with the product image captured by the imaging unit 7 and receives the comparison result from the other device.

  A plurality of examples will be described below, and the above-described embodiments will be described in more detail. The present invention is not limited in any way by the following examples.

  FIG. 8 is a diagram illustrating a configuration example of the POS register device. As shown in FIG. 8, the first embodiment shows an example in which the information processing apparatus 10 and the product image presentation method described above are applied to a POS (Point Of Sale) register apparatus 30. The POS register device 30 according to the first embodiment includes a checker device 31 and a cashier device 41. The checker device 31 registers the product as a payment target, and sends information on the payment target product to the cashier device 41. The cashier apparatus 41 performs a settlement process for a product to be settled. The payment target can be registered in both the checker device 31 and the cashier device 41.

  The checker device 31 and the cashier device 41 according to the first embodiment include a CPU 1, a memory 2, an input / output I / F 3, a communication unit 4, and the like, as in FIG. The information processing apparatus 10 described above can be applied to both the checker apparatus 31 and the cashier apparatus 41, and the above-described product image presentation method can be executed in either the checker apparatus 31 or the cashier apparatus 41.

  The display unit 5 described above corresponds to at least one of the display unit 32 a and the display unit 42 of the touch panel unit 32. The input unit 6 described above corresponds to at least one of the touch panel 32 b, the keyboard 33, and the keyboard 43 of the touch panel unit 32. The imaging unit 7 described above corresponds to the scanner device 34. The scanner device 34 captures a product image that reflects the appearance of the product held over the reading window 36. The above-described sound collecting unit 8 is attached to, for example, the member 35 in the checker device 31, the operator of the POS register device 30, and the like.

  According to the first embodiment, when the operator of the POS register device 30 cannot accurately determine a product when registering the product as a liquidation target, the operator can hold the product over the reading window 36 to obtain a candidate product image related to the product. As a result, the product can be judged and the product can be registered as a liquidation target.

  In the first embodiment, as illustrated in FIG. 8, the POS register device 30 includes the checker device 31 and the cashier device 41, but the POS register device 30 may be configured by only the cashier device 41. In this case, the cashier device 41 only needs to have the scanner device 34.

  FIG. 9 is a diagram illustrating a configuration example of a POS system. As illustrated in FIG. 9, Example 2 illustrates an example in which the information processing apparatus 10 and the product image presentation method described above are applied to a POS system 50. The POS system 50 according to the second embodiment includes a server device 51 and a store side configuration 52, and the server device 51 and the store side configuration 52 are communicably connected via a communication network 53. The store-side configuration 52 includes a store terminal 54, a stand 55, and the like that can be connected to the communication network 53 via a communication relay device. The stand 55 is a device for connecting the store terminal 54 and other devices (such as a handy scanner, a cash drawer, a receipt printer, etc.) in a communicable manner.

  The store terminal 54 is a general-purpose computer such as a PC (Personal Computer), a notebook PC, a tablet-type terminal, a smartphone, and the like, and exchanges data with the server device 51 to thereby exchange the data with the POS register device 30 described above. A similar function is realized. The store terminal 54 includes a touch panel unit 56, an imaging unit 57, a microphone (not shown), and the like. The display unit 5 described above corresponds to the display unit of the touch panel unit 56. The input unit 6 described above corresponds to the touch panel of the touch panel unit 56. The imaging unit 7 described above corresponds to the imaging unit 57. The sound collecting unit 8 described above corresponds to a microphone built in the store terminal 54.

  The server device 51 is a general server computer, and is realized as a WEB server, an application server, or the like. The store terminal 54 and the server device 51 have a CPU 1, a memory 2, an input / output I / F 3, a communication unit 4, and the like as in FIG.

  The information processing apparatus 10 described above can be realized as only the server apparatus 51, only the shop terminal 54, or both. Similarly, the above-described product image presentation method can be executed by only the server device 51, only the shop terminal 54, or both. For example, when the WEB system is used as an interface between the server device 51 and the store terminal 54, all the processing units described above are realized by the server device 51, and the store terminal 54 may operate as a mere display unit. . Conversely, the entire processing unit may be realized in the store terminal 54. Further, a part of the processing units may be realized by the server device 51 and the remaining processing units may be realized by the store terminal 54. According to the example of FIG. 5, the comparison processing unit 14 and the information storage unit 18 may be realized by the server device 51, and the other processing units may be realized by the store terminal 54.

  According to the second embodiment, when the operator of the store terminal 54 cannot accurately determine the product when registering the product as a liquidation target, the operator can display a candidate product image related to the product by holding the product over the imaging unit 57. As a result, the product can be judged and the product can be registered as a liquidation target. Not only at the time of liquidation, but also in the case where the product presented by the customer and asked for the product information cannot be determined by himself, the operator of the store terminal 54 determines the product by the same operation and asks the customer of the product. Can teach information.

  The information processing apparatus 10 and the product image presentation method described above are not applicable only to the POS register device 30 and the POS system 50, and can be applied to any device that supports the determination of products. For example, the information processing apparatus 10 and the product image presentation method described above may be applied to a general-purpose computer such as a PC or a smart device used by general users. In this case, when the user wants to accurately grasp what the product is in his / her hand, the user views the candidate product image related to the product and information related to the candidate product by photographing the product with the imaging unit of the computer. be able to.

Thus, the information processing apparatus according to the present invention is not excluded from being realized by a plurality of apparatuses (a plurality of computers, a casing having a plurality of CPUs).
Moreover, each above-mentioned embodiment can be combined in the range in which the content does not conflict.

  Part or all of the above embodiments can be specified as follows. However, each embodiment is not limited to the following description.

1. At least one candidate product indicating the product candidate using the comparison result between the reference image of the product and the product image captured by the imaging unit, and auxiliary information for assisting the identification of the product in the product image An identification means for identifying an image;
Display processing means for causing the display unit to display the identified at least one candidate product image;
An information processing apparatus comprising:
2. The specifying means specifies a plurality of candidate product images each indicating a reference image product whose comparison result satisfies a predetermined condition, and uses the auxiliary information to select at least one of the specified candidate product images. Narrow down one candidate product image,
1. The information processing apparatus described in 1.
3. Information acquisition means for acquiring the auxiliary information;
Further comprising
The specifying unit specifies a plurality of candidate product images each indicating a product of a reference image for which the comparison result satisfies a predetermined condition,
The information acquisition means acquires the auxiliary information from the product image by an acquisition method corresponding to the plurality of candidate product images specified using the comparison result.
1. Or 2. The information processing apparatus described in 1.
4). The auxiliary information includes attribute information of a person related to the product in the product image.
1. To 3. The information processing apparatus according to any one of the above.
5). A confirmation means for confirming the identification of the product;
Further comprising
The auxiliary information includes information on a product that has already been confirmed by the confirmation unit for assisting in identifying a product that has not yet been confirmed by the confirmation unit.
1. To 4. The information processing apparatus according to any one of the above.
6). The auxiliary information includes speaker attribute information obtained by analysis on an audio signal obtained from a sound collection unit.
1. To 5. The information processing apparatus according to any one of the above.
7). The auxiliary information is acquired in a predetermined time width including when the product image is acquired.
1. To 6. The information processing apparatus according to any one of the above.
8). The auxiliary information includes at least one of a product information label, a product bundling member, a decorative member and accessories attached to the product in the product image, and a container and a packaging member in which the product in the product image is stored. Extracted from the
1. To 7. The information processing apparatus according to any one of the above.
9. In the product image presentation method executed by at least one computer,
At least one candidate product indicating the product candidate using the comparison result between the reference image of the product and the product image captured by the imaging unit, and auxiliary information for assisting the identification of the product in the product image Identify the image,
Displaying at least one identified candidate product image on a display unit;
A product image presentation method including the above.
10. The candidate product image is specified as follows:
Identifying a plurality of candidate product images each indicating a product of a reference image for which the comparison result satisfies a predetermined condition,
Using the auxiliary information, narrow down at least one candidate product image from among the plurality of identified candidate product images,
Including The product image presenting method described in 1.
11. The auxiliary information is acquired from the product image by an acquisition method corresponding to a plurality of candidate product images each indicating a product of a reference image for which the comparison result satisfies a predetermined condition.
Further includes: Or 10. The product image presenting method described in 1.
12 Obtaining attribute information of a person related to the product in the product image as the auxiliary information;
Further includes: To 11. The product image presenting method according to any one of the above.
13. Confirm product identification,
In order to assist in identifying a product that has not yet been confirmed, information on a product that has already been confirmed is acquired as the auxiliary information.
Further includes: To 12. The product image presenting method according to any one of the above.
14 Obtaining the speaker attribute information obtained by analyzing the audio signal obtained from the sound collection unit as the auxiliary information;
Further includes: To 13. The product image presenting method according to any one of the above.
15. The auxiliary information is acquired in a predetermined time width including when the product image is acquired.
9. To 14. The product image presenting method according to any one of the above.
16. The auxiliary information includes at least one of a product information label, a product bundling member, a decorative member and accessories attached to the product in the product image, and a container and a packaging member in which the product in the product image is stored. Extracted from the
9. To 15. The product image presenting method according to any one of the above.
17. 9. To 16. A program that causes at least one computer to execute the product image presentation method according to any one of the above.

1 CPU
2 Memory 4 Communication unit 5, 32 a, 42 Display unit 7 Imaging unit 8 Sound collection unit 10 Information processing device 11 Identification unit 12 Display processing unit 13 Information acquisition unit 14 Comparison processing unit 16 Determination unit 30 POS register device 31 Checker device 32 Touch panel Unit 34 Scanner device 41 Cashier device 50 POS system 51 Server device 52 Store side configuration 54 Store terminal 56 Touch panel unit 57 Imaging unit

Claims (9)

At least one candidate product indicating the product candidate using the comparison result between the reference image of the product and the product image captured by the imaging unit, and auxiliary information for assisting the identification of the product in the product image An identification means for identifying an image;
Display processing means for causing the display unit to display the identified at least one candidate product image;
An information processing apparatus comprising: The specifying means specifies a plurality of candidate product images each indicating a reference image product whose comparison result satisfies a predetermined condition, and uses the auxiliary information to select at least one of the specified candidate product images. Narrow down one candidate product image,
The information processing apparatus according to claim 1. Information acquisition means for acquiring the auxiliary information;
Further comprising
The specifying unit specifies a plurality of candidate product images each indicating a product of a reference image for which the comparison result satisfies a predetermined condition,
The information acquisition means acquires the auxiliary information from the product image by an acquisition method corresponding to the plurality of candidate product images specified using the comparison result.
The information processing apparatus according to claim 1 or 2. The auxiliary information includes attribute information of a person related to the product in the product image.
The information processing apparatus according to any one of claims 1 to 3. A confirmation means for confirming the identification of the product;
Further comprising
The auxiliary information includes information on a product that has already been confirmed by the confirmation unit for assisting in identifying a product that has not yet been confirmed by the confirmation unit.
The information processing apparatus according to any one of claims 1 to 4. The auxiliary information includes speaker attribute information obtained by analysis on an audio signal obtained from a sound collection unit.
The information processing apparatus according to any one of claims 1 to 5. The auxiliary information is acquired in a predetermined time width including when the product image is acquired.
The information processing apparatus according to any one of claims 1 to 6. In the product image presentation method executed by at least one computer,
At least one candidate product indicating the product candidate using the comparison result between the reference image of the product and the product image captured by the imaging unit, and auxiliary information for assisting the identification of the product in the product image Identify the image,
Displaying at least one identified candidate product image on a display unit;
A product image presentation method including the above.   A program that causes at least one computer to execute the product image presentation method according to claim 8.

Images